Information processor, information processing method and program

Abstract

To improve operability by a gesture on a plurality of devices.SOLUTION: An information processor includes: a detection part for detecting execution of a predetermined gesture on any one device out of a plurality of devices connected to a terminal, using image information to be recognized continuously; and a changing part for changing a first device as a target of the predetermined gesture to be valid or invalid to the terminal, and changing the other device to a state different from that of the first device.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] In a situation where there are multiple devices that can be operated by gestures, there is known a technique for limiting the target of operation by gestures to specific devices based on the user's orientation.

[0003] For example, Patent Document 1 discloses a technique that uses gesture input and the user's body orientation to enable multiple users to efficiently use multiple control target devices. Summary of the Invention [Problem to be solved by the invention]

[0004] However, conventional technologies can only operate a single device with a single gesture input. For example, a conference room may have multiple peripheral devices of the same type (such as a microphone or camera), and it is inefficient to input gestures for each peripheral device.

[0005] The present invention has been made in view of the above points, and has an object to improve operability of a plurality of devices using gestures. [Means for solving the problem]

[0006] In order to solve the above problem, the information processing device has a detection unit that uses continuously recognized image information to detect the execution of a predetermined gesture on one of multiple devices connected to a terminal, and a change unit that changes a first device that is the target of the predetermined gesture to an enabled or disabled state for the terminal, and changes the other devices to a state different from that of the first device. [Effects of the Invention]

[0007] This improves the ease of use of gestures for multiple devices. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a diagram illustrating an example of a system configuration according to an embodiment of the present invention. [Figure 2] 2 is a diagram illustrating an example of a hardware configuration of a terminal 10 according to an embodiment of the present invention. [Figure 3] 2 is a diagram illustrating an example of a functional configuration of a terminal 10 according to an embodiment of the present invention. [Figure 4] 10 is a flowchart illustrating an example of a processing procedure executed by the terminal 10 in response to connection of the device 20. [Figure 5] FIG. 2 is a diagram illustrating an example of the configuration of a registration information storage unit 16. [Figure 6] 10 is a flowchart illustrating an example of a processing procedure executed when operating the device 20 by a gesture. [Figure 7] 10A and 10B are diagrams showing an example of a state in which the device 20 is operated by a gesture. [Figure 8] 10 is a flowchart illustrating an example of a processing procedure of a gesture detection process. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a diagram showing an example of a system configuration according to an embodiment of the present invention. FIG. 1 shows a situation in which a terminal 10, devices 20a and 20b (hereinafter referred to as "devices 20" when not distinguishing between them) are arranged in a conference room (hereinafter referred to as "conference room A") at a certain location where a teleconference (remote conference) is held with other locations. Terminal 10 is an information processing device that captures video of conference room A, distributes the video and audio collected by device 20 to other locations, and plays back video and audio from other locations. For example, a PC or dedicated device (e.g., an electronic whiteboard) may be used as terminal 10.

[0010] The terminal 10 includes a distance image sensor 108. The distance image sensor 108 continuously captures (recognizes) images with depth information (hereinafter also referred to as "depth images"). "Continuously capturing" refers to capturing images continuously, and capturing images in a manner that allows the images to change over time. For example, capturing an image as a video is an example of continuous capturing. Depth information indicates the distance to a subject included in the image, and is assigned to each pixel of the image. The depth image is used to recognize the placement position of the device 20 and detect gestures by the user. The distance image sensor 108 may be built into the terminal 10.

[0011] Each device 20 is, for example, a peripheral device used in a conference. In FIG. 1, a microphone is illustrated as an example of a device. The microphone is a device that can collect the entire sound in conference room A. In this case, only one of the microphones needs to be enabled. Alternatively, the microphone may be a highly directional microphone placed for each participant, and if there are three or more participants, three or more microphones may be placed. In this case, only the microphone of the participant who is speaking may be enabled. In this embodiment, an example will be described in which one device 20 is enabled and all other devices 20 are disabled by a single gesture by the user.

[0012] Fig. 2 is a diagram showing an example of the hardware configuration of terminal 10 according to an embodiment of the present invention. Terminal 10 in Fig. 2 includes a drive device 100, an auxiliary storage device 102, a memory device 103, a CPU 104, an interface device 105, a display device 106, an input device 107, a range image sensor 108, and a plurality of USB ports 109, all of which are interconnected via a bus B.

[0013] A program for realizing processing on the terminal 10 is provided by a recording medium 101 such as a CD-ROM. When the recording medium 101 storing the program is set in the drive device 100, the program is installed from the recording medium 101 to the auxiliary storage device 102 via the drive device 100. However, the program does not necessarily have to be installed from the recording medium 101, but may be downloaded from another computer via a network. The auxiliary storage device 102 stores the installed program as well as necessary files, data, etc.

[0014] The memory device 103 reads and stores a program from the auxiliary storage device 102 when an instruction to start the program is received. The CPU 104 realizes functions related to the terminal 10 in accordance with the program stored in the memory device 103. The interface device 105 is used as an interface for connecting to a network. The display device 106 displays images and the like from other locations. The input device 107 is a button, a touch panel, or the like, and is used to input various operation instructions. The speaker outputs audio from other locations. The distance image sensor 108 continuously captures distance images in a predetermined direction of the terminal 10 (at least in the direction in which the device to be operated by the gesture is located). The USB port 109 is used as an interface (connection port) for connecting each device 20.

[0015] The terminal 10 may include a speaker or the like for outputting audio from other locations. The speaker or the like may be built into the terminal 10, or may be connected to the terminal 10 as a peripheral device.

[0016] Fig. 3 is a diagram showing an example of the functional configuration of terminal 10 according to the embodiment of the present invention. As shown in Fig. 3, terminal 10 has an image acquisition unit 11, a registration unit 12, a detection unit 13, an identification unit 14, and a change unit 15. Each of these units is realized by a process executed by CPU 104 of one or more programs installed in terminal 10. Terminal 10 also uses registration information storage unit 16. Registration information storage unit 16 can be realized using, for example, auxiliary storage device 102, or a storage device connectable to terminal 10 via a network.

[0017] The image acquisition unit 11 continuously acquires image information (depth image) captured (recognized) by the distance image sensor .

[0018] The registration unit 12 registers in the registration information storage unit 16 image information (depth image) captured (recognized) of the device 20 connected to one of a plurality of interfaces (USB port 109) to which the device 20 can be connected, and correspondence information between the interface.

[0019] The detection unit 13 detects the execution of a predetermined gesture by using image information (depth image) continuously captured (recognized) by the depth image sensor. In this embodiment, a gesture made by the user's hand is described as an example of the predetermined gesture, but the predetermined gesture may be made by using a part of the body other than the hand or an object other than the hand.

[0020] The identification unit 14 identifies the device 20 that is the target of the predetermined gesture (hereinafter referred to as the "target device 20") based on the image information used to detect the execution of the predetermined gesture and the correspondence information stored in the registration information storage unit 16. In this embodiment, the predetermined gesture is a pointing action with a finger (for example, one finger such as the index finger), and the device 20 located in the direction of the pointing finger is the target of the gesture (the target of operation). In other words, the user can perform a predetermined operation on the device 20 by pointing with the finger at the device 20 that is the target of operation. Note that, when multiple devices 20 are located in the direction of the pointing finger, the identification unit 14 identifies the device 20 closest to the fingertip as the target device 20. This makes it possible to prevent a device 20 unintended by the user from being the target of the gesture even in an environment where multiple devices 20 are located. Furthermore, identifying the target device 20 also means identifying the interface (USB port 109) to which the target device 20 is connected. Therefore, the identifying unit 14 identifies the USB port 109 to which the target device 20 is connected.

[0021] The change unit 15 changes the state of a target device 20 that is the target of a predetermined gesture, among multiple devices 20 connected to the terminal 10, to be valid or invalid with respect to the terminal 10. Whether to make the target device 20 valid or invalid depends on the state of the target device 20 before the predetermined gesture is performed. If the state of the target device 20 is valid before the predetermined gesture is performed, the change unit 15 changes the state of the target device 20 to invalid, and if the state of the target device 20 is invalid before the predetermined gesture is performed, the change unit 15 changes the state of the target device 20 to be valid. In other words, the change unit 15 changes the state of the target device to a state different from the state before the predetermined gesture is performed.

[0022] The change unit 15 also changes the other devices 20 that are not the target of the predetermined gesture to a state (opposite to the state of the target device 20) different from the state (changed state) of the target device 20. Therefore, if the target device 20 is enabled, the other devices 20 are disabled. If the target device 20 is disabled, the other devices 20 are enabled. Here, the other devices 20 may be limited to devices 20 of the same type (camera, microphone, speaker, etc. (type when devices 20 are classified based on use and function)) as the target device 20. For example, when multiple microphones and multiple cameras are connected to the terminal 10, if the target device 20 is a microphone, only the other microphones may be changed to a state different from that of the target device 20. The type of each device 20 can be acquired through USB communication. Note that enabling or disabling a device 20 means enabling or disabling input from the USB port 109 to which the device 20 is connected in a setting in an application (a web conference application in this embodiment) that uses the device 20 on the terminal 10.

[0023] The following describes the processing procedure executed by the terminal 10. First, the processing executed as preparation for enabling operation of the device 20 by gestures will be described.

[0024] FIG. 4 is a flowchart illustrating an example of a processing procedure executed by the terminal 10 in response to connection of the device 20. In FIG.

[0025] When the registration unit 12 detects that the device 20 has been connected to any of the USB ports 109 (Yes in S101), it outputs to the user an instruction to move the device 20 to a range that can be photographed by the distance image sensor 108 (S102). For example, the registration unit 12 may display a message indicating the content of such an instruction on the display device 106. Alternatively, the display device 106 may continuously display distance images that are photographed by the distance image sensor 108 and continuously acquired by the image acquisition unit 11, and the registration unit 12 may output an instruction to move the device 20 to a specific designated area (for example, an area surrounded by a rectangle) in the distance image.

[0026] The user moves and places the device 20 in accordance with such instructions.

[0027] The registration unit 12 performs image recognition of the device 20 based on the distance images continuously acquired by the image acquisition unit 11, and when it detects the placement (stationary) of the device 20 (Yes in S103), it associates the image information (depth image) acquired by the image acquisition unit 11 after placement with the identification information (hereinafter referred to as "port ID") of the USB port 109 whose connection was detected in step S101, and stores (registers) it in the registration information storage unit 16 (S104).

[0028] Fig. 5 is a diagram illustrating an example of the configuration of the registration information storage unit 16. As shown in Fig. 5, the registration information storage unit 16 stores a port ID and image information (depth image) in the arrangement state of the device 20 in association with each other. That is, image information (image information including the device 20) in the arrangement state of the device 20 connected to the USB port 109 and correspondence information between the USB port 109 are registered in the registration information storage unit 16. Note that the depth image is an image that also includes information about the distance to the device 20. Therefore, it can be said that the depth image includes position information (three-dimensional coordinates) of the device 20.

[0029] In addition, since there is little change in the background inside a conference room, it is easy to detect a newly brought-in device 20 based on a depth image. Instead of simply detecting the placement, the user may be asked to point to the placed device 20 with his / her finger. This reduces the possibility of misidentification of the device 20 to be registered.

[0030] 4 does not have to be executed for all devices 20 connected to the terminal 10. The processing procedure in FIG. 4 may be executed only for devices 20 that the user desires to operate by gesture. In this case, if the user connects a device 20 that is not intended to be operated by gesture to the USB port 109, the user may not follow the movement instruction in step S102. If the device 20 is not placed within the specified area even after a predetermined time has elapsed, the registration unit 12 may not execute step S103 and subsequent steps.

[0031] In addition, if information that can identify the device 20 can be obtained from the device 20 when connected via USB, the registration unit 12 may obtain image information of the device 20 via the Internet using the device name or the like as a query, and register the image information in the registration information storage unit 16.

[0032] Next, a description will be given of a processing procedure executed when operating the device 20 by a gesture. Fig. 6 is a flowchart for explaining an example of a processing procedure executed when operating the device 20 by a gesture.

[0033] In step S201, the detection unit 13 executes a gesture detection process based on the depth image captured by the distance image sensor 108 (S201). The gesture detection process refers to a process for detecting (by image recognition) the execution of a predetermined gesture (pointing motion with a finger). Note that step S201 is continuously executed until the predetermined gesture is detected (S202).

[0034] When the execution of a predetermined gesture is detected (Yes in S202), the identification unit 14 identifies the target device 20 based on the image information (hereinafter referred to as "detection image information") used to detect the execution of the predetermined gesture and the correspondence information stored in the registration information storage unit 16 (FIG. 5) (S203). Specifically, the identification unit 14 searches the registration information storage unit 16 (FIG. 5) for image information similar to the image information of the device 20 included in the detection image information (hereinafter referred to as "detection device image information"). The image information similar to the detection device image information refers to image information including a portion of the detection device image information having image and depth information similar to the image and depth information of the range of the target device 20. In other words, the image information similar to the detection device image information refers to image information in which the device 20 has the shape (appearance) indicated by the detection device image information and is located at the position indicated by the detection device image information. The identification unit 14 identifies the port ID (hereinafter referred to as the "target port ID") associated with the searched image information.

[0035] Next, the change unit 15 identifies other devices 20 of the same type as the target device 20 and their port IDs (S204). For example, the identification unit 14 acquires the type of the target device 20 by communicating with the device 20 connected to the USB port 109 associated with the target port ID (i.e., the target device 20). The change unit 15 acquires the type of each device 20 connected to a USB port 109 associated with a port ID other than the target port ID by communicating with the device 20. The change unit 15 identifies the port ID of the device 20 and the USB port 109 for which the same type as the type of the target device 20 has been acquired as the other devices 20 and their port IDs of the same type as the target device 20.

[0036] Next, the change unit 15 changes the state of the target device 20 to a state different from that before the execution of the predetermined gesture (S205). Specifically, the change unit 15 changes the input from the USB port 109 associated with the target port ID to valid or invalid.

[0037] Next, the change unit 15 changes the state of another device 20 of the same type as the target device 20 to a state different from the changed state of the target device 20 (S206). Specifically, the change unit 15 changes the input from the USB port 109 to which the other device is connected to between enabled and disabled.

[0038] 7 is a diagram showing an example of a state in which device 20 is operated by a gesture. In FIG. 7, a state in which device 20b is pointed at with a finger (i.e., a state in which device 20b is the target of the gesture) is shown. As a result, device 20b is enabled, and device 20a, which is a microphone like device 20b, is disabled.

[0039] In this way, according to the processing procedure of FIG. 6, a single gesture on the target device 20 can change the state of not only the target device 20 but also other devices 20.

[0040] Note that, when the state of the target device 20 is enabled, the terminal 10 may not execute the processing procedure of Fig. 6 for a predetermined period thereafter so that the state of the target device 20 is not disabled against the user's intention due to a subsequent erroneous detection. For example, the detection unit 13 may not detect a predetermined gesture for the predetermined period. The predetermined period may be a period until the end of the conference, or may be a predetermined time such as 30 minutes or 1 hour.

[0041] Next, details of step S201 will be described. Fig. 8 is a flowchart for explaining an example of a processing procedure for detecting a gesture.

[0042] In step S301, the image acquisition unit 11 acquires a depth image (image and depth information) captured by the distance image sensor .

[0043] Next, the detection unit 13 determines whether a hand is captured in the depth image (S302). Such a determination may be made using a machine learning model capable of detecting an object included in the image.

[0044] If a hand is captured in the depth image (Yes in S302), the detection unit 13 determines whether the hand is in a clenched state based on the depth image (S303). Such a determination may be made using a machine learning model capable of determining the state of the hand included in the image.

[0045] If the hand is in a clenched state (Yes in S303), the detection unit 13 determines whether one finger is sticking out based on the depth image (S304). Such a determination may be made using a machine learning model that can determine the shape or state of the hand included in the image. Furthermore, the determinations in steps S302 and S303 may be made simultaneously using the same machine learning model.

[0046] If one finger is sticking out (Yes in S304), the detection unit 13 detects that a predetermined gesture is being performed (S305).

[0047] On the other hand, if the hand is not shown (No in S302), the hand is not in a clenched position (No in S303), or one finger is not sticking out (No in S304), steps S301 and onwards are repeated.

[0048] In addition, when the device 20 can be alternately enabled and disabled by a single action (pointing with a finger) as in the present embodiment, the detection unit 13 only needs to detect one action. Therefore, even when a gesture is detected using a machine learning model, the learning cost of the machine learning model can be reduced.

[0049] As described above, according to the present embodiment, when a gesture is executed on one of the multiple devices 20 connected to the terminal 10, not only is the state of the target device 20 changed, but the states of the other devices 20 can also be changed to states different from those of the target device 20. That is, it is possible to reduce the need to perform an operation to disable the other devices 20. Therefore, it is possible to improve the operability of gestures on the multiple devices 20.

[0050] Each function of the present embodiment can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, as well as devices such as an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), and a conventional circuit module designed to execute each function described above.

[0051] In the present embodiment, the terminal 10 is an example of an information processing device, but another computer connected to the terminal 10 may be used as the information processing device.

[0052] Although the embodiments of the present invention have been described in detail above, the present invention is not limited to such specific embodiments, and various modifications and variations are possible within the scope of the gist of the present invention as described in the claims.

[0053] For example, aspects of the present invention are as follows.

[0054] <1> a detection unit that detects, using continuously recognized image information, a predetermined gesture being performed on any one of a plurality of devices connected to the terminal; a change unit that changes a first device that is a target of the predetermined gesture among the devices to a valid or invalid state for the terminal, and changes the other devices to a state different from that of the first device; An information processing device comprising:

[0055] <2> the terminal has a plurality of interfaces to which devices can be connected; a registration unit that registers image information in which a device connected to any one of the interfaces is recognized and information corresponding to the interface in a storage unit; an identification unit that identifies the first device that is the target of the predetermined gesture based on image information used to detect the execution of the predetermined gesture and the correspondence information; and the change unit changes an input from a first interface corresponding to the first device among the plurality of interfaces to a valid or invalid state, and changes an input from an interface other than the first interface to a state different from that of the input from the first interface; Characterized by <1> The information processing device described in

[0056] <3> the change unit changes the state of the first device to a state different from the state before the predetermined gesture is performed. Characterized by <1> or <2> The information processing device described in

[0057] <4> the detection unit does not detect the predetermined gesture for a predetermined period after the state of the first device is changed to enabled; Characterized by <1> ~ <3> 2. The information processing device according to claim 1 ,

[0058] <5> the predetermined gesture is a pointing motion, When a plurality of devices are arranged in the direction pointed by the finger, the identification unit identifies the device closest to the fingertip as the first device. Characterized by <2> The information processing device described in

[0059] <6> a detection step of detecting, using continuously recognized image information, whether a predetermined gesture has been performed on any one of a plurality of devices connected to the terminal; a change procedure for changing a first device that is the target of the predetermined gesture to a valid or invalid state for the terminal, and changing the other devices to a state different from that of the first device; An information processing method characterized by being executed by a computer.

[0060] <7> a detection step of detecting, using continuously recognized image information, whether a predetermined gesture has been performed on any one of a plurality of devices connected to the terminal; a change procedure for changing a first device that is the target of the predetermined gesture to a valid or invalid state for the terminal, and changing the other devices to a state different from that of the first device; A program that causes a computer to execute the following. [Explanation of symbols]

[0061] 10 devices 11 Image acquisition unit 12 Registration Department 13. Detection unit 14 Specific section 15 Changes 16 Registration information storage unit 20 equipment 100 Drive device 101 Recording media 102 Auxiliary storage device 103 Memory Device 104 CPU 105 Interface Device 106 Display device 107 Input Device 108 Range image sensor 109 USB ports B Bus [Prior art documents] [Patent documents]

[0062] [Patent Document 1] Japanese Patent Application Laid-Open No. 2015-50640

Claims

1. a detection unit that detects, using continuously recognized image information, a predetermined gesture being performed on any one of a plurality of devices connected to the terminal; a change unit that changes a first device that is the target of the predetermined gesture to a valid or invalid state for the terminal, and changes the other devices to a state different from that of the first device; An information processing device comprising:

2. the terminal has a plurality of interfaces to which devices can be connected; a registration unit that registers image information in which a device connected to any one of the interfaces is recognized and information corresponding to the interface in a storage unit; an identification unit that identifies the first device that is the target of the predetermined gesture based on image information used to detect the execution of the predetermined gesture and the correspondence information; and the change unit changes an input from a first interface corresponding to the first device among the plurality of interfaces to a valid or invalid state, and changes an input from an interface other than the first interface to a state different from that of the input from the first interface; 2. The information processing apparatus according to claim 1, wherein:

3. the change unit changes the state of the first device to a state different from a state before the execution of the predetermined gesture.

2. The information processing apparatus according to claim 1, wherein:

4. the detection unit does not detect the predetermined gesture for a predetermined period after the state of the first device is changed to enabled; 2. The information processing apparatus according to claim 1, wherein:

5. the predetermined gesture is a pointing motion, When a plurality of devices are arranged in the direction pointed by the finger, the identification unit identifies the device closest to the fingertip as the first device.

3. The information processing apparatus according to claim 2, wherein:

6. a detection step of detecting, using continuously recognized image information, whether a predetermined gesture has been performed on any one of a plurality of devices connected to the terminal; a change procedure for changing a first device that is the target of the predetermined gesture to a valid or invalid state for the terminal, and changing the other devices to a state different from that of the first device; An information processing method characterized by being executed by a computer.

7. a detection step of detecting, using continuously recognized image information, whether a predetermined gesture has been performed on any one of a plurality of devices connected to the terminal; a change procedure for changing a first device that is the target of the predetermined gesture to a valid or invalid state for the terminal, and changing the other devices to a state different from that of the first device; A program that causes a computer to execute the following.

Images