Information processing apparatus, information processing method, and program

Abstract

To easily control projection on a projection target.SOLUTION: A control circuit 11 of an information processing apparatus 10 specifies operation information of a projector 30 that performs projection on a projection target 50 on the basis of motion of the projection target 50 moved by an operator (user) 40, and controls the projection on the projection target 50 through the projector 30 on the basis of the specified operation information.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] Conventionally, a display device (projector) has been disclosed that changes the display state of an image displayed on a projection target (screen) when a gesture corresponding to a preset condition is made by a user's fingers (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2014-115802 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, the display device disclosed in Patent Document 1 has a problem in that if the user's hands are occupied for some reason, it may be difficult to perform gestures with the fingers, making it difficult to control the projection target.

[0005] The present invention has been made in consideration of such problems, and has an object to enable a user to easily control projection onto a projection target. [Means for solving the problem]

[0006] In order to solve the above problems, the information processing device according to the present invention comprises: Identifying operation information of a projector that projects onto the projection target based on a movement of the projection target by a user; A control means for controlling the projection onto the projection target by the projector based on the identified operation information; The present invention is characterized by comprising: Effect of the Invention

[0007] According to the present invention, projection onto a projection target can be easily controlled. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a schematic diagram showing a configuration of an information processing system. [Diagram 2] FIG. 2 is a block diagram showing a functional configuration of an information processing device. [Diagram 3] 10 is a flowchart showing a control procedure of a projector control process. [Figure 4] FIG. 13 is a diagram illustrating an example of an operation table. [Diagram 5] FIG. 11 is a diagram illustrating an example of the operation of the projector. [Figure 6] FIG. 11 is a diagram illustrating an example of the operation of the projector. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0010] <Outline of the information processing system> FIG. 1 is a schematic diagram showing the configuration of an information processing system 1 according to the present embodiment. The information processing system 1 includes an information processing device 10, an image capturing device 20, and a projector (projector) 30. The information processing device 10 is communicatively connected to the image capturing device 20 and the projector 30 wirelessly or via a wire, and is capable of transmitting and receiving data such as control signals and image data between the image capturing device 20 and the projector 30.

[0011] The information processing device 10 of the information processing system 1 recognizes the movement of the projection target 50 (for example, a panel, a book, etc.) based on the captured image of the operator (user) 40 holding the projection target 50 captured by the imaging device 20, and also recognizes the face direction and eye state (direction of gaze and open / closed state of the eyes) of the operator 40, and controls the operation of the projector 30 (image projection operation, operation to change various settings, etc.) according to the recognized movement of the projection target 50 and the face direction and eye state of the operator 40. As a result, even if the operator 40's hands are full, the projector 30 can be operated by, for example, switching the image projected by the projector 30 to the next image by the operator 40 tilting the projection target 50 to the right and directing his / her gaze to the right, or scrolling the image projected by the projector 30 to the left in the same direction as the gaze by the operator 40 sharply moving the projection target 50 back and forth and directing his / her gaze to the left.

[0012] <Configuration of information processing system> <Information processing device> FIG. 2 is a block diagram showing the functional configuration of the information processing device 10. As shown in FIG. The information processing device 10 includes a control circuit 11, a RAM (Random Access Memory) 12, a storage unit 13, an operation unit 14, a display unit 15, a communication unit 16, and a bus 17. The various units of the information processing device 10 are connected to each other via the bus 17. In this embodiment, the information processing device 10 is a notebook PC, but is not limited thereto and may be, for example, a stationary PC, a smartphone, or a tablet terminal.

[0013] The control circuit (control means) 11 is a processor that reads and executes a program 131 stored in the storage unit 13 and performs various arithmetic processing to control the operation of the information processing device 10. The control circuit 11 may be a single CPU (Central Processing Unit), or may be configured with multiple processors that combine other processors such as a GPU (Graphics Processing Unit) and a DPU (Data Processing Unit) in addition to the CPU as necessary. In this case, the multiple processors may execute the multiple processes executed by the control circuit 11 of this embodiment. In addition, the multiple processors may be involved in a common process, or the multiple processors may independently execute different processes in parallel.

[0014] The RAM 12 provides a working memory space for the control circuit 11 and stores temporary data.

[0015] The storage unit 13 is a non-transitory recording medium readable by the control circuit 11 as a computer, and stores a program 131 and various data. The storage unit 13 includes a non-volatile memory such as a hard disk drive (HDD) or a solid state drive (SSD). The program 131 is stored in the storage unit 13 in the form of a computer-readable program code. The data stored in the storage unit 13 includes an operation table 132 (see FIG. 4) used when controlling the operation of the projector 30.

[0016] The operation unit 14 has at least one of a touch panel superimposed on the display screen of the display unit 15, a physical button, a pointing device such as a mouse, and an input device such as a keyboard, and outputs operation information corresponding to an input operation on the input device to the control circuit 11.

[0017] The display unit 15 includes a display device such as a liquid crystal display, and performs various displays on the display device in accordance with display control signals from the control circuit 11.

[0018] The communication unit 16 is configured with a network card or a communication module, and transmits and receives data between the image capture device 20 and the projector 30 in accordance with a predetermined communication protocol.

[0019] <Photographic equipment> 1 is, for example, a three-dimensional (3D) camera that captures an image of a shooting range including an operator 40 and a projection target 50 held by the operator 40, and generates color image data related to a two-dimensional color image of the shooting range, and generates depth image data related to a depth image including depth information of the shooting range. The shooting device 20 transmits the generated image data (color image data and depth image data) to the information processing device 10.

[0020] The imaging device 20 continuously captures images of the operator 40 and the projection target 50 facing the imaging device 20 at a predetermined frame rate. Note that the imaging device 20 only needs to be able to capture an image of a capture range including the operator 40 and the projection target 50, and may be configured to use, for example, a three-dimensional (3D) camera and a two-dimensional (2D) camera in combination, or may be configured with only a 2D camera.

[0021] <Projector> 1 projects (forms) an image on a projection surface of a projection target 50 by irradiating projection light having an intensity distribution according to image data of the projection image with high directivity. In detail, the projector 30 includes a light source, a display element such as a digital micromirror device (DMD) that adjusts the intensity distribution of the light output from the light source to form an optical image, and a projection lens group that collects the optical image formed by the display element and projects it as an image. The projector 30 changes the image to be projected and changes settings related to the projection mode (brightness, color, etc.) according to a control signal transmitted from the information processing device 10.

[0022] In this embodiment, it is assumed that the projection target 50 undergoes translational movement and rotational movement while being held by the operator 40. For this reason, the projector 30 assumes that the image projected onto the projection target 50 will be distorted in the vertical or horizontal direction due to the translational operation or rotational movement of the projection target 50, and has a correction function for correcting the distorted image or rotated image so that it can be viewed normally.

[0023] <Operation of information processing system> Next, an operation of the information processing system 1 will be described. Specifically, an operation in which the control circuit 11 of the information processing device 10 recognizes the movement of the projection target 50 and the face direction and eye state of the operator 40 to control the operation of the projector 30 will be described with reference to Fig. 3 to Fig. 6. In order to realize the above operation, the control circuit 11 executes a projector control process shown in Fig. 3.

[0024] FIG. 3 is a flowchart showing a control procedure of the projector control process. The projector control process is executed, for example, when the information processing device 10, the image capturing device 20, and the projector 30 are powered on and an instruction action based on the face direction and eye state of the projection target 50 or the operator 40 for operating the projector 30 is accepted. Note that the projector control process is assumed to be such that the processes of steps S1 to S7 described below are executed once every predetermined number of frame periods.

[0025] When the projector control process is started, the control circuit 11 transmits a control signal to the photographing device 20 to cause the photographing device 20 to start photographing (step S1).

[0026] Next, the control circuit 11 acquires image data (color image data and depth image data) relating to the image captured by the imaging device 20 (step S2).

[0027] Next, the control circuit 11 recognizes the movement of the projection target 50 based on the image data acquired in step S2, and also recognizes the face direction and eye state of the operator 40 (step S3). Note that the recognition process of the movement of the projection target 50 and the face direction and eye state of the operator 40 is performed using known technology, so detailed description thereof will be omitted here.

[0028] Next, the control circuit 11 judges whether or not the movement of the projection target 50 recognized in step S3 and at least one of the face direction and eye state of the operator 40 meet the predetermined conditions set in the operation table 132 shown in FIG. 4 (step S4). For example, if the recognized movement of the projection target 50 is a "tilt right" operation and the recognized eye state of the operator 40 is a "look right" state, the condition corresponding to the operation content "page forward" in the operation table 132 is met. Here, "tilt right" means that the operator 40 moves the right side of the projection target 50 relatively lower than the left side of the projection target 50 as seen from the operator 40, so that it reaches a threshold value (a moving distance in the vertical direction). Also, "look right" means that the operator 40 turns the line of sight to the right as seen from the operator 40, so that it reaches a threshold value (a rotation angle in the horizontal direction), as shown in FIG. 5(a). In addition, if the recognized movement of the projection target 50 is a "move right (translate right)" operation and the recognized face direction of the operator 40 is in a "turn right" state, the condition corresponding to the operation content "display the right side of the content" in the operation table 132 is met. "Move right (translate right)" means that the operator 40 moves the projection target 50 approximately in parallel to the right side as seen by the operator 40, without changing the direction of the projection target 50, so as to reach a threshold value (movement distance in the left-right direction). "Turn right" means that the operator 40 turns the face direction to the right side as seen by the operator 40, so as to reach a threshold value (rotation angle in the left-right direction). In addition, if the recognized movement of the projection target 50 is a "rotate right" operation and the recognized eye state of the operator 40 is in a "eyes open" state, the condition corresponding to the operation content "increase brightness level" in the operation table 132 is met. "Rotate to the right" means that the operator 40 rotates the projection target 50 to the right (clockwise) as viewed from above, using a virtual axis that runs along the up-down direction of the projection target 50 as the rotation axis, so that the threshold value (rotation angle) is reached.If the recognized movement of the projection target 50 is a "rotate right" movement, the amount of rotation or the rotation speed of the movement is analyzed, and the increase range for increasing the brightness level is set based on the analyzed amount of rotation or the rotation speed. Similarly, if the recognized movement of the projection target 50 is a "rotate left" movement, the amount of rotation or the rotation speed of the movement is analyzed, and the decrease range for decreasing the brightness level is set based on the analyzed amount of rotation or the rotation speed.

[0029] In step S4, if it is determined that the movement of the projection target 50 recognized in step S3 and the facial direction and eye state of the operator 40 do not meet the specified conditions set in the operation table 132 shown in FIG. 4 (step S4; NO), the control circuit 11 advances the processing to step S7.

[0030] Furthermore, in step S4, if it is determined that the movement of the projection target 50 recognized in step S3 and the face direction and eye state of the operator 40 meet the predetermined conditions set in the operation table 132 shown in Fig. 4 (step S4; YES), the control circuit 11 specifies the operation content of the projector 30 corresponding to the corresponding condition (step S5). For example, as described above, if the recognized movement of the projection target 50 is a "tilt right" operation and the recognized eye state of the operator 40 is a "look right" state, the operation content "page forward" is specified from the operation table 132.

[0031] Next, the control circuit 11 transmits a control signal to the projector 30 to perform an operation corresponding to the operation content identified in step S5 (step S6). Specifically, as shown in FIG. 5(a), when the operator 40 tilts the projection target 50 to the right and looks to the right, a control signal to perform an operation corresponding to the operation content "page forward" (see FIG. 4) is transmitted to the projector 30. As a result, the projector 30 performs an operation to switch the image Im1 that has been projected on the projection target 50 to the image Im2 of the next page, as shown in FIG. 5(b), based on the control signal to perform an operation corresponding to the operation content "page forward" transmitted from the information processing device 10. Also, as shown in FIG. 6(a), when the operator 40 tilts the projection target 50 to the left and looks to the left, a control signal to perform an operation corresponding to the operation content "page back" (see FIG. 4) is transmitted to the projector 30. As a result, based on a control signal for performing an operation corresponding to the operation content "go back to page" transmitted from the information processing device 10, the projector 30 performs an operation of switching the image Im1 that had been projected onto the projection target 50 to the image Im3 of the previous page, as shown in FIG. 6(b).

[0032] Next, the control circuit 11 determines whether or not to end the acceptance of instruction actions based on the face direction and eye state of the projection target 50 and the operator 40 for operating the projector 30 (step S7). Here, the control circuit 11 determines to end the acceptance of instruction actions based on the face direction and eye state of the projection target 50 and the operator 40 when, for example, an operation to turn off the power of the information processing device 10, the image capturing device 20, or the projector 30 has been performed.

[0033] In step S7, if it is determined that the acceptance of instruction actions based on the projection target 50, the face direction, and the eye state of the operator 40 for operating the projector 30 should not be terminated (step S7; NO), the control circuit 11 returns the process to step S2 and repeats the subsequent processes.

[0034] Furthermore, in step S7, if it is determined that the acceptance of instruction actions based on the projection target 50 and the face direction and eye state of the operator 40 for operating the projector 30 should be terminated (step S7; YES), the control circuit 11 terminates the projector control process.

[0035] Effect As described above, the control circuit 11 of the information processing device 10 identifies operation information of the projector (projector) 30 that projects onto the projection target 50 based on the movement of the projection target 50 by the operator (user) 40, and controls the projection onto the projection target 50 by the projector 30 based on the identified operation information. This makes it possible to control the projection onto the projection target 50 by the projector 30 by moving the projection target 50 even when the operator 40's hands are full. Therefore, the projection onto the projection target 50 can be easily controlled.

[0036] The control circuit 11 also recognizes the movement of the projection target 50 by the operator 40, and recognizes the face direction of the operator 40, and specifies the operation information of the projector 30 that projects onto the projection target 50 based on the recognized movement of the projection target 50 and the face direction of the operator 40. As a result, the operation information of the projector 30 is specified by a combination of the movement of the projection target 50 and the face direction of the operator 40, so that the variation regarding the operation of the projector 30 can be increased. In this way, since the operation information is specified only by a combination of the movement of the projection target 50 and the face direction of the operator 40, even if the operator 40 mistakenly changes the face direction without being aware of the operation, if the movement of the projection target 50 is not a movement that instructs the operation, it is not determined that a specific operation instruction has been given, so that a malfunction can be prevented. Also, even if the operator 40 mistakenly moves the projection target 50 without being aware of the operation, if the face direction is not a direction that instructs the operation, it is not determined that a specific operation instruction has been given. In the above embodiment, the movement of the projection target 50 and the face direction of the operator 40 are combined, but the movement of the projection target 50 and the state of the eyes of the operator 40 may be combined.

[0037] In addition, the control circuit 11 recognizes the movement of the projection target 50 by the operator 40, and recognizes the state of the operator's 40's eyes, and specifies the operation information of the projector 30 that projects onto the projection target 50 based on the recognized movement of the projection target 50 and the state of the operator's 40's eyes. As a result, the operation information of the projector 30 is specified by a combination of the movement of the projection target 50 and the face direction and eye state of the operator 40, so that the variation regarding the operation of the projector 30 can be further increased. In this way, since the operation information is specified only by a combination of the movement of the projection target 50 and the face direction and eye state of the operator 40, even if the operator 40 mistakenly moves the projection target 50 without being aware of the operation and the face direction and eye state are in a specific state that instructs the operation, if the movement of the projection target 50 is not a movement that instructs the operation, it is not determined that a specific operation has been instructed, so that a malfunction can be prevented. Also, even if the operator 40 mistakenly moves the projection target 50 without being aware of the operation, if the face direction and eye state are not in a specific state that instructs the operation, it is not determined that a specific operation has been instructed.

[0038] <Other> The description of the above embodiment is merely an example of the information processing device, the information processing method, and the program according to the present invention, and the present invention is not limited to this. For example, in the above embodiment, the information processing device 10, the image capturing device 20, and the projector 30 are separate devices. However, the present invention is not limited to this. For example, the information processing device 10 and the photographing device 20 may be integrated. As an example, the photographing device 20 may be incorporated in the bezel (frame) of the display unit 15 of the information processing device 10. Also, the information processing device 10 and the projector 30 may be integrated. For example, the function of the information processing device 10 may be incorporated in the projector 30, and the processing executed by the information processing device 10 may be executed by a control circuit (not shown) of the projector 30. In this case, the projector 30 corresponds to the "information processing device". Also, the photographing device 20 and the projector 30 may be integrated. For example, the photographing device 20 may be incorporated in the housing of the projector 30. Also, the information processing device 10, the photographing device 20, and the projector 30 may all be integrated.

[0039] 3 is merely an example, and is not limited thereto. For example, in step S2 of the projector control process, after acquiring image data from the image capture device 20, the control circuit 11 of the information processing device 10 may determine whether the operator 40 is holding the projection target 50 with both hands based on the image data, and may perform processing from step S3 onward if it is determined that the operator 40 is holding the projection target 50 with both hands. On the other hand, if it is determined that the operator 40 is not holding the projection target 50 with both hands, the control circuit 11 may detect a gesture made by the fingers of the operator 40, and transmit a control signal to the projector 30 to cause the projector 30 to perform an operation according to the detected gesture.

[0040] 3, a predetermined gesture (for example, a gesture such as tapping twice on the projection surface of the projection target 50 with the fingers of the hand holding the projection target 50) made by the operator 40 may be used as a trigger to start recognition of the eye condition of the operator 40. This is to prevent eye fatigue caused by the operator 40 being too concerned about the eye condition and losing freedom of line of sight or opening and closing the eyes (blinking) becoming unnatural if the eye condition is constantly monitored.

[0041] 3, the movement of the projection target 50 is recognized, and the face direction and eye state of the operator 40 are recognized, but the recognition targets related to the operator 40 are not limited to these. For example, the leg movements and waist movements of the operator 40 may be added to the above targets or may be substituted for the above targets. Furthermore, when the operator 40 does not dictate, the lip movements of the operator 40 may be added to the above targets or may be substituted for the above targets.

[0042] Also, in step S4 of the projector control process shown in FIG. 3, if it is recognized in step S3 that the eyes of the operator 40 are closed, the control circuit 11 may advance the process to step S7, i.e., not operate the projector 30.

[0043] 4 is merely an example and is not limited thereto. Also, for example, the operation table 132 may be customized based on the operation of the operation unit 14 of the information processing device 10 by the operator 40 or the like.

[0044] In the above description, an example has been disclosed in which the HDD or SSD of the storage unit 13 is used as a computer-readable medium for the program according to the present invention, but the present invention is not limited to this example. As other computer-readable media, information recording media such as flash memory and CD-ROM can be applied. In addition, a carrier wave is also applied to the present invention as a medium for providing data of the program according to the present invention via a communication line.

[0045] Furthermore, the detailed configurations and detailed operations of the components of the information processing device 10, the photographing device 20, and the projector 30 in the above embodiment can of course be modified as appropriate within the scope of the present invention, for example, by incorporating at least one of the photographing device 20 and the projector 30 in the information processing device 10, or by incorporating the photographing device 20 in the projector 30. [Explanation of symbols]

[0046] 1. Information Processing Systems 10. Information processing device 11 Control circuit 12 RAM 13 Storage section 131 Programs 132 Operation Table 14 Control section 15 Display section 16 Communications Department 17 Bus 20 Imaging Equipment 30 Projector 40 Operator 50 Projected object

Claims

1. Identifying operation information of a projector that projects onto the projection target based on a movement of the projection target by a user; A control means for controlling the projection onto the projection target by the projector based on the identified operation information; An information processing device comprising:

2. The control means Recognizing a movement of the projection target by the user and recognizing a movement of the user; Identifying the operation information based on the movement of the projection target and the movement of the user.

2. The information processing apparatus according to claim 1,

3. The control means Recognizing a movement of the projection target by the user and recognizing a face direction of the user; Identifying the operation information based on the movement of the projection target and the face direction of the user.

3. The information processing apparatus according to claim 2.

4. The control means Recognizing a movement of the projection target by the user and recognizing an eye state of the user; Identifying the operation information based on the movement of the projection target and the state of the user's eyes.

3. The information processing apparatus according to claim 2.

5. The control means If it is determined that the user is not holding the projection target with both hands, a gesture of the user's fingers is recognized; and Identifying the operation information based on the recognized gesture of the user's fingers.

2. The information processing apparatus according to claim 1,

6. The control means A predetermined gesture of the user that is different from the state of the user's eyes can be recognized; Recognizing an eye state of the user when the predetermined gesture is recognized.

5. The information processing apparatus according to claim 4.

7. An information processing method executed by a computer of an information processing device, comprising: Identifying operation information of a projector that projects onto the projection target based on a movement of the projection target by a user; controlling the projection onto the projection target by the projector based on the identified operation information; 23. An information processing method comprising:

8. The computer of the information processing device, Identifying operation information of a projector for projecting onto a projection target based on a movement of the projection target by a user; controlling the projection onto the projection target by the projector based on the identified operation information; A program characterized by executing a process.

Images