Information processing device, information processing method, and program

Abstract

Provided are an information processing device, an information processing method, and a system, which can improve user convenience by guiding the operating body to an area where the position of the operating body cannot be identified when operating the operating body in an area where the position of the operating body cannot be identified. The information processing device includes: a control unit (20, 140, 170) that performs recognition processing for recognizing the position of the operating body in a projection area based on a captured image of the projection area; an operation detection processing for detecting that an operation on the operating body has been performed based on sensor data output by a sensor provided on the operating body; and a projection control processing for causing a projection unit to project an image for guiding the operating body to a position in the projection area to a projection area where the position of the operating body can be identified by the recognition processing when an operation on the operating body is detected by the operation detection processing and the position of the operating body is not identified by the recognition processing.

Description

Technical Field

[0001] The present disclosure relates to an information processing device, an information processing method, and a program. Background Art

[0002] In recent years, various display devices have been developed, such as fixed display devices including TV receivers (hereinafter referred to as TV) and projectors, and mobile display devices including smart phones and notebook PCs (personal computers). In addition, with the development of display devices, interfaces and operation methods for controlling the devices have been diversified.

[0003] For example, regarding projectors, in addition to projectors that project images onto a screen perpendicular to the ground, as represented by home theaters currently used, the emergence of projection mapping technology has also increased the number of cases where images are projected anywhere such as table surfaces, walls, ceilings, etc.

[0004] Furthermore, in some cases, a pointer or the like for indicating an image projected by a projector is used for presentation. In these cases, a pointer that outputs infrared light can be utilized. In addition, a system is proposed in which each system uses a digital pen that outputs IR (infrared spectrum) light to display (project) the trajectory of a digital pen through a projector. In particular, the system is useful for digitizing blackboard writing in the field of education.

[0005] For example, the following PTL1 discloses a technology that images an optical signal (infrared light) output from a stylus pen by an IR camera mounted on a projector, and outputs position information of the optical signal when an optical signal whose signal level exceeds a predetermined threshold is detected.

[0006] Reference List

[0007] Patent Literature

[0008] PTL1: Japanese Unexamined Patent Application Publication No. 2018-198858 Summary of the invention

[0009] Problems to be solved by the present invention

[0010] Here, even if an operating tool such as an electronic pen that outputs IR light operates normally, a situation may occur where the camera cannot detect the IR light of the electronic pen because the user (operator) himself / herself shields the IR light of the electronic pen depending on the projection area, the positional relationship between the user and the camera. However, a typical user who does not grasp the position of the camera or the detection mechanism cannot understand why drawing has stopped, thus causing inconvenience to the user.

[0011] Means of solving the problem

[0012] According to the present disclosure, an information processing device is proposed, including a controller, which performs recognition processing, operation detection processing and projection control processing, the recognition processing recognizes the position of an operating tool within a projection area based on a captured image obtained by imaging the projection area, the operation detection processing detects an operation performed on the operating tool based on sensor data output by a sensor included in the operating tool, and in a case where the operation performed on the operating tool is detected by the operation detection processing and the position of the operating tool is not recognized by the recognition processing, the projection control processing causes the projection unit to project an image for guiding the operating tool to a position inside the projection area and at which the position of the operating tool can be recognized by the recognition processing.

[0013] According to the present disclosure, an information processing method performed by a processor is proposed, the method comprising: identifying a position of an operating tool within a projection area based on a captured image obtained by imaging the projection area; detecting that an operation has been performed on the operating tool based on sensor data output by a sensor included in the operating tool; and, in a case where the operation performed on the operating tool is detected and the position of the operating tool is not identified, causing a projection unit to project an image for guiding the operating tool to a position inside the projection area and where the position of the operating tool can be identified through recognition processing.

[0014] According to the present disclosure, a program is proposed for causing a computer to function as a controller that performs the following processes: recognition processing, operation detection processing, and projection control processing, wherein the recognition processing recognizes the position of an operating tool within a projection area based on a captured image obtained by imaging the projection area, the operation detection processing detects that an operation has been performed on the operating tool based on sensor data output by a sensor included in the operating tool, and, in a case where an operation performed on the operating tool is detected by the operation detection processing and the position of the operating tool is not recognized by the recognition processing, the projection control processing causes the projection unit to project an image for guiding the operating tool to a position inside the projection area and at which the position of the operating tool can be recognized by the recognition processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] [ Figure 1 ] Figure 1 : is a diagram illustrating an overview of an information processing system according to an embodiment of the present disclosure.

[0016] [ Figure 2 ] Figure 2 is a diagram showing shielding in a case where a user is located between the projector and the projection surface according to the present embodiment.

[0017] [ Figure 3 ] Figure 3: is a diagram showing shielding in a case where a user is located between a camera and an operation tool according to the present embodiment.

[0018] [ Figure 4 ] Figure 4 is a block diagram showing an example of the overall configuration of the information processing system according to the present embodiment.

[0019] [ Figure 5 ] Figure 5 is a diagram showing an example of a guide image according to the present embodiment.

[0020] [ Figure 6 ] Figure 6 : is a flowchart showing an example of the flow of a display process of a guide image according to the present embodiment.

[0021] [ Figure 7 ] Figure 7 : is a flowchart showing an example of the flow of a display process of a guide image corresponding to an unrecognizable factor according to the present embodiment.

[0022] [ Figure 8 ] Figure 8 is a diagram illustrating recognition of a user's position and occlusion caused by the user according to the present embodiment.

[0023] [ Fig. 9 ] Fig. 9 : is a diagram showing an example of a guide image giving a notification of shielding caused by a user according to the present embodiment.

[0024] [ Fig.10 ] Fig.10 : is a diagram showing a drawable area according to the present embodiment.

[0025] [ Fig.11 ] Fig.11 is a diagram showing a case where a virtual shadow image is displayed as a guide image according to the present embodiment.

[0026] [ Fig.12 ] Fig.12 : is a diagram showing the vertical direction of a drawable area according to a modified example of the present embodiment.

[0027] [ Fig.13 ] Fig.13 : is a diagram showing an example of a guide image that clearly indicates the vertical direction of a drawable area according to a modification example of the present embodiment.

[0028] Mode for carrying out the invention

[0029] Hereinafter, some embodiments of the present disclosure are described in detail with reference to the accompanying drawings. It should be noted that in the following embodiments, components having substantially the same functional configuration are indicated by the same reference numerals, and thus redundant descriptions thereof are omitted.

[0030] It should be noted that the description is given in the following order.

[0031] 1. Overview of an information processing system according to an embodiment of the present disclosure

[0032] 2. Configuration Example

[0033] 2-1. Output device 200

[0034] 2-2. Camera 300

[0035] 2-3. Operation tool 400

[0036] 2-4. Information Processing Device 100

[0037] 3. Operation processing

[0038] 3-1. Display of guidance image

[0039] 3-2. Display of guidance image corresponding to unrecognizable factors

[0040] 4. Modify the example

[0041] 5. Conclusion

[0042] <<1. Overview of information processing system according to one embodiment of the present disclosure>>

[0043] Figure 1 1 is a diagram showing an overview of an information processing system according to an embodiment of the present disclosure. Figure 1 As shown, the information processing system according to the present embodiment includes a projector 210 for projecting an image, a projection area 211 (in Figure 1 , an area photographed by the camera 300 (so-called viewing angle) is shown as the identification area 301 , the camera 300 performing imaging, and the information processing device 100 controlling image projection performed by the projector 210 .

[0044] In addition, when the user performs a drawing operation on the projection area 211 using the operating tool 400 (so-called electronic pen), a light emitting portion (e.g., IR_LED (light emitting diode)) provided at the pen tip of the operating tool 400 emits light (specifically, for example, the light emitting portion emits light by the pressing force of the pen tip), and the information processing device 100 can detect the light emitting portion from the captured image taken by the camera 300, and recognize the position of the operating tool 400 (i.e., the drawing position). The camera 300 is preset to an imaging range (recognition area 301) that is a range equal to or slightly larger than the projection area 211 of the projector 210, and images the light (e.g., IR light) emitted by the operating tool 400. The information processing device 100 continuously recognizes the position of the operating tool 400 based on the captured image taken by the camera 300, generates a trajectory image L indicating the drawing operation relative to the operating tool 400, and causes the trajectory image L to be projected by the projector 210.

[0045] As a result, the present system can freely perform drawing using the operating tool 400 on any place (projection area 211) such as a table surface, a wall, a floor, etc.

[0046] It should be noted that the light emitting portion of the operating tool 400 may be implemented by an IR_LED. In this case, in order to avoid erroneous operation due to light other than IR light emitted by the operating tool 400, the camera 300 may be configured to perform imaging via a visible light cutoff filter (i.e., using an IR camera as the camera 300). In such a case, only the IR light of the pen tip is detected from the captured image taken by the camera 300.

[0047] However, if the IR light is blocked by any shielding object or is outside the field of view of the camera 300 ( Figure 1 If drawing is performed outside the recognition area 301 shown in FIG, the system cannot detect IR light and cannot project a trajectory image.

[0048] Here, a description will be given of a shading problem assumed when drawing is performed using the operation tool 400.

[0049] Figure 2 2 is a diagram showing shielding (shielding of the projector 210) in a case where a user is located between the projector 210 and the projection surface (projection area 211) according to the present embodiment. Figure 2 In the example shown, the user's shadow causes a projection unavailable area (shielded area 213) to be generated in projection area 211. Generally, when projector 210 performs projection, the user's shadow appears, making it relatively easy for the user to realize that he / she is shielding projector 210.

[0050] On the contrary, it is difficult for the user to notice that he / she is shielding the camera 300 because no shadow appears in the case of shielding the camera 300. Figure 3 Give a description.

[0051] Figure 3 303 is a diagram showing shielding (shielding of the camera 300) in a case where a user is located between the camera and the operating tool according to the present embodiment. In this case, an area (shielding area 303) is generated in which the IR light emission of the operating tool 400 is undetectable because the IR light emission is hidden by the user's body. In such a case, the information processing device 100 cannot recognize the position (trajectory) of the IR light emission and cannot generate the trajectory image L; therefore, even if the operating tool 400 is normally operated, the user cannot perform drawing. It should be noted that a shadow does not appear in the shielding area 303 as in the case of shielding the projector 210; therefore, it is difficult for the user to notice that he or she is a shielding object.

[0052] Furthermore, in the case of a system that projects a trajectory image, it is difficult to understand where an image can be drawn when nothing is drawn, and a situation may arise where drawing starts outside the viewing angle of the camera 300 (outside the recognition area) and cannot be executed.

[0053] Regarding such a problem that the camera 300 cannot recognize the operating tool 400 that is operating normally due to shielding, etc., no feedback is performed to the user, and the user must determine by himself why drawing becomes unavailable. In some cases, this can be improved by actions such as checking the position of the camera 300 and changing the user's standing position. However, it is difficult to distinguish whether drawing is unavailable because the user is unaware of the detection mechanism or due to other reasons (for example, shielding not caused by the user but battery exhaustion of the operating tool 400, shielding caused by other furniture or movable objects (curtains, etc.), etc.), and therefore, inconvenience is caused.

[0054] Therefore, the present disclosure proposes a mechanism capable of enhancing the convenience of the user by guiding the user to an appropriately recognizable area in a case where the position of an operating tool being operated is in an unrecognizable area.

[0055] <<2. Configuration Example>>

[0056] Figure 4 1 is a block diagram showing an example of the overall configuration of an information processing system according to the present embodiment. Figure 4 As shown, the information processing system according to the present embodiment includes an information processing device 100 , an output device 200 , a camera 300 , and an operation tool 400 .

[0057] <2-1. Output device 200>

[0058] The output device 200 includes a projector 210, a TV 220, a tablet computer 230, a smart phone 240, and a PC 250. It should be noted that the present system may include one or more of these devices, these devices combined as the output device 200, or the system may include a plurality of devices of one type among these devices. Figure 1 In the illustrated example, a system configuration including the projector 210 is shown as an example.

[0059] The projector 210 is a projector device that projects an image anywhere in space. The projector 210 may be, for example, a fixed wide-angle projector, or may be a so-called mobile projector including a movable part (e.g., a Pan / Tilt drive type) capable of changing the projection direction. The projection surface is not limited to a plane, and may be a curved surface or may be divided into a plurality of surfaces. In addition, the projector 210 may be implemented by a plurality of units.

[0060] The TV 220 is a device that outputs images and sounds by receiving radio waves of television broadcasts. The tablet computer 230 is typically a mobile device having a larger screen than the screen of the smart phone 240, which can perform radio communication, and can output images, sounds, vibrations, etc. The smart phone 240 is typically a mobile device having a smaller screen than the screen of the tablet computer 230, which can perform radio communication, and can output images, sounds, vibrations, etc. The PC 250 may be a fixed desktop PC, a mobile laptop PC, and can output images, sounds, etc.

[0061] The output device 200 outputs information based on the control performed by the information processing device 100. In addition to the content of the information to be output, the information processing device 100 can also control the output method. For example, the information processing device 100 can control the projection direction of the projector 210.

[0062] In addition, the output device 200 may include a component capable of performing any output other than those outputs described above. For example, the output device 200 may include a speaker and a unidirectional speaker. The speaker converts audio data into an analog signal via a DAC (digital-to-analog converter) and an amplifier, and performs output (playback). A unidirectional speaker is a speaker capable of forming directivity in a single direction. The information processing device 100 can control the directivity of the unidirectional speaker.

[0063] In addition, the output device 200 may include a lighting device, an air conditioner, a music playback device, and the like.

[0064] <2-2. Camera 300>

[0065] The camera 300 is an IR recognition camera for recognizing an IR LED provided on the operating tool 400, and is an exemplary sensor device capable of performing imaging via an IR filter and detecting only IR light.

[0066] In addition to the camera 300, the system may include other sensors that sense various types of information. For example, the system may include an RGB camera, a depth sensor, a microphone, etc. to sense information about the user and the space in which the user is located. The depth sensor is a device that obtains depth information, such as an infrared distance measuring device, an ultrasonic distance measuring device, a LiDAR (laser imaging detection and ranging), or a stereo camera. In addition, a ToF (time of flight) camera that can obtain high-precision distance images can be used. A microphone is a device that collects ambient sound and outputs audio data that is converted into a digital signal via an amplifier and an ADC (analog-to-digital converter). The microphone may be an array microphone.

[0067] The sensor device senses information based on control performed by the information processing device 100. For example, the information processing device 100 may control the zoom ratio and imaging direction of the camera 300.

[0068] In addition, the sensor device may also include a component capable of performing any sensing other than the above-mentioned sensing. For example, the sensor device may include a device to which information is input by a user, such as a mouse, a keyboard, a touch panel, a button, a switch, a joystick, etc. In addition, the sensor device may also include various sensors, such as an acceleration sensor, a gyroscope sensor, a geomagnetic sensor, an optical sensor, an illumination sensor, a force sensor, an ultrasonic sensor, an air pressure sensor, a gas sensor (Co2), a thermal imaging camera (far infrared camera), etc.

[0069] The number of the output devices and the number of the sensor devices may be plural.

[0070] <2-3. Operation tool 400>

[0071] The operation tool 400 is held by the user, and is used when the user draws a trajectory on the projection area 211. Specifically, the IR_LED 430 is provided at the tip (pen tip) of the operation tool 400 as an example of a light emitting portion, and when drawing is performed by bringing the pen tip into contact with the projection area 211 such as a wall, the switch 440 (sensor) is turned on by pushing the pen tip, and the IR_LED 430 is controlled to be turned on by the CPU 420. On the contrary, when the pen tip is separated from the projection area 211 such as a wall, the pen tip is released without being pressed, the switch 440 is turned off, and the IR_LED 430 is controlled to be turned off by the CPU 420.

[0072] It should be noted that although an example has been described in which the IR_LED 430 emits light by the pressing force of the pen tip on a grounded surface, the present embodiment is not limited to this, and the IR_LED 430 can emit light by detecting the operation of a button, switch, etc. set to the operating tool 400 or by detecting the operation relative to the operating tool using a motion sensor included in the operating tool.

[0073] In addition, the operating tool 400 includes a communication module 410. The communication module 410 can be coupled to the information processing device 100 via wires or radio to send and receive data. For example, the communication module 410 communicates with the information processing device 100 via a wired / wireless LAN (local area network), Wi-Fi (registered trademark), Bluetooth (registered trademark), near field communication, etc.

[0074] The CPU 420 sends a predetermined command to the information processing apparatus 100 based on the user's operation on a button or switch provided to the operating tool 400, the movement of the pen, etc. Specifically, the CPU 420 sends sensor data detected by the switch 440 (i.e., data indicating that the switch is turned on by pressing the tip, indicating that an operation (drawing operation) has been performed on the operating tool 400) from the communication module 410 to the information processing apparatus 100 in real time.

[0075] It should be noted that although the case of drawing a trajectory using the operating tool 400 is mainly described here as an example, the operation using the operating tool 400 according to the present embodiment is not limited to this, and various operations such as clicking operations, dragging and dropping operations, page turning, scrolling operations, etc. can also be performed on the displayed image.

[0076] <2-4. Information Processing Device 100>

[0077] The information processing device 100 includes an I / F (Interface) 110 , a light emitting recognizer 120 , a memory 130 , an operation detector 140 , an image generator 150 , a coordinate converter 160 , a display controller 170 , and a user position recognizer 180 .

[0078] -I / F 110

[0079] The I / F 110 is a coupling device that couples the information processing device 100 to other devices. The I / F 110 is implemented, for example, by a USB (Universal Serial Bus) connector, a wired / wireless LAN (Local Area Network), Wi-Fi (registered trademark), Bluetooth (registered trademark), a portable communication network (LTE (Long Term Evolution), 3G (Third Generation Mobile Communication System)), etc., and performs input and output of information to and from the output device 200, the camera 300, and the operation tool 400. By performing communication with these devices via the I / F 110, the information processing device 100 can control the output device 200, the camera 300, and the operation tool 400. More specifically, for example, the information processing device 100 can perform these controls by outputting control signals to the output device 200, the camera 300, and the operation tool 400 via the I / F 110.

[0080] - Luminous identifier 120

[0081] The luminescence identifier 120 detects IR light from a captured image (IR captured image) imaged by the camera 300, identifies the position coordinates of the detected IR light (e.g., the position coordinates of a bright spot), accumulates the position coordinates in the memory 130, and outputs the fact that the IR light has been detected (e.g., the fact that the position of the bright spot has been identified) to the image generator 150. In addition, the luminescence identifier 120 also has a function (tracking function) of tracking the progress of the detected IR light and acquiring the trajectory of the IR light.

[0082] -Memory 130

[0083] The memory 130 stores the light emission position coordinates recognized by the light emission identifier 120. The memory 130 may also store the installation position information of the output device 200 and the camera 300, space recognition information, user position information, and various sensor data. The space recognition information is information acquired / recognized from the projection space (projection environment) by the information processing device 100 based on various sensor data (captured images (visible light images, infrared images) taken by a camera or a bird's-eye view camera, depth information acquired by a depth sensor, distance information acquired by a distance measurement sensor, temperature information acquired by a thermal imaging sensor, voice information acquired by a microphone, etc.) acquired from a sensor device (not shown). Specifically, for example, the information processing device 100 may recognize: the three-dimensional shape of the projection space (projection environment); the three-dimensional shape, three-dimensional position, and projectable area (e.g., a plane area with a predetermined size) of a real object existing in the projection space; the three-dimensional position, posture, gesture, speech, etc. of the user. The information processing device 100 may then store them in the memory 130 as space recognition information.

[0084] - Operation detector 140

[0085] The operation detector 140 has a function of detecting that an operation has been performed on the operation tool 400 based on sensor data received from the operation tool 400. Specifically, for example, upon receiving sensor data detected by a switch 440 provided to the operation tool 400 (i.e., data indicating that the switch is turned on by pressing the tip), the operation detector 140 is able to detect that an operation (in this case, a drawing operation) has been performed on the operation tool 400. In addition, the operation detector 140 outputs the operation detection result to the image generator 150.

[0086] - Image Generator 150

[0087] The image generator 150 generates an image to be displayed by the output device 200. For example, the image generator 150 generates an image (trajectory image L) showing a trajectory drawn by the operating tool 400 in response to a drawing operation performed by the operating tool 400. Specifically, the image generator 150 generates an image showing the trajectory based on display coordinate information corresponding to the trajectory of the IR light acquired from the coordinate converter 160, and outputs the image to the display controller 170.

[0088] Furthermore, in a case where the light emission recognizer 120 does not recognize light emission (i.e., does not recognize the position of the operating tool 400) even though the operation detector 140 detects an operation (in this case, a drawing operation) on the operating tool 400, the image generator 150 according to the present embodiment generates an image (a guide image) for guiding the operating tool 400 to an area (hereinafter also referred to as a "drawable area") within the projection area 211 and recognizable by the camera 300. In a case where light emission is not recognized even though the operation on the operating tool 400 is detected, for example, it is assumed that the user performs an operation outside the recognition area 301 of the camera 300, or the IR light of the operating tool 400 is shielded by the user, furniture, or a movable object (curtain, etc.).

[0089] here, Figure 5 An example of a guide image according to the present embodiment is shown. As described above, in a case where light emission is not recognized even when an operation of the operating tool 400 is detected, since the operating tool 400 is in a normally usable state (battery is not exhausted, etc.) and in a drawing-enabled state, it is desirable to display an illustration, an icon, a guide, etc. indicating the fact that drawing can be performed in an intuitively understandable manner as a guide image. Specifically, for example, Figure 5 The illustrated images and the icon of the pen shown in the guide images 422a and 422b, or a guide display for prompting drawing, such as “Start drawing!”, etc., may be displayed on the drawable area.

[0090] -Coordinate converter 160

[0091] In response to a request from the image generator 150 , the coordinate converter 160 calls out information of the trajectory of the target from the memory 130 , calculates coordinates of an image displaying the target based on the information of the trajectory, and outputs the calculated coordinates to the image generator 150 .

[0092] -Display controller 170

[0093] The display controller 170 controls the image generated by the image generator 150 to be output to the output device 200 via the I / F 110, and controls the display of the image performed by the output device 200. For example, the display controller 170 performs control so that the trajectory image generated by the image generator 150 is projected by the projector 210. In addition, the display controller 170 performs control so that the guide image generated by the image generator 150 is projected by the projector 210. The guide image may be displayed at a predetermined position such as the edge of the drawable area, may be displayed at a position where the IR light is finally recognized, or may be displayed at a position corresponding to the installation direction of the camera 300 or the projector 210 (for example, in the case where the camera 300 is located in the right direction of the drawable area, it is displayed at the right edge). This makes it possible to guide an appropriately recognizable area to enhance the convenience of the user.

[0094] - User location identifier 180

[0095] The user position identifier 180 has a function of identifying the position of the user based on the captured image. For example, in the case where the present system uses an RGB IR camera as the camera 300, a visible light image and an IR image can be acquired simultaneously. In this case, the user position identifier 180 is able to identify the position of the person from the visible light image by skeleton estimation or the like. Based on the identified position of the user and whether luminous identification based on the IR image is possible, the information processing device 100 is able to determine that the luminous emission of the operating tool 400 is shielded by the user. It should be noted that when performing skeleton estimation based on the visible light image, it is also possible to obtain not only the skeleton position but also the area of ​​the human body by using the inter-frame difference method. In addition, in addition to the IR camera, a thermal imaging camera, a depth camera or a ToF sensor can be prepared for identifying luminescence, thereby extracting the outline of the user (the area of ​​the human body) (it is hoped that the optical axes of these sensors are as close to the IR camera as possible).

[0096] The above-mentioned light emitting identifier 120, operation detector 140, image generator 150, coordinate converter 160, display controller 170 and user position identifier 180 can be implemented by a controller (not shown) provided to the information processing device 100. The controller functions as an arithmetic processing unit and a control unit, and controls the overall operation in the information processing device 100 according to various programs. For example, the controller is implemented by an electronic circuit such as a CPU (central processing unit) or a microprocessor. In addition, the controller may include a ROM (read-only memory) that stores programs and arithmetic parameters to be used and a RAM (random access memory) that temporarily stores parameters that are appropriately changed.

[0097] In addition, the configuration of the information processing device 100 according to the present embodiment is not limited to Figure 4 For example, the information processing device 100 may be configured by a plurality of devices, or Figure 4 At least some of the functional configurations of the information processing device 100 shown may be provided to the output device 200, the camera 300, or the operation tool 400. In addition, the information processing device 100 does not necessarily have Figure 4 All functional configurations shown.

[0098] In addition, the information processing device 100 can also be implemented by a PC, a smart phone, an edge server, an intermediate server, or a cloud server.

[0099] <<3. Operation Process>>

[0100] Next, an example of the flow of a display process of a guide image according to the present embodiment will be described with reference to the drawings.

[0101] <3-1. Display of guidance image>

[0102] Figure 6 1 is a flowchart showing an example of the flow of a display process of a guide image according to the present embodiment. Figure 6 As shown, first, the information processing apparatus 100 determines whether the switch 440 of the pen tip is pressed by the operation detector 140 (by a drawing operation) based on the sensor data transmitted from the operation tool 400 (step S103).

[0103] Next, if it is determined that the switch 440 of the pen tip is pressed (step S103 / Yes), the light emission identifier 120 identifies IR light from the captured image (IR image) of the camera 300 (step S106). Here, as an example, the IR light recognition process is performed when it is detected that the switch 440 of the pen tip is pressed. However, the present embodiment is not limited thereto, and the IR light recognition process may be performed continuously (always).

[0104] Thereafter, if IR light is recognized (step S106 / Yes), the information processing device 100 performs trajectory drawing (step S109). Specifically, the information processing device 100 generates a trajectory image consistent with the position coordinates of the IR light through the image generator 150, and performs control through the display controller 170 so that the projector 210 displays (projects) the trajectory image.

[0105] On the contrary, if the IR light is not recognized (step S106 / No), the information processing device 100 displays a UI (a guide image) for guiding the operating tool 400 to a writable location (a drawable area within the projection area 211 and recognizable by the camera 300) (step S112). Specifically, the information processing device 100 generates a guide image (or acquires a guide image prepared in advance), and performs control through the display controller 170 so that the projector 210 displays (projects) the guide image. As a result, the user can intuitively understand that there is no depletion of the battery of the operating tool 400 or a system failure, and can perform drawing, and can perform drawing around the displayed guide image.

[0106] Thereafter, while the switch 440 of the pen tip continues to be pressed (while the operation detector 140 continues to detect the operation in the operation tool 400) (step S115), the information processing apparatus 100 repeats the processing shown in the above-mentioned steps S106 to S112.

[0107] <3-2. Display of a guidance image corresponding to an unrecognizable factor>

[0108] In the case where the information processing device 100 according to the present embodiment also determines based on the user's position that the factor for not being able to recognize IR light is shielding caused by the user, the information processing device 100 is able to give a notification that shielding caused by the user is occurring as a guide image and prompt the user to take appropriate action.

[0109] Figure 7 is a flowchart showing an example of the flow of a display process of a guide image corresponding to an unrecognizable factor.

[0110] exist Figure 7 In steps S103 to S109 shown in FIG. Figure 6 In other words, if the switch 440 of the pen tip of the operating tool 400 is pressed (by a drawing operation) (step S103), the information processing apparatus 100 recognizes IR light (step S106), and performs trajectory drawing (generation and display of a trajectory image) (step S109).

[0111] Next, the information processing device 100 determines whether the IR light has become unrecognizable at the user position (step S120). Specifically, in the case where the IR light has become unrecognizable by the light emitting identifier 120 even if the operation detector 140 detects the drawing operation, and the position where the IR light has become unrecognizable is the user position recognized by the user position identifier 180, it can be understood that the IR light is shielded by the user. Figure 8 is a diagram showing the identification of the user's position and the occlusion caused by the user. For example, in the case where the present system uses an RGB IR camera as the camera 300, the following can be obtained: Figure 8 The visible light image 304 is shown, and the user position identifier 180 identifies the user position through skeleton estimation, etc. based on the visible light image 304. Thereafter, in the case where the IR light tracked by the luminous identifier 120 moves toward the area identified as the user position and becomes unrecognizable (lost) (in the middle of the figure), the image generator 150 is able to determine that the IR light is shielded by the user.

[0112] Next, if the IR light has become unrecognizable at the user's position (step S120 / Yes), the image generator 150 determines that the IR light is shielded by the user, generates an image (guidance image) giving a notification that the IR light is shielded by the user, and the display controller 170 controls the generated guidance image to be projected by the projector 210 (step S123). Here, Fig. 9 An example of a guide image giving a notification of a shading caused by a user according to the present embodiment is shown. Fig. 9 The guide image 422d is an icon image that instructs the user how to enter the camera's field of view and stand in the path. Fig. 9 The guide image 422e is a guide display such as “your shadow is in the path.” By displaying such a guide image, the user can intuitively understand that drawing (to be recognized) cannot be performed due to the user's standing position, and can make improvements by confirming the position of the camera 300 and changing the user's standing position.

[0113] In addition, the display controller 170 controls the guide image to be displayed on the drawable area. Fig.10 , the drawable area according to this embodiment will be described. Fig.10As shown, the recognition area 301 corresponding to the imaging range of the camera 300 may be preset to be equal to or slightly larger than the range of the projection area 211. In addition, in the present embodiment, the projection area 211 and the recognition area 301, and also excluding the shielding area 213 (the area where the user shields the projection area 211 of the projector 210) and the shielding area 303 (the area where the user shields the recognition area 301 of the camera 300) are defined as the drawable area 421. The display controller 170 may display the guide image 422 at a predetermined position such as the edge of the drawable area 421, at a position where the IR light has been recognized for the last time, or at a position corresponding to the installation direction of the camera 300 or the projector 210 (for example, the right edge when the camera 300 is located in the right direction of the drawable area).

[0114] The shielded area 213 and the shielded area 303 can be appropriately calculated based on the visible light image acquired by the visible light camera, the three-dimensional position of the user, the three-dimensional position of the camera 300, the three-dimensional position of the projector 210, the three-dimensional position of the projection area 211, the three-dimensional position of the identification area 301, the positional relationship and distance between them, etc.

[0115] Here, although a real shadow is generated in the shielded area 213, no shadow is generated in the shielded area 303 that shields the camera 300; therefore, the information processing device 100 can generate a virtual shadow image indicating the shielded area 303 (at least the outline of the shielded area 303) as a guide image through the image generator 150, and display the virtual shadow image. Fig.11 is a diagram showing a case where a virtual shadow image is displayed as a guide image. Fig.11 As shown, in the drawable area 421, a real shadow appears in the shielded area 213 of the projection of the user shielding the projector 210, and no shadow appears in the shielded area 303 of the imaging of the user shielding the camera 300; therefore, a virtual shadow image 422f is generated, and the projector 210 displays the virtual shadow image 422f at the position of the shielded area 303. As a result, the user can intuitively understand that it is impossible to perform drawing in the user's shadow area, and to improve by changing the standing position, etc. It should be noted that an example of generating an image indicating the outline of the shielded area 303 and displaying it as a guide image is described here, but the present embodiment is not limited to this. For example, the image generator 150 of the information processing device 100 can generate an image corresponding to the position and size of the shielded area 303 (for example, a simple rectangular image including the shielded area 303) as a guide image. For example, in the case where a process that can reduce the processing load compared to the contour extraction is appropriate, the image generator 150 of the information processing device 100 can generate and display a rectangular image corresponding to the position and size of the shielded area.

[0116] In addition, in the case where projection is performed by the projector 210, the shadow of the user usually appears; therefore, the user can relatively easily notice that the user is shielding the projector 210. However, in the case where the background color of the image to be projected is black and the information to be displayed is white, since nothing is projected on the projection area by default, no shadow will appear. In this case, the information processing device 100 can also display a virtual shadow on the shielding area 213 of the projector 210. The shielding area 213 of the projector 210 can be calculated based on the position of the user, the installation position of the projector 210, and the location of the projection area. Alternatively, in the case where the projector 210 and the camera 300 are installed close to each other and their optical axes are as close to or coaxial as possible, the shielding area 303 of the camera 300 is represented as a virtual shadow, so that the user also recognizes the virtual shadow as the shadow of the projector 210. It should be noted that in this case as well, the guide image indicating the shielding area 213 of the projector 210 is not limited to the image extracted by the contour. For example, the image generator 150 of the information processing device 100 may generate an image corresponding to the position and size of the mask area 213 (eg, a simple rectangular image including the mask area 213 ).

[0117] Thereafter, if the shielding is improved by changing the user's standing position, etc. (i.e., when the IR light becomes recognizable by the luminous identifier 120 again) (step S126 / yes), the information processing device 100 generates a trajectory image and performs control to display the trajectory image again (step S129).

[0118] On the contrary, in the process of recognizing IR light (in the process of drawing), if IR light is not recognizable (step S106 / No) or if IR light becomes not recognizable at a position other than the user's position (step S120 / No), the information processing device 100 writes the IR light in a writable place (step S112) in a manner consistent with the reference. Figure 6 The UI (such as the one for guiding the operation tool 400 to the writable place (drawable area) is displayed in the same manner as the embodiment described above. Figure 5 The guide images 422a to 422c shown). The case where the IR light is not recognizable at a position other than the user's position is assumed to be a case where the IR light is outside the recognition area 301 (moved out of the viewing angle) or a case where the IR light is blocked by a shielding object other than the user (e.g., furniture or a movable object). As a result, the user can intuitively understand that it is not a battery exhaustion or a system failure of the operating tool 400, and can perform drawing, and can perform drawing around the displayed guide image. It should be noted that in the case where a shielding object has been identified by object recognition based on a captured image, or in the case where it has been identified that shielding has occurred due to any shielding object, the information processing device 100 can display a guide image for giving a notification of the shielding.

[0119] While the switch 440 of the pen tip continues to be pressed (step S132), the information processing apparatus 100 repeats the processing shown in the above-described steps S106 to S112 and S120 to S129.

[0120] <<4. Modification Example>>

[0121] Hereinafter, modified examples of the present embodiment will be described.

[0122] <4-1. Explicit indication of the vertical direction of the drawable area by guiding the image>

[0123] The guide image according to the present embodiment may also indicate the vertical direction of the drawable area according to the installation position of the projector 210 or the camera 300 .

[0124] Fig.12 : is a diagram showing the vertical direction of the drawable area according to a modified example of the present embodiment. Fig.12 As shown, for example, a case is assumed where the top surface of the workbench 20 is a projection area and the user can draw from any direction.

[0125] Around the workbench 20, a projector 210 that projects a trajectory image onto the top surface of the workbench 20 and a camera 300 that images IR light emission (picture) of the operating tools 400a and 400b are installed.

[0126] Here, depending on the positional relationship of projector 210 and camera 300 relative to the user, there may be a place where shading caused by the user is likely to occur (ie, an undesirable disadvantageous position).

[0127] For example, Fig.12 As shown, shielding does not occur at the standing position of the user U1, but at the standing position of the user U2, shielding area 213 is generated by shielding the projection of the projector 210 that performs projection from the rear of the user U2, and shielding area 303 is generated by shielding the imaging of the camera 300. It is impossible to perform drawing (recognition, projection) in the shielding area 213 and the shielding area 303, and there is a disadvantage that the size of the drawable area 421 is reduced.

[0128] Therefore, in this modified example, in order to clearly indicate the desired standing position (drawing direction) to the user, in the case of shielding (for example, the case where the operating tool 400b used by the user U2 is being operated and the light emission is not recognized, etc.), the information processing device 100 can display a guide image indicating the vertical direction. The vertical direction can be determined according to the positions of the projector 210 and the camera 300. That is, the guide image indicating the vertical direction is used to guide the user to the desired and advantageous standing position (direction) corresponding to the positions of the projector 210 and the camera 300.

[0129] Fig.13 is a diagram showing an example of a guide image that clearly indicates the vertical direction of a drawable area according to the present modification example. Fig.13 As shown, a frame-shaped guide image 422g including an illustration of an animal such as indicating the top and bottom may be displayed. In addition, other illustrations, icons, guide displays (characters), etc. indicating the top and bottom may be displayed. This allows the user to intuitively understand the preferred standing position (drawing direction).

[0130] <4-2. Arrangement of Projector 210 and Camera 300>

[0131] In the above embodiment, the case where the projector 210 and the camera 300 are installed separately from each other has been described, but the present embodiment is not limited thereto. The projector 210 and the camera 300 may be installed so that the centers of their optical axes are as close to or coaxial as possible to each other, so that shielding of the projector 210 is regarded as shielding of the camera 300, and a guide display may be performed.

[0132] <<5. Conclusion>>

[0133] That is, in the case where the projector 210 and the camera 300 are installed so that the centers of their optical axes are as close to or coaxial as possible with each other, it is expected that the position of the shielding area 303 in the case where the user shields the imaging of the camera 300 is substantially the same as the position of the shielding area 213 in the case where the user shields the projection of the projector 210. Therefore, the information processing device 100 can regard the position of the user's shadow (the shielding area 213 of the projector 210) as the shielding area 303 of the camera 300, and in the case where IR light becomes unrecognizable near it, as shown in FIG. Fig. 9 As shown, a notification may be given that the user is an unidentifiable factor.

[0134] The preferred embodiments of the present disclosure are described in detail above with reference to the accompanying drawings, but the technical scope of the present disclosure is not limited to such embodiments. Obviously, a person of ordinary skill in the art of the present disclosure can derive various changes and modifications within the scope of the technical ideas described in the attached claims, and it should be understood that such changes and modifications naturally fall within the technical scope of the present disclosure.

[0135] Furthermore, a computer program may be created for causing hardware such as a CPU, ROM, and RAM built into the information processing device 100, the output device 200, the camera 300, or the operation tool 400 to exhibit the functions of the above-described information processing device 100, the output device 200, the camera 300, or the operation tool 400. Furthermore, a storage medium storing the computer program is also provided.

[0136] In addition, the effects described herein are only illustrative and exemplary, and not restrictive. That is, in addition to the above effects, or instead of the above effects, the technology according to the present disclosure may also play other roles that are obvious to those skilled in the art based on the description of this article.

[0137] It should be noted that the present technology can have the following configurations. (1)

[0139] An information processing device, comprising:

[0140] A controller that does:

[0141] recognition processing of recognizing a position of an operating tool within the projection area based on a captured image obtained by imaging the projection area,

[0142] an operation detection process of detecting that an operation is performed on the operation tool based on sensor data output by a sensor included in the operation tool, and

[0143] A projection control process, wherein, when an operation performed on the operating tool is detected by the operation detection process and the position of the operating tool is not identified by the recognition process, the projection control process causes the projection unit to project an image for guiding the operating tool to a position inside the projection area and at a position where the position of the operating tool can be identified by the recognition process. (2)

[0145] The information processing device according to (1), wherein the controller executes a projection control process that causes the projection section to project the image for the guidance at a predetermined position located at an edge of the projection area. (3)

[0147] The information processing device according to (1) or (2), wherein the controller:

[0148] identifying occluded regions that are occluded by occluding objects, and

[0149] A projection control process is performed that causes the projection section to project the image for the guidance at a position that is inside the projection area and at which the position of the operating tool can be identified by the identification process, the position not including the shielding area. (4)

[0151] The information processing device according to (3), wherein the shielding area includes, within the projection area, an area where projection cannot be performed due to a shielding object or an area where a position of the operating tool cannot be recognized due to the shielding object. (5)

[0153] An information processing device according to any one of (1) to (4), wherein the controller performs projection control processing, wherein the projection control processing causes the projection unit to project the image used for the guidance at a position located within the projection area and where the position of the operating tool can be identified by the recognition processing, and the position corresponds to the installation position of the imaging device that acquires the captured image or the projection device that performs the projection. (6)

[0155] An information processing device according to any one of (1) to (5), wherein the controller performs projection control processing, wherein the projection control processing causes the projection unit to project the image used for the guidance at a position that is within the projection area and at which the position of the operating tool can be identified by the recognition processing, and the position is the position at which the operating tool has been last identified. (7)

[0157] The information processing device according to (3) or (4), wherein the controller:

[0158] identifying the location of the obstructing object, and,

[0159] In a case where the controller determines, based on the position of the shielding object that has been identified, that the factor why the position of the operating tool cannot be identified is shielding caused by the shielding object, a projection control process is performed, wherein the projection control process causes the projection unit to project an image giving a notification of the factor as an image for the guidance. (8)

[0161] The information processing device according to (7), wherein the shielding object is a user who operates the operating tool. (9)

[0163] The information processing device according to (7) or (8), wherein the controller performs projection control processing, the projection control processing causing the projection unit to project an image corresponding to the position and size of the shielding area within the projection area as the image for the guidance. (10)

[0165] The information processing device according to (9), wherein the controller executes a projection control process that causes the projection section to project an image indicating a contour of a shielding area within the projection area as the image for the guidance. (11)

[0167] An information processing device according to any one of (1) to (10), wherein the controller performs projection control processing based on the positional relationship between the imaging device and the projection device relative to the projection area, and the projection control processing causes the projection unit to project an image indicating a vertical direction as an image for the guidance. (12)

[0169] The information processing device according to any one of (1) to (11), wherein the controller recognizes position coordinates of the operating tool within the projection area by detecting light emission performed by a light emitting section included in the operating tool from the captured image. (13)

[0171] The information processing device according to any one of (1) to (12), wherein the controller detects, based on the sensor data, that a user has performed a drawing operation using the operating tool on the projection area as the operation performed on the operating tool. (14)

[0173] The information processing device according to any one of (1) to (13), further comprising:

[0174] An interface outputs a control signal to the projection unit. (15)

[0176] An information processing method executed by a processor, the method comprising:

[0177] recognizing a position of an operating tool within the projection area based on a captured image obtained by imaging the projection area;

[0178] detecting that an operation is performed on the operating tool based on sensor data output by a sensor included in the operating tool; and

[0179] When an operation performed on the operating tool is detected and the position of the operating tool is not recognized, the projection section is caused to project an image for guiding the operating tool to the inside of the projection area and at a position where the position of the operating tool can be recognized by the recognition process. (16)

[0181] A program for causing a computer to function as a controller for executing the following processing,

[0182] a recognition process of recognizing a position of an operating tool within the projection area based on a captured image obtained by imaging the projection area;

[0183] an operation detection process for detecting that an operation has been performed on the operating tool based on sensor data output by a sensor included in the operating tool; and a projection control process for causing a projection unit to project an image for guiding the operating tool to a position inside the projection area and at which the position of the operating tool can be identified by the identification process, when an operation performed on the operating tool is detected by the operation detection process and the position of the operating tool is not identified by the identification process.

[0184] Reference Symbols List

[0185] 100 Information processing device

[0186] 110 I / F

[0187] 120 Luminous Identifier

[0188] 130 Memory

[0189] 140 Operation Detector

[0190] 150 Image Generator

[0191] 160 Coordinate Converter

[0192] 170 Display Controller

[0193] 180 User Location Identifier

[0194] 200 Output device

[0195] 210 Projector

[0196] 211 Projection Area

[0197] 213 Shielded Area

[0198] 220 TV

[0199] 230 Tablets

[0200] 240 Smartphone

[0201] 250 PC

[0202] 300 cameras

[0203] 301 Identification Zone

[0204] 303 Shielded Area

[0205] 304 Visible light images

[0206] 400 Operation Tools

[0207] 410 Communication Module

[0208] 421 Drawing Area

[0209] 422 (422a to 422g) Guide image

[0210] 430 IR_LED

[0211] 440 Switch

Claims

1. An information processing device, comprising: A controller that does: recognition processing of recognizing a position of an operating tool within the projection area based on a captured image obtained by imaging the projection area, an operation detection process for detecting that an operation has been performed on the operating tool based on sensor data output by a sensor included in the operating tool, and a projection control process for causing a projection unit to project an image for guiding the operating tool to a position that is within the projection area and that can be identified by the identification process, when the operation performed on the operating tool is detected by the operation detection process and the position of the operating tool is not identified by the identification process.

2. The information processing device according to claim 1, wherein: The controller performs a projection control process that causes the projection section to project an image for guidance to a predetermined position located at an edge of the projection area.

3. The information processing device according to claim 1, wherein: The controller: identifying occluded regions that are occluded by occluding objects, and A projection control process is performed that causes the projection section to project the image for guidance to a position inside the projection area and at which the position of the operating tool can be recognized by the recognition process, the position excluding the shielding area.

4. The information processing device according to claim 3, wherein: The shielding area includes, within the projection area, an area where projection cannot be performed due to a shielding object or an area where the position of the operating tool cannot be recognized due to the shielding object.

5. The information processing device according to claim 1, wherein: The controller performs projection control processing, which enables the projection unit to project the image for guidance into the projection area and the position of the operating tool to be identified by the recognition processing, and the position corresponds to the installation position of the imaging device that obtains the captured image or the projection device that performs the projection.

6. The information processing device according to claim 1, wherein: The controller executes a projection control process that causes the projection section to project an image for guidance inside the projection area and at a position where the position of the operating tool can be recognized by the recognition process, the position being the position where the operating tool has been recognized for the last time.

7. The information processing device according to claim 3, wherein: The controller: identifying the location of the occluding object, and, In a case where the controller determines, based on the position of the shielding object that has been identified, that the factor why the position of the operating tool cannot be identified is shielding caused by the shielding object, a projection control process is performed, wherein the projection control process causes the projection unit to project an image giving a notification of the factor as an image for guidance.

8. The information processing device according to claim 7, wherein: The shielding object is a user who operates the operating tool.

9. The information processing device according to claim 7, wherein: The controller executes a projection control process that causes the projection section to project an image corresponding to a position and a size of a shielding area inside the projection area as an image for guidance.

10. The information processing device according to claim 9, wherein: The controller executes a projection control process that causes the projection section to project an image indicating a contour of the shielding area inside the projection area as an image for guidance.

11. The information processing device according to claim 1, wherein: The controller performs a projection control process according to a positional relationship between an imaging device and a projection device with respect to the projection area, the projection control process causing the projection section to project an image indicating a vertical direction as an image for guidance.

12. The information processing device according to claim 1, wherein: The controller recognizes the position coordinates of the operating tool inside the projection area by detecting light emission performed by a light emitting section included in the operating tool from the captured image.

13. The information processing device according to claim 1, wherein: The controller detects, based on the sensor data, that a user has performed a drawing operation using the operating tool on the projection area as the operation performed on the operating tool.

14. The information processing device according to claim 1, further comprising: An interface outputs a control signal to the projection unit.

15. An information processing method executed by a processor, the method comprising: recognizing a position of an operating tool inside the projection area based on a captured image obtained by imaging the projection area; detecting that an operation is performed on the operating tool based on sensor data output by a sensor included in the operating tool; as well as When an operation performed on the operating tool is detected and the position of the operating tool is not recognized, the projection section is caused to project an image for guiding the operating tool to a position within the projection area and which can be recognized by the recognition process.

16. A storage medium storing a computer program for causing a computer to function as a controller for executing the following processing: a recognition process of recognizing a position of an operating tool within the projection area based on a captured image obtained by imaging the projection area; an operation detection process of detecting that an operation is performed on the operation tool based on sensor data output by a sensor included in the operation tool; as well as, A projection control process, in which, when an operation performed on the operating tool is detected by the operation detection process and the position of the operating tool is not recognized by the recognition process, the projection control process causes the projection unit to project an image for guiding the operating tool to the following position: the position is within the projection area and the position of the operating tool can be recognized by the recognition process.

Images