A device, computer program and method

Abstract

A device comprising: an image capturing unit configured to capture an image; and circuitry configured to: analyse the captured image to identify the presence of a human appendage in the captured image; authenticate the identified appendage; and in the event of a positive authentication, output a control signal based upon the pose of the authenticated appendage.

Description

[0001] A DEVICE, COMPUTER PROGRAM AND METHOD

[0002] BACKGROUND

[0003] Field of the Disclosure

[0004] The present technique relates to a device, computer program and method.

[0005] Description of the Related Art

[0006] The “background” description provided herein is for the purpose of generally presenting the context of the disclosure. Work of the presently named inventors, to the extent it is described in the background section, as well as aspects of the description which may not otherwise qualify as prior art at the time of filing, are neither expressly or impliedly admitted as prior art against the present technique.

[0007] Some recent headsets such as Virtual Reality and / or Augmented Reality headsets include a hand tracker that allows a user to interact with virtual reality applications or operate applications or other objects such as machinery or drones using their hands. Generally, these hand trackers use hand pose information to allow interaction between the real and virtual space.

[0008] A problem has been identified with this type of operation. In instances, it is possible for another person to place their hand in the field of view of the camera on the headset and take control of the interaction. This may cause an annoyance to the legitimate user of the headset in the case of a virtual reality application or may be dangerous in the case of operating an object such as machinery or a drone.

[0009] It is an aim of the disclosure to address this issue.

[0010] SUMMARY

[0011] According to embodiments of the disclosure, there is provided a device comprising: an image capturing unit configured to capture an image; and circuitry configured to: analyse the captured image to identify the presence of a human appendage in the captured image; authenticate the identified appendage; and in the event of a positive authentication, output a control signal based upon the pose of the authenticated appendage.

[0012] The foregoing paragraphs have been provided by way of general introduction, and are not intended to limit the scope of the following claims. The described embodiments, together with further advantages, will be best understood by reference to the following detailed description taken in conjunction with the accompanying drawings.

[0013] BRIEF DESCRIPTION OF THE DRAWINGS A more complete appreciation of the disclosure and many of the attendant advantages thereof will be readily obtained as the same becomes better understood by reference to the following detailed description when considered in connection with the accompanying drawings, wherein:

[0014] Figure 1 shows a device 100 according to embodiments of the disclosure;

[0015] Figures 2A and 2B show an example use case according to embodiments of the disclosure;

[0016] Figure 3 shows a database of hand poses and corresponding operations;

[0017] Figure 4 shows a flow chart S300 explaining the authentication process according to embodiments of the disclosure; and

[0018] Figure 5 shows a flow chart 500 explaining embodiments of the disclosure.

[0019] DESCRIPTION OF THE EMBODIMENTS

[0020] Referring now to the drawings, wherein like reference numerals designate identical or corresponding parts throughout the several views.

[0021] Numerous modifications and variations of the present disclosure are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims, the disclosure may be practiced otherwise than as specifically described herein.

[0022] Figure 1 shows a device 100 according to embodiments of the disclosure. The device 100 is, in embodiments, a headset such as virtual reality headset or an augmented reality headset or a mobile telephone or portable computer or the like. The device 100 includes a 110 which is circuitry configured to perform various functions according to embodiments. The device 100 also includes a camera 120, a storage 130, a screen 140 and a communication unit 150 connected to the 110.

[0023] In embodiments, the camera 120 is configured to capture an image. This image will, in embodiments, include a human appendage. A human appendage in embodiments may include one or more hands, fingers, one or more arms, legs, face or the like. The human appendage will include one or more physiological or anatomical characteristics that are unique to the appendage. The one or more physiological or anatomical characteristics can therefore be used to authenticate the person.

[0024] In embodiments, a multispectral sensor (not shown) may be provided in the device 100. The multispectral sensor may be used in conjunction with or instead of the camera 120 in order to authenticate the person. The multispectral sensor is configured to authenticate the person using known skin spectrum authentication techniques. Accordingly, the multispectral sensor may be used in conjunction with the captured image of the human appendage or as an alternative to the captured image to authenticate the person.

[0025] The storage 130 is, in embodiments, storage configured to store a computer program that contains computer readable instructions which, when run on the processing circuitry 110, configures the processing circuitry 110 to perform a method according to embodiments. In addition, the storage 130 may include a secure area that is configured to store an authentication model of an authorised human appendage. In other words, the secure area of the storage 130 stores the model used to authorise a human appendage captured in an image. By placing the model of the authorised human appendage in a secure area, the model cannot be tampered with by nefarious individuals.

[0026] In embodiments, the communication unit 150 connects the device 100 to one or more of a network and / or other apparatus. The communication unit 150 connects the device 100 to the network and / or other apparatus using either a wired or wireless connection. The device 100 may therefore communicate with a server (not shown) over the network to balance processing load.

[0027] In embodiments, the screen 140 is viewed by an authorised user. In embodiment, the screen 140 may be used to view the image captured by the camera 120 or may be semi-transparent allowing the user to see the real-life surroundings with information being overlaid onto the real-life surroundings by control signals generated by the processing circuitry 110 thus providing an augmented reality view.

[0028] In embodiments the processing circuitry 110 may generate the control signals which displays a virtual reality view on the screen 140.

[0029] In embodiments, the device 100 is configured to control a different apparatus to which the device 100 is connected via a direct wired or wireless link or over a network such as the Internet using the communication unit 150. For example, the device 100 may be used to control the operation of a different apparatus such as a drone, robot, vehicle or surgical system for example. In this case, the screen 140 may display an image captured by the different apparatus to allow the user to control the different apparatus. The control signals generated by the processing circuitry 110 are sent to the different apparatus using the communication unit 150.

[0030] In operation, the processing circuitry 110 generates the control signal or control signals in response to the pose or a gesture provided by the human appendage. For example, in the case of the human appendage being a hand, the control signal is generated by a gesture or pose made by an authorised user of the device 100.

[0031] Embodiments of the disclosure are described with reference to an example use case shown in Figure 2A and Figure 2B. Figure 2A shows a user 200 wearing the device 100. The user 200 is controlling a flying drone 210. Figure 2B shows the screen 140. In this case, the screen 140 in the device 100 is semitransparent allowing the user to see the flying drone 210. Moreover, the camera 120 is forward facing (i.e. is facing the same direction as the user is facing).

[0032] In order to control the operation of the drone, the user performs various hand poses. A non-exhaustive list of hand poses and associated controls is provided in Figure 3. These hand poses and associated controls are stored in storage 130. Although Figure 3 shows a list of hand poses, of course, in the event of the human appendage being a different part of the human anatomy such as a leg, the poses associated with each control operation will vary. In instances, more than one human appendage may be used to control the operation of the drone 210. For example, a combination of the same or different human appendages may be used in concert to perform a larger number of operations. Similarly, the poses may be predefined or may be selected by a user. In embodiments, if the user has an impairment (such as a medical condition like arthritis), the poses may be customised to allow the user to perform all the desired operations.

[0033] Referring to Figure 3, a database of hand poses and the corresponding operations is shown. A pointing finger pose 230 is shown. In embodiments, the pointing finger pose 230 is used to indicate a direction of movement. In other words, once the pointing finger pose 230 is recognised by the device 100, the pointing finger will be tracked and the direction of movement of the pointing finger will be used by the processing circuitry 110 in the device 100 to output a control signal to the drone to move the drone in the direction of movement of the pointing finger.

[0034] A cupped hand pose 240 is also shown. In embodiments, the cupped hand pose 240 is an example of where two human appendages are used in concert to define an operation. The cupped hand pose 240 is used by the processing circuitry 110 in the device 100 to generate a control signal instructing the drone 100 to land.

[0035] An open hand pose 250 is shown. In embodiments, the open hand pose 250 is used so that when captured by camera 120 and detected by the processing circuitry 110 in the device 100, the processing circuitry 110 outputs a control signal to the drone to capture an image using its on-board camera (not shown). In other words, when an open hand pose is detected in the image captured by the camera 120 of the device 100, a control signal is output by the processing circuitry 110 and is sent to the drone using the communication unit 150. The drone will then capture an image using its on-board camera.

[0036] A closed fist pose 260 is shown. In embodiments, the closed hand pose 260 is used to alter the flight speed of the drone. In particular, if the closed hand pose 260 is detected by the device 100 in the captured image and is moved upward, the flight speed of the drone is increased and if the closed hand pose 260 is detected by the device 100 and is moved downward, the flight speed of the drone is decreased. As is described above, the processing circuitry 110 of the device outputs one or more control signal to the drone using the communication unit 150 and the drone uses the control signal to adjust its speed.

[0037] Referring back to Figure 2, the control of the drone using the pose of the hand is described. The user 200, in embodiments, wears the device 100 on their head. Of course, the device 100 may be embodied as a hand-held device such as a phone or tablet and so is not limited to a head-mounted device. The user then brings their hand into the field of view of the camera 120 mounted on the device 100. In embodiments, the processing circuitry 110 of the device 100 detects the presence of a hand in the image captured by the camera 120 and identifies the hand.

[0038] In order to reduce the possibility of a third party controlling the flight of the drone, prior to the processing circuitry 110 outputting control signals to the drone 210, the processing circuitry 110 authenticates the identified hand as being a hand that is authorised to control the drone. This authentication will be explained later but may or may not involve the user performing a different pose to the controlling pose. In embodiments, the authentication may involve the use of a multispectral sensor on the device 100 (not shown) which allows the hand to be authenticated using a known skin spectrum authentication method.

[0039] In the event of a positive authentication, any hand pose in the image will be compared with the stored hand poses as explained in Figure 3 and the processing circuitry 110 will output a control signal based upon the pose of the authenticated hand. In the example shown in Figure 2A and 2B, the hand pose 230A in the first image (the pointing finger) is recognised as an operation to move the drone in the direction of movement of the pointing finger. Therefore, when detected and authenticated hand performing the pointing finger hand pose 230B moves downward 220’ in a second captured image, the processing circuitry 110 outputs one or more control signal to the drone 210 and the drone 210 moves downwards 220.

[0040] It is noted that whilst the above image processing and authentication may be performed within the device 100, in embodiments, the image processing to detect and / or authenticate the user’s hand (or other human appendage) may be carried out on a server located on a network or Internet. In this case, the image captured by the camera 120 is uploaded to the server via the communication unit 150. Indeed, in embodiments, some image processing may be carried out on the device 100 and some on a server. For example, in the event of a poor communication link between the device 100 and the server, the processing circuitry 110 may perform the hand detection and authentication. In other instance, where there is a good communication link between the device 100 and the server for example, the device 100 may upload the captured image to the server and the server may perform all of the authentication and detection and may instruct the processing circuitry 110 to output the control signal to the drone 210. Of course, embodiments are envisaged where the device 100 uploads the image to the server, the server may detect the location of the human appendage and provide this to the device 100. The device 100 may then perform the authentication and pose comparison.

[0041] In embodiments, the authentication of the human appendage may occur for every frame of video. In other words, and in embodiments, where the camera 120 is capturing a video, the authentication will occur on each frame of video. Of course, the disclosure is not so limited and it is envisaged that once the human appendage has been authenticated as being an authorised user, authentication may occur only periodically. For example, the authentication may occur every predetermined number of frames or after a predetermined time. The authentication may occur again, in embodiments, after the previously authenticated human appendage is removed from the field of the view of the camera 120 and then reintroduced back into the field of view. In other words, whilst the authenticated appendage remains in consecutive images, and is thus tracked, no second authentication event is required. However, if the authenticated appendage leaves the field of view, and so tracking is stopped, re -authentication is be required. This reduces the processing on the system.

[0042] In the event that a hand (or other human appendage) is captured by the camera 120 and is not authorised to control the drone 210, the hand will fail the authentication. In the event that a hand that has failed the authentication is located in a captured image, according to embodiments, the device 100 will ignore any poses performed by the unauthorised hand and so the processing circuitry 110 will not output any control signals in response to these poses. In embodiments, the user of the device 100 may be alerted to the presence of an unauthorised hand in the captured image using an audible or visual indication such as an audible alarm, vibration of the device 100 or a coloured light appearing on the screen 140.

[0043] In instances there may be more than one human appendage captured in an image by the camera 120. In the event that none of the captured appendages are authorised, none of the poses performed by the unauthorised appendages will be used to output the control signal or signals.

[0044] However, the captured image may include one appendage that is authorised and one or more appendage that is not authorised. In this instance, in embodiments, the pose of the authorised appendage is used to generate and output the control signal or signals. In embodiments, as mentioned above, the unauthorised appendage is not used to generate and output the control signal or signals. However, in embodiments, the unauthorised appendage is highlighted on the screen 140 so that the user of the device 100 can move the field of view of the camera 120 or move the appendage to avoid the unauthorised appendage being in the field of view of the camera 120.

[0045] In order to authenticate the human appendage such as a hand, a known biometric identification and authentication process is used. One example system which relates to authentication of hands is described in [1], the contents of which is hereby incorporated by reference. However, other known types of biometric authentication techniques for the hand or other human appendage are envisaged.

[0046] Figure 4 shows a flow chart S300 explaining the authentication process according to embodiments of the disclosure. The process starts in step S310. The process moves to step S315 where an image is captured by the camera 120. The process moves to step S320 where hand detection is performed on the image. In the case that no hand is detected, the process waits until a hand is detected.

[0047] The process moves to step S325 and S330. In step S325, a stored personalised 3D model of the human appendage (such as the hand) is retrieved from either the storage 130 or from a server (not shown) as appropriate. This 3D model is of the authorised human appendage and is generated and stored prior to the commencement of the process S300. In order to generate the 3D model, the authorised human appendage is moved into a predetermined number of poses and an image captured at each pose. This is a known technique and will not be described hereinafter.

[0048] Returning to step S330, in embodiments, the detected hand is cropped from the image. In this case, the authentication is carried out on the cropped detected hand (the cropped image) to reduce the amount of processing required compared to performing the authentication on the entire image. The detected hand cropped from the image is fed to step S335 and S350.

[0049] In step S335, the pose of the detected hand is determined using known techniques. In other words, the pose of the hand captured in the image is determined. One example of such a known technique is described in [2]; the contents of which is hereby incorporated by reference.

[0050] This pose is fed into step S340 where the pose of the hand captured in the image is applied to the 3D model of the authorised human hand. In other words, the 3D model of the authorised human hand is changed to replicate the pose performed by the hand captured in the image. This generates a posed 3D model of the authorised human hand.

[0051] The posed 3D model of the authorised human hand is fed into step 345 where the posed 3D model is rendered as a 2D image.

[0052] The rendered 2D image is fed into step S350 where the rendered 2D image is compared with the detected hand cropped from the captured image. In the event of a positive comparison (i.e. where the rendered 2D image is similar or the same as the detected hand cropped from the captured image), the captured hand is authorised and the pose determined in step S335 is used to determine operation desired by the user. In the event of a negative comparison (i.e. where the rendered 2D image is not similar or the same as the detected hand cropped from the captured image), the captured hand is not authorised and the detected hand is ignored and in embodiments a notification is issued to the user as explained above.

[0053] The process then ends at step S355.

[0054] Although the foregoing describes controlling a different object (such as a drone), the disclosure is not so limited. In embodiments, sign language poses are used and an associated control signal is generated which controls a display to show a transcription of the poses into text.

[0055] Figure 5 shows a flow chart 500 explaining embodiments of the disclosure. The process starts at step 510. The process moves to step 520 where an image is captured. The process moves to step 530 where the captured image is analysed to identify the presence of a human appendage in the captured image. The process moves to step 540 where the identified appendage is authenticated and in the event of a positive authentication, the process moves to step 550 where a control signal is output based upon the pose of the authenticated appendage. The process ends in step 560. In so far as embodiments of the disclosure have been described as being implemented, at least in part, by software-controlled data processing apparatus, it will be appreciated that a non-transitory machine- readable medium carrying such software, such as an optical disk, a magnetic disk, semiconductor memory or the like, is also considered to represent an embodiment of the present disclosure.

[0056] It will be appreciated that the above description for clarity has described embodiments with reference to different functional units, circuitry and / or processors. However, it will be apparent that any suitable distribution of functionality between different functional units, circuitry and / or processors may be used without detracting from the embodiments.

[0057] Described embodiments may be implemented in any suitable form including hardware, software, firmware or any combination of these. Described embodiments may optionally be implemented at least partly as computer software running on one or more data processors and / or digital signal processors. The elements and components of any embodiment may be physically, functionally and logically implemented in any suitable way. Indeed the functionality may be implemented in a single unit, in a plurality of units or as part of other functional units. As such, the disclosed embodiments may be implemented in a single unit or may be physically and functionally distributed between different units, circuitry and / or processors.

[0058] Although the present disclosure has been described in connection with some embodiments, it is not intended to be limited to the specific form set forth herein. Additionally, although a feature may appear to be described in connection with particular embodiments, one skilled in the art would recognize that various features of the described embodiments may be combined in any manner suitable to implement the technique.

[0059] Embodiments of the present technique can generally described by the following numbered clauses:

[0060] 1. A device comprising: an image capturing unit configured to capture an image; and circuitry configured to: analyse the captured image to identify the presence of a human appendage in the captured image; authenticate the identified appendage; and in the event of a positive authentication, output a control signal based upon the pose of the authenticated appendage.

[0061] 2. A device according to clause 1, wherein the circuitry is configured to: generate an alert for the user in the event of a negative comparison.

[0062] 3. A device according to clause 2, wherein the circuitry is configured to generate a visual or audible alert for the user. 4. A device according to any preceding clause, wherein the circuitry is configured to crop the human appendage from the captured image and authenticate the identified appendage from the cropped image.

[0063] 5. A device according to any preceding clause, further comprising a multispectral sensor, wherein the circuitry is configured to perform authentication based upon skin spectrum of the human appendage captured by the multispectral sensor.

[0064] 6. A device according to any preceding clause, further comprising a communication unit, wherein the communication unit is configured to output the control signal to an external object.

[0065] 7. A device comprising: an image capturing unit configured to capture a video; and circuitry configured to: analyse a frame in the captured video to identify the presence of a human appendage in the captured frame; authenticate the identified appendage; and in the event of a positive authentication, output a control signal based upon the pose of the authenticated appendage in the captured frame; track the authenticated appendage in a consecutive frame, and output the control signal based upon the pose of the tracked appendage in the consecutive frame.

[0066] 8. A method comprising: capturing an image; analysing the captured image to identify the presence of a human appendage in the captured image; authenticating the identified appendage; and in the event of a positive authentication, outputting a control signal based upon the pose of the authenticated appendage.

[0067] 9. A method according to clause 8, comprising: generating an alert for the user in the event of a negative comparison.

[0068] 10. A method according to clause 9, comprising generating a visual or audible alert for the user.

[0069] 11. A method according to any one of clauses 8 to 10, comprising cropping the human appendage from the captured image and authenticating the identified appendage from the cropped image. 12. A method according to any one of clauses 8 to 11, further comprising performing authentication using a multispectral sensor based upon skin spectrum of the human appendage captured by the multispectral sensor.

[0070] 13. A method according to any one of clauses 8 to 12, further comprising outputting the control signal to an external object.

[0071] 14. A method comprising: capturing a video; analysing a frame in the captured video to identify the presence of a human appendage in the captured frame; authenticating the identified appendage; and in the event of a positive authentication, output a control signal based upon the pose of the authenticated appendage in the captured frame; tracking the authenticated appendage in a consecutive frame, and outputting the control signal based upon the pose of the tracked appendage in the consecutive frame.

[0072] 15. A computer program comprising computer readable instructions which, when loaded onto a computer, configures the computer to perform a method according to any one of clauses 8 to 14.

[0073] References

[0074]

[0001] 1 IK Hands: Gender Recognition and Biometric Identification Using a Large Dataset of Hand

[0075] Images. AFIFI, Mahmoud 1711.04322 (arxiv.org)

[0076] [2] MANUS: Markerless Grasp Capture using Articulated 3D Gaussians, POKHARIYA et al 2312.02137 (arxiv.org)

Claims

CLAIMS1. A device comprising: an image capturing unit configured to capture an image; and circuitry configured to: analyse the captured image to identify the presence of a human appendage in the captured image; authenticate the identified appendage; and in the event of a positive authentication, output a control signal based upon the pose of the authenticated appendage.

2. A device according to claim 1, wherein the circuitry is configured to: generate an alert for the user in the event of a negative comparison.

3. A device according to claim 2, wherein the circuitry is configured to generate a visual or audible alert for the user.

4. A device according to claim 1, wherein the circuitry is configured to crop the human appendage from the captured image and authenticate the identified appendage from the cropped image.

5. A device according to claim 1, further comprising a multispectral sensor, wherein the circuitry is configured to perform authentication based upon skin spectrum of the human appendage captured by the multispectral sensor.

6. A device according to claim 1, further comprising a communication unit, wherein the communication unit is configured to output the control signal to an external object.

7. A device comprising: an image capturing unit configured to capture a video; and circuitry configured to: analyse a frame in the captured video to identify the presence of a human appendage in the captured frame; authenticate the identified appendage; and in the event of a positive authentication, output a control signal based upon the pose of the authenticated appendage in the captured frame; track the authenticated appendage in a consecutive frame, and output the control signal based upon the pose of the tracked appendage in the consecutive frame.

8. A method comprising: capturing an image; analysing the captured image to identify the presence of a human appendage in the captured image;authenticating the identified appendage; and in the event of a positive authentication, outputting a control signal based upon the pose of the authenticated appendage.

9. A method according to claim 8, comprising: generating an alert for the user in the event of a negative comparison.

10. A method according to claim 9, comprising generating a visual or audible alert for the user.

11. A method according to claim 8, comprising cropping the human appendage from the captured image and authenticating the identified appendage from the cropped image.

12. A method according to claim 8, further comprising performing authentication using a multispectral sensor based upon skin spectrum of the human appendage captured by the multispectral sensor.

13. A method according to claim 8, further comprising outputting the control signal to an external object.

14. A method comprising: capturing a video; analysing a frame in the captured video to identify the presence of a human appendage in the captured frame; authenticating the identified appendage; and in the event of a positive authentication, output a control signal based upon the pose of the authenticated appendage in the captured frame; tracking the authenticated appendage in a consecutive frame, and outputting the control signal based upon the pose of the tracked appendage in the consecutive frame.

15. A computer program comprising computer readable instructions which, when loaded onto a computer, configures the computer to perform a method according to claim 8.

Images