Method and apparatus for operating a device by a member of a group

Abstract

A method of operating a device by controlling the device based on input received from group members uses a sensor unit for monitoring each group member for detecting an instruction provided by a group member. The instruction comprises a visual or an audible instruction. Upon detection, a control unit associates a control command with the instruction. The control unit provides the control command to the device. To establish the group, the method comprises performing an authentication by a user input unit for interacting with a user for adding the user as a group member. To identify the control command, the method comprises associating the detected instruction with an instruction group member among the group members, accessing a member profile comprising a set of reference instructions from a memory unit, and matching the instruction with a reference instruction selected from the set of reference instructions in the profile. An apparatus is arranged for performing the method.

Description

Technical Field

[0001] In general, the present invention relates to a method for operating a device based on input received from one or more group members, the method comprising the steps of: monitoring each group member using a sensor unit to detect at least one instruction provided by an instruction member among the one or more group members, the instruction including at least one of visual or auditory instructions; identifying a control command associated with the instruction by a control unit upon detection of the instruction; and providing the control command to the device by the control unit for controlling the device. The invention further relates to apparatus and computer program products for performing such a method. Background Technology

[0002] Methods and systems for operating devices by means of gestures or spoken voice are available in various variations and domains. Early implementations of such systems simply relied on motion detection by motion sensors to operate devices, such as lights, cameras, or alarm systems. More sophisticated implementations are able to detect specific motion patterns, enabling users to aim and operate the device with specific gestures or to perform operations on the device that are less sensitive to any kind of motion in front of the sensors.

[0003] Face detection and recognition in images and videos are becoming standard features / capabilities in many systems, such as Google's Picasa or Windows Live. Other devices for tracking the presence and location of people in enclosed environments are also present and being used, for example, RFID-based solutions that can continuously track people wearing RFID tags using long-range detection and triangulation techniques.

[0004] European Patent No. EP0919906 discloses a control method and system that allows input in the form of gestures to control a device. The method includes an operator selection section that identifies attributes of the person making the gesture. Additional information about the operator's attributes is used to prioritize some operators over others.

[0005] Published U.S. patent application US2008 / 0004951 discloses an architecture for presenting advertisements in real time in a retail setting. The sensor component includes sensors for collecting information about one or more customers as they pass through the store. The sensors may include capabilities for, for example, image processing, audio processing, light sensing, speed sensing, orientation sensing, proximity sensing, facial recognition, gesture recognition, transaction recognition, and biometric sensing. The customer component analyzes the information and generates a profile of the customer. Advertisements targeting the customer are selected and presented as the customer approaches the presentation system within the store. The advertising component facilitates the dynamic presentation of the targeted advertisements to the individual based on the profile. The customer component may use machine learning and inference to infer information during analysis.

[0006] In medical settings (such as a hospital operating room), groups of personnel can be assigned to perform specific team tasks supported by medical equipment. For example, in a surgical operating room, several people (the entire surgical team) work together to perform surgery collaboratively. The surgery is typically directed by the surgeon, who is also the primary team member. The surgeon relies on the anesthesiologist to administer anesthesia and on other medical personnel to perform other tasks. The surgical team members collectively control several medical systems. These systems are generally operated via physical buttons or switches. Generally, the primary surgeon (one or more) cannot operate the medical systems themselves (either because their hands are busy with the patient, or because their hands are sterile while the parts they need to touch are not) and requests operation from other medical personnel.

[0007] In “HCI Design in the OR: A Gesturing Case-Study”, a computerized system for controlling medical equipment in the operating room by means of a gesture-based interface, authored by Ali Bigdelou et al. in the October 2012 issue of Technische Universitat Munchen and Klinikum Munchen-Pasing.

[0008] The document “Voice or Gesture in the Operating Room” in the April 18, 2015 issue of Suite 701, New York, NY10121-0701, USA, 2 Penn Plaza, presents another such system that allows surgeons to choose between gestures or voice control. Summary of the Invention

[0009] The objective of this invention is addressed by the independent claims. Advantageous embodiments are defined in the dependent claims.

[0010] The present invention aims to provide a device control method based on visual or auditory commands, which can be implemented for operation by a team of operators (such as a surgical team) jointly performing specific tasks.

[0011] The present invention achieves these and other objectives by providing a method for operating a device based on input control devices received from one or more group members. The method includes: monitoring each group member using a sensor unit to detect at least one instruction provided by an instruction member among the one or more group members. The instruction may include at least one of visual or auditory instructions. The method further includes a control unit identifying a control command for controlling the device, the control command being associated with the instruction, upon detecting the instruction. The control unit provides the control command to the device. To establish the group, the method further includes: performing authentication of the user via a user input device for user interaction to add the user as a group member. To identify the control command, the method includes: associating the detected instruction with the instruction member among the group members by the control unit; accessing member profile data of the instruction member from a memory unit, the member profile data of the instruction member including a reference instruction set; and matching the detected instruction with a given reference instruction selected from the reference instruction set in the member profile data of the instruction member to identify the control command.

[0012] As can be recognized, the use of gesture-based device control has the potential to enable surgeons to have direct control over the medical system in situations that would not be possible in conventional settings. For example, it can be preferred when performing specific actions such as (time-critical) actions, due to its reduced response time and lower risk of error (it should be recognized that verbal communication between team members can be error-prone, especially in the presence of background noise). This invention is based on the insight that conventional gesture-based control methods are unsuitable for implementation in critical medical settings such as surgery. These conventional methods often lack sufficient reliability. Furthermore, these methods do not take into account the roles and responsibilities of team members. For example, in the operating room, the roles and responsibilities of the entire surgical team are clearly defined, such as actions to prevent conflict and the optimal application of said members' skills.

[0013] The terms visual and auditory instructions should be understood to include any instruction that can be detected using sensor units capable of detecting changes over time in a specific area. These instructions include gestures, verbal instructions, sound instructions (e.g., clapping), and postures adopted by group members (e.g., temporarily holding hands in a fixed position). These instructions can also be combinations of the above, such as the detection of sounds accompanying corresponding movements of group members. For example, clapping accompanied by the detection of specific group members' hands moving towards each other, or a voice command including the detection of lip movement. Other examples are gestures that include additional voice commands, such as raising both hands in a crossed manner while uttering the sound of "complete stop," or making a lifting motion with one hand while uttering the name of a parameter to be increased (e.g., raising one hand while saying "adrenaline" and "slowly").

[0014] The method of the present invention performs authentication before adding group members to the group. The group of personnel who can control the device is therefore clearly defined for each group task. For example, each person (user) participating in performing the group task must first join the group by completing the authentication session. Without authentication, personnel are ignored by the system performing the method. After authentication, the system monitors the authenticated group members because each group member is monitored. In some embodiments, authentication may occur at the start of the group task (e.g., before the start of the surgery). In other embodiments, it may also be possible to add group members at any time from the start (e.g., later during the surgery to allow additional skills to be added to the entire surgical team if these skills are unexpectedly needed).

[0015] In addition to using authentication to identify group members, this invention also takes into account the roles and responsibilities of the group members. This is achieved by the control unit associating the detected instruction with an instruction member in the group to identify the control command, and then matching it with a reference instruction in the instruction member's member profile. Furthermore, the control unit accesses the instruction member's member profile data from a memory unit.

[0016] This has several advantages. As illustrated, by verifying instructions relative to member profiles, no group member can provide instructions that he or she is not authorized to provide. For example, in a particular embodiment, the identification of control commands may further include verification of the authorization status obtained from the member profile data of the instruction group member. Other embodiments may associate member profile data with categories, such as based on defined functional groups corresponding to the roles of group members in the group. Thus, the authorization status can be derived based on such a category.

[0017] Furthermore, in certain embodiments, the use of reference instructions can also allow for the personalization of gestures and voice commands used by each group member. Even in embodiments that rely on standardized instructions (such as standardized gestures, voice commands, sounds, or postures), the meaning of such instructions can be personalized. This allows for a full consideration of the roles and responsibilities of group members.

[0018] In some embodiments, the monitoring step includes continuously tracking each of the group members once the corresponding group member has been authenticated. Continuous tracking of each member within the corresponding group after authentication prevents errors in member identification. Once authentication is performed, the presence of each group member in the image received from the sensor unit can be continuously tagged, for example, to enable the association of detected instructions with the corresponding group member. Furthermore, it prevents instructions provided by group members from being missed by the system executing the method.

[0019] In some embodiments, availability of detailed information about which team member provided which detected instruction further allows for the automatic and electronic tracking of group tasks to be performed. All instructions can be logged, for example, in a log file for later evaluation and training purposes.

[0020] The method of the present invention is not limited to the control of a single device, but can be implemented to control multiple devices according to one embodiment. According to such an embodiment, the control unit may be included in a central unit that receives sensor signals from the sensor unit, and a memory unit is accessible to the control unit. The method according to this embodiment may include the step of: identifying candidate devices for receiving control commands by the control unit, wherein the candidate devices are identified by matching the detected instructions with reference instructions in the member profile data of the instruction group members.

[0021] According to another aspect, an apparatus is provided for controlling the operation of a device via an input control device received from one or more group members of a group, the apparatus comprising: a sensor unit for monitoring each group member to detect at least one instruction provided by an instruction member among the one or more group members, the sensor unit being arranged to receive at least one of a visual image or an auditory signal, such as for detecting a visual or auditory instruction; and a control unit communicatively connected to the sensor unit for receiving a signal output from the sensor unit, the control unit including a processor for identifying a control command for controlling the device upon detecting the instruction, the control command being associated with the instruction, wherein the control unit is communicatively connected to the device. The control unit is configured to provide the control command to the device; a user input device for interacting with a user to perform authentication of the user in order to add the user as a group member to establish the group; and a memory unit for storing one or more member profile data of the group members, each member profile data including a reference instruction set for the group member; wherein the control unit is further arranged by the processor to associate a detected instruction with the instruction group member in the group; access the member profile data of the instruction group member from the memory unit; and match the detected instruction with a given reference instruction selected from the reference instruction set in the member profile data of the instruction group member to identify the control command.

[0022] In one embodiment, the user input device may include at least one element from the group comprising: a biometric scanner, including an iris scanner or retinal scanner or a fingerprint scanner for authenticating the user based on individual eye characteristics; and a keyboard, a touch-sensitive screen or a camera for entering personal login details.

[0023] Furthermore, according to some embodiments, the sensor unit includes at least one element from a group comprising: an imaging device or imaging arrangement including one or more cameras for observing an area; and an audio sensor device or audio sensor arrangement including one or more microphones for sensing audio signals.

[0024] According to a third aspect, the present invention relates to a computer program product downloadable from a communication network and / or stored on a computer-readable and / or microprocessor-executable medium, the product comprising program code instructions for implementing the method according to a first aspect of the invention for controlling a device by means of data received from one or more group members of a group. Furthermore, according to another aspect, the present invention relates to a data carrier comprising such a computer program product. Attached Figure Description

[0025] The invention will be described below with reference to the accompanying drawings and some specific embodiments thereof, wherein: Figure 1 The illustration shows the configuration of the present invention; Figure 2 The method of the present invention is illustrated schematically; Figure 3 The system according to the present invention is illustrated schematically; Figure 4 The method of the present invention is illustrated schematically. Detailed Implementation

[0026] Where references to "gestures" are made in the description, it should be understood that any other type of instruction mentioned anywhere in this document may therefore be replaced, such as verbal instructions, sound instructions (e.g., clapping), gestures taken by group members (e.g., temporarily holding a hand in a fixed position), or any combination of such instructions. Furthermore, the term authentication should be interpreted as any action or method that allows for the reliable establishment of the identity of a user or person. Of course, the desired level of reliability may vary depending on the implementation. For some implementations, identification by means of access badges may be sufficient, while for others, fingerprints or retinal scanning are preferred.

[0027] exist Figure 1 The diagram schematically illustrates the arrangement in which the invention is applied. Figure 1 In this process, multiple group members 1, 2, and 3 can operate device 6 while performing tasks. Figure 1This is merely illustrative and illustrates an office environment in which colleagues 1, 2, and 3 use some equipment (such as fax equipment 6). Team members 1, 2, and 3 may have different roles and responsibilities within the team, and correspondingly, instructions given by any one of team members 1, 2, or 3 may have different priorities for being followed by equipment 6. A tracking system 8 (e.g., one or more cameras) can monitor the area where team members 1, 2, and 3 are working. The tracking system 8 enables monitoring of each team member, such as using gestures or sounds made by any team member to identify them. As indicated by double arrows 9, 10, and 11 respectively, each team member 1, 2, and 3 can provide input and receive feedback to equipment 6. As will be further explained below, the tracking system 8 can be communicatively connected to a control unit (not shown) that receives information about whether an instruction has been given by one of team members 1, 2, or 3.

[0028] Before being added as a group member, an authentication session may be required for any of the group members 1, 2, or 3 to be added. After authentication, the control unit can access a personal profile or member profile from the member unit. Using the member profile, conversations 9, 10, and 11 between any of the group members 1, 2, or 3 and device 6 can be personalized. Personalization of conversations can include each of the group members 1, 2, and 3 using personalized gestures as input to device 6. In other words, gestures can have different meanings depending on the group member (1, 2, or 3) providing the instruction. Furthermore, any feedback from device 6 to the instructing group member (1-3) in conversations 9, 10, and 11 can also be personalized. As will be appreciated, personalization can also include each group member 1, 2, and 3 having their own personalized set of actions (for which the group member is authorized). Specific functions of device 6 may be accessible only to group member 1, while other functions may be available to all group members.

[0029] Tracking system 8 can be a human tracking system based on existing tracking functions (such as face detection and recognition, RFID tags, etc.). (Reference) Figure 2 First, in step 19, each person to be tracked (1, 2, or 3) is uniquely identified during the authentication session using a reliable mechanism (such as retinal scanning, fingerprint recognition, access badges, etc.). Note that such a mechanism can be used continuously for tracking. In many cases, authentication prepares for momentary identification and therefore requires additional devices for maintaining tracking of group members. Therefore, a combination of an identification mechanism used one-time or intermittently with a continuous tracking mechanism (step 23) is preferred. In some embodiments, such as in Figure 2As indicated by arrow 16, authentication session 19 can obtain personal information from member profiles stored in member data set 15 (e.g., available from a memory unit). Once authentication is complete, the identified person is added as a group member. As indicated by arrow 20, some (or even all) of the member profile data available in member data set 15 can be provided to the system for further use. Alternatively, a reference to the member profile data in the memory unit can be provided. This can also, for example, enable tagging of persons tracked by tracking system 8 to associate any instruction (see description of events below) with the member profile of the person providing the instruction (now a group member).

[0030] During the tracking process 23, each person being tracked interacts with the system (e.g., via gestures, spoken words, postures, etc.) and triggers an event 25. An event may include one or more, or all, of the following: identity, location, position, timestamp, and input instructions (gestures, postures, voice commands, etc.). Event 25 makes the control unit available for further processing. The software controlling the control unit treats this event as input to step 26. A second input 27 for controlling the system is obtained from a member data set 15 containing all identities and their roles, permissions, and known gestures and personal preferences. Based on the two inputs 25 and 27, the control unit in step 26 makes a decision on the appropriate action for the device 6 to be controlled and provides sufficient feedback to the group member who issued the instruction. Because the system knows who issued the command (using input 25) and what their preferences are (based on member profile data 27 obtained from member data set 15), this feedback can be personalized. This allows for a highly personalized two-way dialogue between each individual and the complex system (and in parallel for multiple individuals). Based on the result of step 26, control command 29 is provided to device 6, and device 6 causes the required action to be executed in step 30.

[0031] exist Figure 3 The diagram illustrates another example of a system 35 relating to a medical device according to an embodiment of the present invention. Figure 3The diagram illustrates an operating room including an operating table 37 on which surgery will be performed on patient 38. Around the table 37, multiple surgical team members 39, 40, 41, 42, 43, 44, and 45 are schematically indicated. Each of the surgical team members 39-45 has a different role and responsibility within the team. For example, surgeon 39 may be the head of the team leading the surgery. Surgeon 39 is assisted by assistant surgeon 42 and cardiologist 43. An anesthesiologist 41 administers anesthesia while continuously monitoring patient 38's vital signs. Further appearing are surgical assistants 40 and 44 and other medical practitioners 45 on standby. Monitoring displays 54 and 55 are strategically placed in the operating room and can be used by the surgical team to provide information or images, etc., to support the surgery. Lighting in the operating room is provided by lighting unit 56. Additionally, the medical support equipment consists of devices 48, 50, and 52 performing different functions.

[0032] The operating room is equipped with several cameras 60A, 60B, 60C, and 60D that together form a tracking system 60. As will be recognized, although in Figure 3 The example illustrates four cameras, but the number of cameras or other sensing units required to enable the tracking of group members 39-45 can vary for each case. For example, in some embodiments, a single sensing unit or camera may be sufficient, while in other embodiments, several different rooms interconnected by doors and corridors may be equipped with twenty or thirty cameras together to form the tracking system. Furthermore, in place of cameras or other sensor units, other sensor units may be applied, such as microphones, RFID scanners, thermal sensors, or any combination of suitable sensors for tracking people that may be part of the tracking system 60.

[0033] exist Figure 3 In this embodiment, all the controlled devices 48, 50, 52, 54, 55, and 56 and cameras 60A-60D are interconnected via a network (62) schematically indicated by dashed lines. Network 62 is also connected to a central unit 64. As will be appreciated, network 62 can be a wired network, a wireless network, or a combination of wired and wireless networks for interconnecting with the aforementioned devices and cameras. Those skilled in the art will be able to select the most suitable network combination for the specific situation.

[0034] Sensor units 60A-60D (cameras) continuously track the movement and posture of any of the group members 39-45 and any audio signals. When placed on four different sides of the operating room, sensor units 60A-60D together will be able to detect any gestures, postures, or audio signals provided by any group member intended as instructions for controlling any of the devices 48, 50, 52, 54, 55, and 56. The sensor units or cameras 60A-60D can be mounted in an elevated position relative to the group members 39-45, or can be placed in different strategic configurations to enable effective registration of the provided instructions.

[0035] Before being tracked by the system using sensor units 60A-60D, each member of the entire surgical team 39-45 must complete an authentication session with the system, such as to enable the system to associate these personnel with member profiles stored in memory unit 71 of central unit 64. The authentication session can be performed in various ways as indicated above. Figure 3 In this embodiment, a standard login process performed at input terminal 76 can be used to log group members into the system. Alternatively or otherwise, central unit 64 can be connected to any kind of biometric sensor, such as a retinal or iris scanner or fingerprint scanner. Once the authentication session is completed, each member of the surgical team who has performed the authentication session is added to the system as a group member. Sensing units 60A-60D continuously track each group member based on information obtained during the authentication session and member profiles available from memory unit 71. The authentication session can use auxiliary cameras to record any non-permanent attributes of the group members, such as the color of their clothing or any other attributes that may not be the same every day. During tracking, if any group member temporarily leaves the field of view of the sensor units or cameras 60A-60D, the continuous tracking of that group member is interrupted, and this may require the group member to perform the authentication session again upon returning to the field of view. As will be appreciated, in other embodiments, this can be reliably resolved by any system capable of recovering from or resolving the interruption (e.g., an RFID scanner). To eliminate as many errors as possible, in demanding situations and environments such as medical settings, it is preferable to redo the authentication session.

[0036] If one of the group members 39-45 performs a gesture, such as one used to provide instructions to the system, this can be recorded by sensor units 60A-60D during tracking, and the position of each group member can be continuously tagged by the system, for example. When a gesture is performed, because the person is tagged in the data (e.g., images) of the sensor units 60A-60D, the system can immediately associate the instruction with the group member providing the instruction. This information can be provided as an event to the communication unit 70 in the central unit 64 via network 62. The control unit 65 receives this information from the communication unit 70. The control unit 65 includes a processor 68. The control unit uses the processor to identify the control command associated with the provided instruction when an instruction is detected. For this purpose, the detected instruction is associated with the instruction group member. The memory unit 71 is accessed to retrieve the member profile data of the instruction group member, and a match is made between the detected instruction and a reference or reference instruction in the member profile data. Once a match is made, the expected control command to be provided to any one or more of the devices 48, 50, 52, 54, 55, 56 is identified. Figure 3 In the embodiment illustrated in the diagram, the central unit 64 controls the operation of multiple devices 48-56 using not only the intended control commands identified but also the member profiles of the intended devices. As described above, the gestures and meanings of each member in group members 39-45 can be personalized using the member profile data stored in memory 71.

[0037] Despite Figure 3 The central unit 64, communication unit 70, and memory unit 71, which include control unit 65 and processor 68, are illustrated as a single device, but in different embodiments, one or more of these units may be implemented as independent devices or in combination with other such components (in order to cooperate, such as to form a control unit).

[0038] exist Figure 4 The method according to the present invention is further illustrated schematically in the figure. Figure 4 The illustration shows a memory unit 71 and a tracking system 60, generally indicated as a single device. The tracking system 60 may consist of multiple sensor units (such as camera units, microphones, or any other scanners that can be used to track people). For simplicity, in... Figure 4 The image shows a single device 60. Figure 4 The illustration also shows multiple devices 78, 78' and 78'' that will be controlled using the method of the present invention.

[0039] The method may begin by adding one or more group members to form a group. This is achieved using an authentication session 82. Authentication session 82 can be applied to all group members at the beginning of the method of the present invention. However, those skilled in the art will understand that an authentication session can be executed at any time during the method of the present invention to add additional group members to the group. Therefore, although step 82 is illustrated at the beginning of the method, it will be appreciated that the authentication session can also be applied at any other time. Authentication session 82 is initiated by the active action of a user who wishes to be added to the group as a group member.

[0040] In step 83, input for performing authentication is received from the user. This input can be obtained using standard login methods such as... Figure 3 Login devices such as device 76 are provided, but may also involve devices such as... Figure 3 The authentication is performed using a biometric scanner such as scanner 74. Once the input has been received, in step 85, the authentication session determines whether a new group member can be added to the group. To perform step 85, the authentication session 82 may optionally access memory unit 71 to retrieve group profile 89, for example, to check whether the user to be added as a group member is authorized to be part of the specific group to which the application is applied. For example, a specific all member may be authorized to be part of a specific team, but not to be part of other teams. Thus, such user member profiles may appear in the system of the present invention; however, even if the member profile appears in the system's memory unit 71, access to the group profile will be denied in step 85 if the group member is not part of the group.

[0041] If authentication fails or access is denied, the authentication session returns to step 83, as indicated by arrow 86, awaiting input from the same or another user. If the authentication session is successful, the method continues in step 87 by providing the tracking system with details about being a member of a group.

[0042] In step 90 following authentication, each member of the authenticated group is continuously monitored by tracking system 60. Inputs received from the tracking system leading to this step are generally indicated by double arrows 94. Using authentication information, individuals appearing in the data received from the tracking system can be continuously tagged in step 92. In step 95, tracking system 60 determines whether one or more members of the group have provided instructions detected as gestures, voices, or postures. Upon successful confirmation of instruction reception in step 95, step 100 is executed (via branch 99). Tracking (step 90) and tagging (step 92) are performed continuously while the control unit executes step 95 or any subsequent steps. This is to ensure no instructions are missed. The control unit can receive instructions at any time, and when system resources may be temporarily busy, such instructions can be, for example, stored in memory, processed based on priority settings, or forwarded to auxiliary units. Figure 4 In step 100, branch 97 indicates continuous monitoring of the instructions or actions to be processed—the process is not stopped until the last instruction is processed. In step 100, the detected instruction is associated with the group member who provided the instruction using the tag added in step 92, and the system continues with the identification of the control command associated with the received instruction in step 103. In step 105, member profile data from the memory cell is accessed, and in step 107, the detected instruction is compared with reference instructions in the member profile data, such as to match the detected instruction with any reference instructions in the reference instructions. Once such a match is made, the control command associated with the matching reference instruction can be identified. Optionally, if the method is used to control multiple devices (i.e., devices 78, 78', and 78'' here), in step 109, the member profile data from memory cell 71 can also be used to identify the device to which the control command is intended. Therefore, as an additional option in step 111, the method can also verify whether the control command intended to be provided by a group member to the corresponding device is authorized to that group member. The verification can be performed in different ways; for example, authorization can be identified using member profile data or group profile data from memory cell 71.

[0043] In step 115, as indicated by arrow 117, the identified control command is provided to the identified device 78, 78', or 78''. Furthermore, in step 119, the identified control command, along with, for example, a timestamp, the group member providing the instruction, and any other relevant data, is stored in a log held in memory unit 71. Such a log can be used for training purposes or for quality checks.

[0044] This invention proposes a system solution that identifies the person controlling (some) complex devices. Therefore, the interaction between the user and the system becomes a personalized, two-way dialogue that allows multiple users to interact with the device simultaneously. This is achieved by tracking each individual and using their identity to label every input they give to the system.

[0045] At any given moment, more than one person in the operating room may need to control parts or specific features of the medical system. Avoiding giving inconsistent or conflicting commands to the system is crucial. Conventional systems are unaware of who is behind the control action (in other words, what the role, authority, and personal preferences of the human behind the control action are). By allowing the medical system to be controlled via human gestures / gestures / voice, it becomes important for the system to know the human performing the control action to ensure the medical system behaves in a consistent and predictable manner. All personnel in the operating room can be uniquely identified (e.g., using retinal controls or fingerprints) and then continuously tracked (e.g., using facial recognition). By tracking each individual, the input given to the system can be tagged using the identity of the person issuing the input. The system can then respond in a way that leads to a highly personalized dialogue between the user and the system.

[0046] This invention proposes first uniquely identifying an individual (e.g., retinal control or fingerprint), and then continuously tracking (facial recognition, etc.) them and their commands. Their gestures / gestures / voice commands are analyzed by the tracking system and matched against a set of commands known to the user. The identified commands become events tagged with their unique identity, location, and timestamp. Events combining an individual's role, permissions, and personal preferences (each user's unique set of gestures) lead to the system's reaction and appropriate feedback. The system can react differently to the same output depending on the identity tags attached to it. This personalized dialogue can occur in parallel for multiple users of the same system. The degree of personalization by the system is flexible: users can provide their own preferred set of gestures regarding how the system should react or respond; and users can provide their own preferred feedback mechanisms. Due to the system's established awareness of the user, better temporal coherence in the interaction between the user and the system can be achieved.

[0047] As an advantage of this invention, the detectability of gestures can be improved because the system can be tuned to a variety of gestures made by each user. Furthermore, the system can be adapted to users with disabilities or preferences arising from the type of work they perform (e.g., a user who cannot use their arm can have their own personalized set of commands to which the system responds). Further, the invention can be used to personalize the way a user interacts with the system; for example, a surgeon can raise their arm to trigger a scan, while another surgeon can shake their head. Moreover, the invention can be used to establish a trajectory of all events. For example, in a surgical setting, this can be used to later detect errors or the cause of specific problems with a patient; and in environment-dependent workflows, such a system provides assistance in performing workflow analysis.

[0048] The method and system of the present invention can be extended by additional systems that include managing the identities and personal preferences of users within the management system. Another possible extension is to provide the system with contextual awareness derived from the conversations of people in a room.

[0049] Although the invention has been described in the foregoing description and illustrated with reference to some specific embodiments thereof, the scope of the invention is defined only by the scope of the appended claims.

Claims

1. A method for operating a device (78, 78', 78'') by controlling the device based on input received from one or more group members (39-45) of the group, the method comprising the steps of: - Authentication (83) of each user is performed via user input devices (74, 76) for interaction with one or more users, in order to add the user as a group member, so as to establish the group such that only the authenticated group members are monitored; - Use sensor unit (60) to monitor (90) each certified group member for detecting at least one instruction provided by an instruction group member among the one or more certified group members, the instruction including at least one of visual or auditory instructions; - When the control unit (65) detects the instruction, it identifies (103) a control command for controlling the device, the control command being associated with the instruction, and the identification of the control command includes: a) The control unit (65) associates the detected instruction with the instruction member in the certified group member (100). b) Accessing (105) the member profile data of the instruction group member from memory cell (71), the member profile data of the instruction group member including a reference instruction set, and c) Match the detected instruction with a given reference instruction selected from the reference instruction set in the member profile data of the instruction group member (107) to identify the control command; as well as - The control unit (65) provides (115) the control command to the device (78, 78', 78'') only if the detected instruction matches the reference instruction in the member profile data of the instruction group member.

2. The method according to claim 1, wherein, The monitoring (90) steps include: continuously tracking each certified group member.

3. The method according to any one of claims 1 or 2, wherein, The identification (103) of the control command further includes: for the identified control command, verifying the authorization status obtained from the member profile data of the instruction group member (111).

4. The method according to any one of claims 1 or 2, further comprising the step of storing (119) the instructions, the identified control commands and the member identifiers of the instruction group members in a memory unit (71).

5. The method according to any one of claims 1 or 2, wherein, The authentication (83) step includes the further step of accessing (82) the group profile from the memory unit (71) before adding the user as a group member, to enable the authenticated user to be identified as an authorized group member.

6. The method for operating a plurality of devices (78, 78', 78'') according to any one of claims 1 or 2, wherein, The control unit (65) is included in a central unit (64) that receives sensor signals from the sensor unit (60), and wherein the memory unit (71) is accessible by the control unit (65), the method further comprising the following steps: - Candidate devices (78, 78', 78'') for receiving the control command are identified (109) by the control unit (65), wherein the candidate devices are identified by matching the detected instruction with the reference instruction in the member profile data of the instruction group member.

7. The method according to any one of claims 1 or 2, wherein, The sensor unit (60) is configured to include at least one element of the group consisting of: receiving a visual image for detecting gestures or postures, and receiving an audio signal for detecting speech or other sounds.

8. A means for controlling the operation of devices (48, 50, 52, 54, 55, 56) by means of input received from one or more group members (39-45) of a group, the means comprising: - Sensor unit (60A-D) for monitoring each group member to detect at least one instruction provided by an instruction group member among the one or more group members, the sensor unit being arranged to receive at least one of a visual image or an auditory signal to detect a visual or auditory instruction; as well as - A control unit (65), which is communicatively connected to the sensor unit (60A-D) for receiving signals output from the sensor unit, the control unit including a processor (68) for identifying, upon detecting the instruction, a control command for controlling the device (48, 50, 52, 54, 55, 56), the control command being associated with the instruction, wherein the control unit is communicatively connected to the device; The device is characterized in that it further comprises: - User input devices (74, 76) for interacting with a user to perform authentication of the user, adding the user as a group member, establishing the group, and monitoring only authenticated group members; and - Memory unit (71) for storing one or more member profile data of the certified group members, the member profile data of each certified group member including a reference instruction set for the group member; The control unit (65) is further arranged by the processor (68) to associate the detected instruction with the instruction group member among the certified group members; access the member profile data of the instruction group member from the memory unit (71); and match the detected instruction with a given reference instruction selected from the reference instruction set in the member profile data of the instruction group member to identify the control command, and provide the control command to the device (48, 50, 52, 54, 55, 56) only if the detected instruction matches a reference instruction in the member profile data of the instruction group member.

9. The apparatus according to claim 8, wherein, The user input device includes at least one element from the group comprising: a biometric scanner (74) including an iris or retina scanner or a fingerprint scanner for authenticating a user based on individual eye characteristics; a keyboard, a touch-sensitive screen or a camera for entering personal login details.

10. The apparatus according to any one of claims 8 or 9, wherein, The sensor unit includes at least one element from the group comprising: an imaging device or imaging arrangement including one or more cameras (60A-D) for observing an area; and an audio sensor device or audio sensor arrangement including one or more microphones for sensing audio signals.

11. The apparatus according to any one of claims 8 or 9, wherein, The control unit is further arranged by the processor (68) to identify candidate devices (48, 50, 52, 54, 55, 56) for receiving the control command, wherein the candidate devices are identified by means of matching the detected instruction with the reference instruction in the member profile data.

12. A non-volatile computer-readable medium comprising a computer program product stored thereon, the product comprising program code instructions for implementing the method according to any one or more of claims 1-7 for operating the device (78, 78', 78'') by controlling the device based on input received from one or more group members (39-45) of the group.

Images