Providing information about region of interest

Abstract

A mechanism for selectively providing information about a region of interest of a first individual to a second individual. The mechanism determines whether the region of interest falls within a field of view of the second individual. If the region of interest does not fall within the field of view, appropriate information about the region of interest is provided to the second individual.

Description

Technical Field

[0001] The present invention relates to the field of facilitated collaboration between individuals in a shared environment. Background Art

[0002] There are a variety of environments and use case scenarios where multiple individuals need to collaborate in the same physical space to perform or run a technical task. Examples include surgery, repairing or maintaining a piece of equipment (e.g., a car), TV or film production, preparing food in a commercial kitchen, etc.

[0003] In this case, individuals typically communicate with each other in order to improve collaboration. In particular, an individual can focus on an area of ​​interest or point of interest and communicate with another individual about the area of ​​interest / point of interest. As an example, during a surgical operation, a surgeon may ask a surgical assistant to provide a tool for plugging or clamping a bleeding point (which serves as the surgeon's point of interest / area of ​​interest).

[0004] There is a continuing desire to improve collaboration in such environments in order to minimize errors and avoid the likelihood of them occurring. Methods that facilitate such improvements would provide for technical collaboration tasks that perform more efficiently. Summary of the invention

[0005] The invention is defined by the claims.

[0006] According to one aspect of the present invention, a computer-implemented method of assisting a second individual in identifying a region of interest of a first individual is provided, wherein the first individual and the second individual are located in a same physical environment.

[0007] The computer-implemented method includes: determining an area of ​​interest of the first individual within the field of view of the first individual relative to the physical environment near the first individual; determining whether the determined area of ​​interest is within the field of view of the second individual; and in response to determining that the determined area of ​​interest is not within the field of view of the second individual: providing an indicator to the first individual and / or the second individual that the determined area of ​​interest is not within the field of view of the second individual; and / or providing additional information about the determined area of ​​interest to the second individual.

[0008] The proposed method provides techniques for supporting two or more users or individuals within a shared physical environment to facilitate providing (to a second user / individual) information about a point of interest or region of interest of a first user / individual. In particular, a region of interest of a first individual is determined, and if the region of interest is not in the field of view of the second individual, an indicator is provided to the first individual and / or the second individual, and / or additional information about the determined region of interest is provided to the second individual.

[0009] The proposed method thus provides an indicator to the first / second individual and / or provides information to the second user only when the second user cannot see the region of interest. This facilitates reminding the first individual and / or the second individual when the second individual cannot see the region of interest and / or facilitates suggesting the region of interest to the second individual or the second individual's knowledge of the region of interest.

[0010] The proposed method can, for example, address instances where a first user is asking or mentioning to a second user about a point of interest in the first person's field of view (which is not visible to the second person's field of view). Embodiments are intended to notify one or both of the two users of this situation and alert the first / second user to this situation and / or enable the second user to obtain a view of the point of interest or other information about the point of interest.

[0011] The method may also include: in response to determining that the determined area of ​​interest is within the field of view of the second individual, neither providing an indicator to the first individual and / or the second individual that the determined area of ​​interest is not within the field of view of the second individual, nor providing the additional information about the determined area of ​​interest to the second individual.

[0012] It should be understood that other forms of information may be provided to the second user based on the determined region of interest. For example, an indicator of the location of the determined region of interest within the field of view of the second individual may be provided. However, if the region of interest is not within the field of view of the second individual, the information will be different from the information provided.

[0013] The method may include the following steps: in response to determining that the determined region of interest is not within the field of view of the second individual, providing an indicator to the first individual that the determined region of interest is not within the field of view of the second individual. Thus, if the determined region of interest is not within the field of view of the second individual, an indicator ("first indicator") may be provided to the first individual. Of course, it should be understood that if the determined region of interest is within the field of view of the second individual, the first indicator is not provided to the first individual.

[0014] The method may include the following steps: in response to determining that the determined region of interest is not within the field of view of the second individual, providing an indicator to the first individual that the determined region of interest is not within the field of view of the second individual. Thus, if the determined region of interest is not within the field of view of the second individual, an indicator ("second indicator") may be provided to the second individual. Of course, it should be understood that if the determined region of interest is within the field of view of the second individual, the second indicator is not provided to the second individual.

[0015] The step of determining the area of ​​interest may include: obtaining an environmental model representing the physical environment near the first individual; obtaining first gaze data representing the origin and direction of the first individual's gaze relative to the environmental model; and processing at least the environmental model and the first gaze data to identify the area of ​​interest relative to the environmental model.

[0016] In this way, gaze tracking or identification can be used to identify a region of interest for a first individual. Gaze tracking techniques are widely used in the prior art, but are used here to identify a region of interest for helping a second individual (who is not the tracked gazer) gain additional information.

[0017] The environment model and the first gaze data may be derived from the image data.

[0018] The step of determining whether the determined area of ​​interest is within the field of view of the second individual may include: obtaining spatial information, the spatial information identifying the relative position and orientation of the second individual relative to the environmental model; processing the spatial information, the area of ​​interest and the environmental model to determine whether the determined area of ​​interest is within the field of view of the identified second individual relative to the environment near the first individual.

[0019] In particular, the step of processing the spatial information and the environment model may include processing the spatial information and the environment model to identify a field of view of the second individual relative to the environment model. Determining whether the region of interest is within the identified field of view may include determining whether the position of the region of interest relative to the environment model is within the field of view relative to the environment model.

[0020] In at least one example, the step of determining the region of interest also includes obtaining first communication data, the first communication data representing one or more verbal or non-verbal communications from the first individual regarding the region of interest; and wherein one or more of the following are performed: the step of processing at least the environment model and the first gaze data includes processing at least the environment model, the first gaze data, and the first communication data to identify the region of interest within an environment near the first individual.

[0021] The method may also include obtaining second communication data, the second communication data representing one or more verbal or non-verbal communications from the first individual; wherein the step of determining whether the determined area of ​​interest is within the field of view of the second individual is only performed if the second communication data contains verbal or non-verbal communications from the first individual regarding the area of ​​interest.

[0022] In this way, the region of interest can be suggested to the second individual only when the first individual mentions or refers to the region of interest. This avoids or reduces the processing power required to unnecessarily provide additional information when the additional information is not needed (i.e., when the second user does not need to pay attention to the additional information) and reduces distraction to the second individual.

[0023] If present, the first communication data and the second communication data may be the same communication data.

[0024] The further information may include an indicator of a recommended adjustment of the second individual's field of view such that the determined region of interest will fall within the adjusted field of view.

[0025] The further information may include information identifying the determined region of interest and / or an image of the determined region of interest.

[0026] In the event that the determined region of interest is occluded by an object within the field of view of the second individual, the further information may comprise an indicator of the relative position of the occluding object within the field of view of the second individual.

[0027] The method may include providing further information about the determined region of interest to the second individual in response to determining that the determined region of interest is not within the field of view of the second individual. Providing the further information may include providing a visual representation of the further information to the second individual at a user interface.

[0028] There is also provided a computer program product comprising computer program code means which, when executed on a computing device having a processing system, causes the processing system to perform all the steps of any method described herein.

[0029] A processing system for assisting a second individual in identifying an area of ​​interest of a first individual is also provided, wherein the first individual and the second individual are located in the same physical environment. The processing system is configured to: determine an area of ​​interest of the first individual within the field of view of the first individual relative to an environment near the first individual; determine whether the determined area of ​​interest is within the field of view of the second individual; and in response to determining that the determined area of ​​interest is not within the field of view of the second individual: provide an indicator to the first individual and / or the second individual that the determined area of ​​interest is not within the field of view of the second individual; and / or provide additional information about the determined area of ​​interest to the second individual.

[0030] Also proposed is a system comprising said processing system and a user interface configured to provide a visual representation of said further information to said second individual.

[0031] These and other aspects of the invention will be apparent from and elucidated with reference to the embodiment(s) described hereinafter. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] For a better understanding of the invention and to more clearly show how it may be put into practice, reference will now be made to the accompanying drawings, in which:

[0033] Figure 1 The physical environment is illustrated;

[0034] Figure 2 is a flow chart illustrating the method;

[0035] Figure 3 is a flow chart illustrating another method;

[0036] Figure 4 illustrates a system according to an embodiment; and

[0037] Figure 5 A processing system according to an embodiment is illustrated. DETAILED DESCRIPTION

[0038] The present invention will be described with reference to the accompanying drawings.

[0039] It should be understood that the detailed description and specific examples are intended for illustrative purposes only when indicating exemplary embodiments of the apparatus, system and method, and are not intended to limit the scope of the invention. These and other features, aspects and advantages of the apparatus, system and method of the invention will be better understood from the following description, the appended claims and the accompanying drawings. It should be understood that the drawings are merely schematic and are not drawn to scale. It should also be understood that throughout the drawings, the same reference numerals are used to indicate the same or similar parts.

[0040] The present invention provides a mechanism for selectively providing information about a region of interest of a first individual to a second individual. The mechanism determines whether the region of interest falls within the field of view of the second individual. If the region of interest does not fall within the field of view, appropriate information about the region of interest is provided to the second individual.

[0041] Embodiments are based on the recognition that a second individual may not always be able to see a feature / element referenced by a first individual, but that information overload to the second individual will negatively impact their ability to contribute to a collaborative task. Therefore, it is proposed to provide information about a particular feature / element only when the feature / element is not within the second individual's field of view.

[0042] The proposed method can be employed in any scenario where collaborative tasks are being performed (eg, in an operating theater / room, etc.).

[0043] Figure 1Illustrated are use case scenarios in which embodiments can be employed for improved context understanding.

[0044] In particular, Figure 1 A physical environment 100 is schematically illustrated in which a first individual 110 and a second individual 120 are both located. Here, the physical environment 100 is an operating room / room, the first individual 110 is a surgeon, and the second individual 120 is a surgical assistant / hospital assistant. The two individuals are working together (e.g., performing a surgical procedure on a patient 150).

[0045] It should be understood that Figure 1 are schematic, and an actual operating theatre / room would likely contain a significantly greater amount of equipment and / or a significantly greater number of personnel, which have been omitted for clarity.

[0046] In large physical environments, effective communication is critical to achieving successful outcomes for collaborative missions. However, this is not always easy to achieve due to the amount of equipment and people that may be present in such environments.

[0047] For example, during a surgical procedure, the surgeon 110 may need to quickly react / respond to sudden or urgent changes / events (e.g., certain tools need to be provided by the surgical assistant 120 from the tool station 131). Ideally, the surgical assistant would be able to respond or proactively react to the surgeon's requests or communications.

[0048] However, due to the presence of objects, equipment (e.g., tool station 131 or monitoring system 132), and / or other people, the assistant does not always have a good or clear view of emerging events that the surgeon can see, which means that the assistant lacks contextual information to respond to the surgeon's requests or communications.

[0049] For example, if a surgeon is making an incision that is causing undesirable bleeding, the surgeon may indicate (audibly) "I need a clamp to stop the bleeding here". However, the surgical assistant may not be in a position to see the bleeding or position the incision, and therefore cannot make a good judgment about what type of clamp to provide. This will affect the speed and / or efficiency with which valuable care is provided to the patient, which can directly affect the likelihood that the surgical procedure will have a successful outcome.

[0050] The above scenario is purely for example, and those skilled in the art will readily appreciate that similar scenarios may occur in this and other technical fields (eg, during car repair, food preparation, etc.).

[0051] The present invention relates to a method for improving the contextual information available to a first individual and / or a second individual based on the first individual's view. This allows the first individual and / or the second individual to perform more appropriate actions based on what the first individual is able to see to reliably assist the individual in performing a technical task (e.g., assisting the second individual in selecting the most appropriate tool for a surgical action).

[0052] In particular, the method proposes to first determine the region of interest of the first individual within the field of view of the first individual. If the region of interest is not within the field of view of the second individual, (additional) information about the region of interest is provided to the second individual, and / or an indicator is provided to the first individual and / or the second individual. The former provides the second individual with useful contextual information for understanding or identifying the region of interest when the second individual can see the region of interest, without interfering with or disturbing the second individual. The latter provides the first individual and / or the second individual with a prompt that the second individual cannot see the region of interest, so as to prompt more effective communication.

[0053] Figure 2 is a flow chart illustrating a computer-implemented method 200 according to an embodiment.

[0054] The method 200 includes step 210 of determining a region of interest of a first individual within a field of view of the first individual relative to a physical environment proximate to the first individual.

[0055] The region of interest may be a point of interest (POR) of the first individual, ie, a region of the environment near the first individual that the first individual is viewing. As another example, the region of interest may be a region surrounding or near a point of interest.

[0056] The method 200 further comprises step 220 of determining whether the determined region of interest is within the field of view of the second individual. The field of view represents the absolute area of ​​the environment that can be seen by the second individual (eg, at a specific point in time).

[0057] The method further includes executing step 230 and / or step 240 .

[0058] Step 230 includes, in response to determining that the determined region of interest is not within the field of view of the second individual, providing to the second individual additional information about the determined region of interest and / or an indicator that the determined region of interest is not within the field of view of the second individual. Various examples of suitable content of the additional information are provided later in this disclosure.

[0059] Step 240 includes providing an indicator to the first user that the determined region of interest is not within the field of view of the second individual. The indicator may include information responsive to determining whether the determined region of interest is within the field of view of the second individual.

[0060] In some examples, step 240 may include, in response to the determined region of interest not being within the field of view of the second individual, providing an indicator to the first individual that the region of interest is not viewable to the second individual. The indicator may be provided in the form of a visual representation at a user interface or the like.

[0061] The first individual can use this information to provide more explicit communication about the area of ​​interest, allowing the second individual to act or respond appropriately. If possible, this can also prompt the first individual to change the environment to enable the second individual to view the area of ​​interest.

[0062] If any information is provided to the first individual and / or the second individual in response to the determined region of interest not being within the field of view of the second individual, provision of that information may cease after the trigger occurs.

[0063] In one example, the trigger may be a predetermined length of time that has passed since the information was provided. In another example, the trigger may be a predetermined length of time that has passed since the first individual's field of view has moved away from the region of interest. In another example, the trigger may be an instruction / command (e.g., a voice request) to the second individual. In another example, the trigger may be a recognition event of the region of interest entering the second individual's field of view. In another example, the trigger may be a predetermined length of time that has passed since the recognition event of the region of interest entering the second individual's field of view.

[0064] If step 220 determines that the determined region of interest is within the field of view of the second individual, method 200 may return to step 210 .

[0065] One example mechanism for performing steps 210 and 220 utilizes an image-based approach.

[0066] In this example, the step 210 of obtaining a region of interest may include obtaining first image data representing a field of view of a first individual. The image data can then be processed to identify a segment within the first image data representing the region of interest. This can be performed using, for example, eye gaze information that identifies the relative eye gaze of the first individual with respect to the first image data, methods of which are well known in the art. In this manner, a particular area or region of the image data (i.e., a segment of the image data representing the region of interest) can be identified.

[0067] Thus, step 210 may include obtaining first image data representing a field of view of a first individual; obtaining eye gaze information identifying a relative eye gaze of the first individual relative to the first image data; and processing the first image data and the eye gaze information to identify a segment representing an area of ​​interest.

[0068] A possible method for identifying a segment representing a region of interest in image data is disclosed in US patent application publication number US2021 / 149482A1. Other methods are also known in the art.

[0069] Step 220 may include determining whether the same region of interest represented in the identified section of the first image data is represented within the second image data representing the field of view of the second individual.

[0070] Step 220 may be performed, for example, by processing identified segments of the first image data (which represent regions of interest) to identify or annotate the regions of interest (e.g., identify one or more physical elements in the regions of interest). The method may utilize a suitably configured / trained machine learning algorithm or computer vision system (e.g., a neural network) to identify or classify these regions or areas.

[0071] The second image data may then be processed to identify whether a corresponding portion of the second image data contains a representation of the region of interest (e.g., whether the identified one or more physical elements are present in the second image data). This may also be performed using a suitably configured / trained machine learning algorithm or computer vision system (e.g., a neural network). For example, a neural network may classify the second image data to predict whether the identified element(s) are present in the second image data.

[0072] As another example, step 220 may be performed by performing a comparison process between identified segments of the first image data and the second image data. If the similarity between the identified segment of the first image data and any segment of the second image data exceeds a predetermined threshold, then the second image data can be predicted to contain a representation of the region of interest.

[0073] Thus, more generally, step 220 may include obtaining second image data representing a field of view of a second individual, processing the identified segment of the first image data and the second image data to determine whether a region of interest represented in the identified segment of the first image data is represented within the second image data.

[0074] The first image data may be obtained, for example, by a camera worn by the first individual. The second image data may be obtained, for example, by a camera worn by the second individual.

[0075] In this way, the image data can be processed to perform steps 210 and 220 of determining whether the region of interest is within the field of view of the second individual. Thus, an image-based approach is achieved.

[0076] Figure 3Another example mechanism for performing steps 210 and 220 is illustrated, which employs a model-based approach to determine whether the region of interest is within the field of view of the second individual.

[0077] In particular, step 210 may include step 311: obtaining an environment model representing the physical environment near the first individual. The environment model may be, for example, a (computer-based) model, map or representation of the physical environment near the first individual. The environment model may be defined in a predetermined coordinate system representing the space in which the environment is located. The predetermined coordinate system may also be referred to as a model coordinate system.

[0078] Typically, an environment model is a data representation of the physical environment surrounding a first individual (i.e., a mathematical representation of a surface near the first individual). The environment model may define or represent (the three-dimensional shape of) the environment near the first individual within a predefined / model coordinate system. In particular, the environment model may define or represent the positioning of at least a surface of the environment near the first individual within the model coordinate system.

[0079] One example of a suitable data representation is a polygonal mesh that defines the locations of vertices of a plurality of imaginary polygons (eg, triangles) that together represent the shape of the physical environment in the vicinity of the first body. Another example is a volumetric mesh.

[0080] Various methods for deriving a suitable model, map or representation of the physical environment may be employed. In one approach, generating a 3D model or representation of the environment may employ a 3D scanner to scan the environment in order to construct the model. Typically, a 3D scanner collects data about the environment (e.g., image data or LIDAR data). The data about the environment can be appropriately processed (e.g., using photogrammetry techniques, etc.) to construct a 3D model of the environment. A suitable example of a 3D scanner is a LIDAR system, but other 3D scanners utilize images or image data captured by a camera.

[0081] Well-known techniques for generating environment models include photogrammetry, structure from motion (SfM), multi-view stereo (MVS) algorithms, and simultaneous localization and mapping (SLAM) techniques.

[0082] As another example, a standardized environment model (e.g., a model representing an average or expected shape and / or structure of an environment) may be obtained in step 311. As an example, some environments may be considered to have an expected or default shape (e.g., a surgical environment), which can be utilized to generate a standardized environment model.

[0083] By way of example only, the expected shape and / or structure of the environment can be constructed by tracking the positioning of equipment within known environmental boundaries. For example, if equipment in the environmental space tracks its own position and orientation, an environmental model can be constructed based on the tracked position (because the shape of the equipment can be known or predetermined).

[0084] Of course, combinations of these procedures may be used. For example, a standardized environment model may be obtained and supplemented with additional model data (eg, model data derived from image data of (one or more) individuals).

[0085] Step 311 may include retrieving the environment model from a memory / storage unit or from a model generator (ie, a model generation unit). The model generator may be configured to generate the environment model, for example, by processing data obtained from a 3D scanner.

[0086] Alternatively, step 311 itself may comprise generating a model using data obtained from a 3D scanner, for example.

[0087] Step 210 may also perform step 312: obtaining first gaze data representing an origin (or "eye position") and direction (or "gaze direction") of the first individual's gaze relative to the environment. In particular, the first gaze data may facilitate identification of a relative origin and direction of the first individual's gaze relative to a model of the environment (i.e., using a model coordinate system).

[0088] In one example, first gaze data may be derived by separately determining 312A a first individual's eye position, which represents the origin of the first individual's gaze, and determining 312B a gaze direction, which represents the direction in which the first individual's eye(s) (or more specifically, pupil(s)) are looking. Both of these data values ​​may be relative to an environment model defined, for example, using a model coordinate system. For example, the eye position may be defined using coordinates of the model coordinate system. The gaze direction may be defined using a vector, etc.

[0089] If the environment model is created using simultaneous localization and mapping (SLAM) technology, the eye position of the first individual relative to the environment model (eg, in the model coordinate system) may be automatically determined in step 312A.

[0090] As another example, step 312A may use any suitable positioning technique that facilitates positioning of eye positions relative to a model of the environment (e.g., in a model coordinate system). This may include determining the eye position of the first individual within a second coordinate system (which is registered or can be registered to the model coordinate system). Suitable methods may employ remote 3D (depth) camera positioning techniques, Wi-Fi positioning, or any other positioning method.

[0091] The (eye) gaze direction of the first individual may be determined, for example, using a gaze direction tracker. A gaze direction tracker identifies the direction in which the individual's (one or both) eyes are looking and is well known in the art. Any gaze direction tracker may be implemented to derive the (eye) gaze direction (e.g., by Other examples will be apparent to those skilled in the art.

[0092] Another approach is to utilize a head orientation tracking system that tracks the orientation of the first individual's head. The orientation of the head indicates the gaze direction of the first individual. During complex technical tasks, individuals are more likely to look directly at the region of interest, so that the orientation of the head (e.g., relative to the environment model) can accurately indicate the gaze direction with sufficient accuracy.

[0093] When the origin (i.e., eye position) and gaze direction have been determined relative to the environment model, a region of interest relative to the environment model may be determined in step 313. Thus, step 210 may include a step 313 of processing at least the environment model and the first gaze data to identify a region of interest within the environment near the first individual.

[0094] Identifying the region of interest may specifically comprise identifying a position or location relative to a model coordinate system, eg the region of interest of the environment model. In other words, the region of interest may be a segment or part of the environment model.

[0095] When the origin and direction of the gaze of the first individual are defined in the same coordinate system as the environment model, the identification of the region of interest can be easily achieved. More specifically, the end point of the gaze on the environment model can be identified. The end point can represent the (first or nearest) intersection between the environment model and an imaginary line originating from the eye position (origin) and extending along the gaze direction. The end point represents a point near or within the region of interest. In some examples, the region of interest can be defined by the position of the end point.

[0096] As an example, the region of interest can be identified by tracking a line of sight from the origin of the first individual's gaze along the gaze direction of the first individual. The first or closest intersection between the tracked line of sight and the environment model can define the end point of the first individual's gaze, which is a location in / near the region of interest.

[0097] In some examples, a more accurate determination or identification of the region of interest can be achieved by processing additional information (in step 313) to identify the region of interest.

[0098] For example, the first communication data may also be used to identify the region of interest in step 313. The first communication data may include one or more verbal or non-verbal communications from the first individual regarding the region of interest. The first communication data may, for example, include audio data (representing verbal communications of the first individual) and / or gesture data (representing gestures of the first individual).

[0099] Thus, step 313 may include processing at least the environment model, the first gaze data, and the first communication data to identify a region of interest within the environment proximate to the first individual.

[0100] A method for using first communication data in identifying a region of interest may be to process (e.g., using a natural language processing system) the first communication data including audio data to extract identification information (e.g., a label or description) of the region of interest contained in the first communication data. The identification information may be used to identify a feature or element of / in an environment model that can be represented by the identification information. The identified feature or element may represent the region of interest.

[0101] In particular, the first gaze data may be used to identify an area of ​​interest or a place of interest. For example, an end point may be determined and the area of ​​interest may be defined as an area within a certain distance from the end point. The first communication data may be used to identify features or elements in an area / place that serve as an area of ​​interest.

[0102] As an example, consider a scenario in which the environment model represents a surgical environment (and accurately represents at least an object undergoing surgery). The first gaze data may identify a region of interest for the first individual, the region containing multiple possible features or elements (e.g., different parts of the object undergoing surgery). The region of interest may be a region containing the endpoint of the first individual's gaze (e.g., a region of a predetermined diameter, where the center of the region is defined by the endpoint). The first communication data may include audio data of the first individual's speech, for example, "I need a clamp to stop this bleeding point." The natural language processing algorithm may identify the region of interest as a "bleeding point." Step 313 may thereafter identify the bleeding point in the region of interest as an area of ​​interest.

[0103] European patent application EP3709060A1 (first inventor is Jan Hoffmann, published on March 19, 2014) discloses another example method for determining a first individual's region of interest ("fixation point"), which can be reused to perform step 210 of method 300.

[0104] Takemura, Kentaro et al., “Estimating 3D point-of-regarding and visualizing gaze trajectories under natural head movements” (Proceedings of the 3010 Eye Tracking Research and Applications Symposium, 3010) describe another method for determining an individual's region of interest (“point”), which can be used to perform step 210 of method 300.

[0105] The identified region of interest may be defined relative to a model of the environment (eg, using one or more coordinates of a model coordinate system). Identifying the region of interest may thus include identifying a location of the region of interest.

[0106] As previously described, step 220 of method 300 includes determining whether the determined region of interest is within the field of view of the second individual.

[0107] For example, step 220 may be performed by determining a field of view of the second individual relative to the environment model (eg, defined in step 311) and determining whether the location of the region of interest (relative to the environment model) falls within the field of view.

[0108] Another method for performing step 220 may be to determine if there are any objects blocking the individual's view toward the region of interest.

[0109] Both methods may share step 321 of obtaining spatial information that identifies the relative position of the second individual, and preferably the eye position of the second individual relative to the model of the environment. Thus, step 321 may include determining the eye position of the second individual within the model coordinate system. Methods for establishing or locating the position of an individual have been previously described (e.g., with reference to step 312A), and these methods may be employed in this scenario.

[0110] As an example, locating the eye position of the second individual in the environment can be accomplished using SLAM technology, a remote 3D (depth) camera, Wi-Fi positioning, or any other positioning method. If SLAM technology is used, the SLAM models generated for the two individuals can be combined to create an environmental model as a reference (e.g., in step 311, if performed).

[0111] In one method for performing step 220, step 220 includes a step 322 of processing the spatial information and the environment model to identify a field of view of the second individual relative to the environment near the first individual.

[0112] Identifying the field of view of an individual can be performed by using the determined position of the second individual as the origin and establishing the field of view using the determined or predicted field of view of the second individual. The field of view (or "visual range") represents the relative angle (e.g., visual angle or boundary) at which an individual can see an object, thereby defining the area that the individual can see by looking straight ahead without moving the eyes or gaze. The field of view of the second individual can be an average (e.g., a global average), a field of view, or a field of view established by previous measurements.

[0113] In one example, a standard measurement result can be used to define a field of view. A definition of a field of view is set forth in Spector RH, "Visual Fields" (Editors: Walker HK, Hall WD, Hurst JW, Clinical Methods: The History, Physical, and Laboratory Examinations, 4th edition, Boston, Butterworth, 1990, Chapter 116).

[0114] In another example, the visual field can be defined using measurements specific to the second individual (eg, established using visual field testing techniques such as confrontation perimetry, kinetic perimetry techniques, etc.).

[0115] "The FieldAnalyzer Primer: Effective Perimetry" by Hepijl, Anders, Vincent Michael Paella, and Boel Bengson (2012) provides examples of expected / average field sizes and techniques for determining an individual's specific field size.

[0116] Determining the field of view may include using the field of view of the second individual and the determined position of the second individual to identify a region of the environment model that may be visible to the second individual. The determined field of view may be represented by a second grid or model within the coordinate system used to define the environment model.

[0117] Once the field of view has been established, it can be determined in detail in step 323 whether the area of ​​interest is within the field of view (e.g., by determining whether the coordinate(s) representing the area of ​​interest fall within the space defined by the determined field of view (e.g., within the space defined by a grid representing the field of view)).

[0118] Alternatively and / or additionally, determining whether the region of interest is within the subject's field of view can be performed by projecting a line of sight from the determined eye position of the second individual to the region of interest in step 323. If one or more objects (e.g., defined by an environment model) block the line of sight, then the region of interest can be predicted to be within the subject's field of view. Otherwise, the region of interest can be predicted to be not within the subject's field of view.

[0119] If this alternative approach is taken, step 322 can be omitted.

[0120] Step 323 may serve as a determination or decision step. In response to determining in step 323 that the determined region of interest is not within the field of view of the second individual, the method may proceed to step 230 and optional step 240. Otherwise, method 300 may be restarted.

[0121] refer to Figure 2 , it has been previously explained that methods 200, 300 may also include step 230: in response to determining that the determined area of ​​interest is not within the field of view of the second individual, providing additional information about the determined area of ​​interest to the second individual, and / or providing an indicator ("second indicator" or "second individual indicator") to the second individual that the determined area of ​​interest is not within the field of view of the second individual.

[0122] A method for providing an indicator and / or additional information is to provide a visual representation of the indicator and / or additional information to the second individual at the user interface. Therefore, step 230 may include providing a visual representation of the indicator and / or additional information to the second individual at the user interface.

[0123] The user interface may, for example, include a wearable user interface, such as a glass or head-mounted head-up display and / or an augmented reality display. The wearable user interface may be configurable to provide visualization of the content of the additional information (e.g., in the form of images, graphical elements, holograms, text, augmented reality, etc.).

[0124] The indicator may take the form of a simple visual indicator (eg, a symbol that changes color, etc.) of whether the first individual's region of interest is within the second individual's field of view.

[0125] Various forms of additional information may be provided to the second individual. Several examples of these forms are described below. The additional information may include any one or more of these examples and / or other examples not explicitly described herein. In some examples, one or more of the forms described below are provided to the second individual only in response to a request or user input from the second individual.

[0126] In one example, the additional information may include an indicator of a recommended adjustment to the second individual's field of view so that the determined region of interest will fall within the adjusted field of view.

[0127] The recommended adjustment can be determined using a model of the environment, a position of the region of interest relative to the environment model, and at least an eye position of the second individual relative to the environment model.

[0128] For example, an imaginary line may be defined between the location of the region of interest and the eye position of the second individual. The recommended adjustment may be a recommended change to the eye position such that the imaginary line does not intersect any other part of the environment model before reaching the location of the region of interest.

[0129] Another approach is to use a defined field of view of the second individual relative to the environment model.The recommended adjustment may be a recommended change to the eye position so that the appropriately adjusted field of view of the second individual (relative to the environment model) encompasses the position of the region of interest relative to the environment model.

[0130] For example, a recommended adjustment can be provided to the second individual via the user interface in the form of an arrow showing where to move to enable the second individual to see the region of interest.

[0131] In an example, the additional information includes identifying information about the determined region of interest. For example, the identifying information may include a label of the region of interest. The label may be derived from an annotated environment model (as an annotated version of the environment model, which identifies different regions / areas of the environment model) and the location of the region of interest relative to the environment model. In a specific example, the annotation closest to the location of the region of interest may be selected as the label of the region of interest, which may be included as identifying information about the determined region of interest.

[0132] In an example, the additional information includes an image of the determined region of interest. As an example, the additional information may include an image taken from the viewpoint of the first individual (e.g., using a head-mounted or body-mounted camera) or an image taken from a camera near the first individual. The image will contain image information about the identified region of interest.

[0133] In an example, where the determined region of interest is occluded by an object within the second individual's field of view, the further information includes an indicator of the relative position of the occluding object within the second individual's field of view ("position indicator").

[0134] This can be achieved, for example, using a model of the environment, the position of the region of interest relative to the environment model, and at least the eye position of the second individual relative to the environment model. An imaginary line may be defined between the position of the region of interest and the eye position of the second individual. The position of occluding objects (i.e., objects located on the imaginary line) may be identified and included in the further information.

[0135] Since the second individual's field of view is known, it is possible to accurately identify the relative position of the occluding object in the second individual's field of view.

[0136] For example, the position indicator may be provided via the user interface in the form of an arrow showing the position of the region of interest (which is not visible to the second individual). As another example, the position indicator may be provided via the user interface in the form of an outline rendering of an occluding object.

[0137] As yet another example, the additional information may include a simulation of the region of interest (and optionally surrounding areas). As an example, the portion of the environment model corresponding to the region of interest may be provided to the second individual. Alternatively, the region of interest may be scanned (e.g., in real time) by a stationary (e.g., using LIDAR) or worn (by the first user) 3D scanner. The 3D scan may be used to generate a digital hologram / copy that can be provided to the second individual. This may be provided, for example, by a suitably configured augmented reality display.

[0138] In an advanced example, the occluding objects may be augmented with a 3D representation of the region of interest from the perspective of the second individual. Thus, the simulation of the region of interest may be appropriately generated and rotated and positioned within the user interface such that it provides a simulated view of the region of interest as it would exist without any occluding objects.

[0139] In another advanced example, an appropriate user interface can be used to effectively render away occluding elements, thereby revealing the region of interest. This represents a more advanced example of rendering the region of interest within a user interface. As an example, a mixed reality headset (e.g., Varjo XR-3) can be used as a user interface to simulate rendering away occluding elements to reveal (a representation of) the region of interest.

[0140] refer to Figure 2 , it has been previously explained that methods 200, 300 may also include step 240: in response to determining that the determined area of ​​interest is not within the field of view of the second individual, providing an indicator ("first indicator" or "first individual indicator") to the first individual that the determined area of ​​interest is not within the field of view of the second individual.

[0141] Method 300 may be configured to perform step 210 and / or step 220 and / or step 230 / 240 (and any subsequent steps) only in response to a trigger from the first individual. The trigger may be a verbal or non-verbal communication from the first individual regarding the region of interest.

[0142] Thus, determining whether the determined region of interest is within the field of view of the second individual may only be performed if the first individual provides information about the region of interest. This will reduce the required processing resources and avoid / reduce potential confusion for the second individual.

[0143] Thus, method 300 may include a step 350 of obtaining second communication data representing one or more verbal or non-verbal communications from the first individual.

[0144] Step 355 may be performed to determine whether the second communication data includes verbal or non-verbal communication from the first individual regarding the region of interest.

[0145] In response to step 355 determining that the second communication data includes verbal or non-verbal communication from the first individual regarding the region of interest, the method moves to step 210. Otherwise, the method returns to step 350.

[0146] Steps 350 and 355 may be performed in alternating positions (eg, between steps 210 and 220 or between steps 220 and 230). A skilled person will be able to appropriately modify the functionality of steps 350 and 355 accordingly.

[0147] Steps 350 and 355 may be performed in any other method or mechanism according to other embodiments (eg, in the image-based mechanism described previously).

[0148] Figure 4 A system 400 is illustrated according to an embodiment.

[0149] System 400 includes a processing system 410 for assisting a second individual in identifying a region of interest of a first individual, wherein the first individual and the second individual are located in a same physical environment.

[0150] The processing system is configured to perform any of the previously described methods.

[0151] Therefore, the processing system is configured to: determine an area of ​​interest of a first individual relative to the environment near the first individual; determine whether the determined area of ​​interest is within the field of view of a second individual; and in response to determining that the determined area of ​​interest is not within the field of view of the second individual, provide an indicator to the first individual and / or the second individual that the determined area of ​​interest is not within the field of view of the second individual; and / or provide additional information about the determined area of ​​interest to the second individual.

[0152] The process for determining a region of interest and / or whether the determined region of interest is within the field of view of the second individual may utilize an environment model.The system 400 may include a model generator or model storage module 420 for generating / storing a model to be provided to a processing system.

[0153] The process for determining the region of interest and / or whether the determined region of interest is within the field of view of the second individual can utilize the determined eye position of the first individual and / or the second individual. The system 400 may include an eye position determiner 430 for generating or determining the eye position of the first individual and / or the second individual. A suitable eye position determiner 430 may employ remote 3D (depth) camera positioning technology, Wi-Fi positioning, or any other positioning method.

[0154] The process for determining the region of interest and / or whether the determined region of interest is within the field of view of the second individual may utilize the determined gaze direction of the first individual and / or the second individual. The system 400 may include a gaze direction tracker 440 for generating or determining a gaze direction of the first individual and / or the second individual.

[0155] The process for providing the indicator(s) to the first individual and / or the second individual and / or providing the second individual with additional information about the determined region of interest may be performed using the system's user interface 450. Example user interfaces include a head mounted user display and / or a stand-alone user display.

[0156] Providing the indicator(s) and / or the (further) information to the first individual and / or the second individual may include providing a visual representation of the indicator(s) and / or the (further) information at a user interface.

[0157] The user interface 450 may also be used to provide the first communication data and / or the second communication data to the processing system 410 , eg for triggering and / or for use within steps of a method performed by the processing system 410 .

[0158] Figure 5 4 is a schematic diagram of a processing system 410 according to an embodiment of the present disclosure. The processing system 410 is configured to execute a method according to an embodiment (e.g., referring to Figure 3 Method 300 described).

[0159] As shown, processing system 410 may include a (data) processing unit 520, a memory 564, and a communication module 568. These elements may communicate with each other directly or indirectly, for example, via one or more buses.

[0160] The processing unit 520 may include a central processing unit (CPU), a digital signal processor (DSP), an ASIC, a controller, an FPGA, another hardware device, a firmware device, or any combination thereof, which are configured to perform the operations described herein. The processing unit 520 may also be implemented as a combination of computing devices (e.g., a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration). In some embodiments, the processor is a distributed processing system (e.g., formed by a group of distributed processors).

[0161] Memory 564 may include cache memory (e.g., cache memory of processing unit 520), random access memory (RAM), magnetoresistive RAM (MRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, solid-state memory devices, hard drives, other forms of volatile and non-volatile memory, or a combination of different types of memory. In an embodiment, memory 564 includes a non-transient computer-readable medium. A non-transient computer-readable medium may store instructions. For example, memory 564 or a non-transient computer-readable medium may have program code recorded thereon, the program code including instructions for causing processing system 410 or one or more components of processing system 410 (particularly processing unit 520) to perform the operations described herein. For example, processing system 410 is capable of running the operations of method 500.

[0162] Instructions 566 may also be referred to as code or program code. The terms "instructions" and "code" should be interpreted broadly to include any type of (one or more) computer-readable statements. For example, the terms "instructions" and "code" may refer to one or more programs, routines, subroutines, functions, processes, etc. "Instructions" and "code" may include a single computer-readable statement or many computer-readable statements. The memory 564 having the code recorded thereon may be referred to as a computer program product.

[0163] The communication module 568 can include any electronic circuit and / or logic circuit that facilitates direct or indirect data communication between the processing system 410 and any external device. In this regard, the communication module 568 can include an input interface for receiving data from an external device (e.g., a model generator, an eye position determiner, a gaze direction tracker, and / or a user interface). The communication module 568 can also include an output interface for controlling an external device (e.g., a user interface) or providing data to an external device (e.g., a user interface).

[0164] In some examples, communications module 568 may facilitate direct or indirect communications between various elements of processing system 410 , such as by using one or more buses or other communications channels.

[0165] It should be understood that the disclosed method is preferably a computer-implemented method. As such, a computer program concept is also proposed, the computer program comprising computer program code for implementing any of the described methods when the program is run on a processing system (e.g., a computer or a group of distributed processors).

[0166] The different code sections, code lines or code blocks of the computer program according to the embodiment can be operated by a processing system or a computer to perform any method described herein. In some alternative implementations, the functions mentioned in (one or more) block diagrams or (one or more) flow charts can occur in the order indicated in the accompanying drawings. For example, two blocks shown in succession can actually be operated substantially simultaneously, or blocks can sometimes be operated in the opposite order, depending on the functions involved.

[0167] The present disclosure proposes a computer program (product) comprising instructions which, when executed by a computer or a processing system, cause the computer or the processing system to perform (the steps of) any of the methods described herein. The computer program (product) may be stored on a non-transitory computer readable medium.

[0168] Similarly, a computer-readable (storage) medium comprising instructions which, when executed by a computer or a processing system, causes the computer or the processing system to perform (the steps of) any of the methods described herein is also proposed. A computer-readable data carrier having a computer program (product) stored thereon is also proposed. A data carrier signal carrying the previously described computer program (product) is also proposed.

[0169] Those skilled in the art will easily develop a processing system for executing any of the methods described herein. Therefore, each step of the flowchart may represent a different action performed by the processing system and may be performed by a corresponding module of the processing system.

[0170] Variations of the disclosed embodiments will be understood and effected by those skilled in the art in practicing the claimed invention, based on a study of the drawings, the present disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the word "a" or "an" does not exclude a plurality. The fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

[0171] A single processor or other unit may implement the functions of several items recited in the claims. The computer program may be stored / distributed on a suitable medium (e.g., an optical storage medium or solid-state medium supplied together with or as part of other hardware), but may also be distributed in other forms (e.g., via the Internet or other wired or wireless telecommunications systems).

[0172] If the term "suitable for" is used in the claims or the specification, it should be noted that the term "suitable for" is intended to be equivalent to the term "configured to". If the term "arranged" is used in the claims or the specification, it should be noted that the term "arranged" is intended to be equivalent to the term "system", and vice versa. Any reference signs in the claims should not be construed as limiting the scope.

Claims

1. A computer-implemented method (200, 300) for assisting a second individual (120) in identifying a region of interest of a first individual (110), wherein: The first individual and the second individual are located in a same physical environment (100), and the computer-implemented method comprises: Determining (210) a region of interest of the first individual within a field of view of the first individual relative to a physical environment proximate to the first individual; determining (220) whether the determined region of interest is within the field of view of the second individual; In response to determining that the determined region of interest is not within the field of view of the second individual: providing (230, 240) to the first individual and / or the second individual an indicator that the determined region of interest is not within the second individual's field of view; and / or providing (230) further information about the determined region of interest to the second individual; and In response to determining that the determined area of ​​interest is within the field of view of the second individual, neither an indicator that the determined area of ​​interest is not within the field of view of the second individual is provided to the first individual and / or the second individual, nor is the additional information about the determined area of ​​interest provided to the second individual.

2. The computer-implemented method of claim 1 , comprising the steps of: In response to determining that the determined region of interest is not within the field of view of the second individual, an indicator is provided to the first individual that the determined region of interest is not within the field of view of the second individual.

3. A computer-implemented method according to any one of claims 1 or 2, wherein: The step of determining the region of interest comprises: obtaining an environment model representing the physical environment near the first individual; obtaining first gaze data representing an origin and direction of a gaze of the first individual relative to the environment model; and At least the environment model and the first gaze data are processed to identify a region of interest relative to the environment model.

4. The computer-implemented method of claim 3, wherein: The step of determining whether the determined region of interest is within the field of view of the second individual comprises: obtaining spatial information identifying a relative position and orientation of the second individual with respect to the environment model; The spatial information and the environment model are processed to determine whether the determined region of interest is within a field of view of the identified second individual relative to an environment proximate to the first individual.

5. A computer-implemented method according to any one of claims 3 to 4, wherein: The step of determining the region of interest further comprises obtaining first communication data representing one or more verbal or non-verbal communications from the first individual regarding the region of interest; and wherein one or more of the following are performed: The step of processing at least the environment model and the first gaze data comprises processing at least the environment model, the first gaze data and the first communication data to identify the region of interest within an environment near the first individual.

6. The computer-implemented method of any one of claims 1 to 2, wherein: The step of determining the region of interest of the first individual comprises: obtaining first image data representing a field of view of the first individual; obtaining eye gaze information that identifies a relative eye gaze of the first individual relative to the first image data; and processing the first image data and the eye gaze information to identify segments representing regions of interest; and The step of determining whether the determined region of interest is within the field of view of the second individual comprises: obtaining second image data representing a field of view of the second individual; and The second image data and the identified segment of the first image data are processed to determine whether the region of interest represented in the identified segment of the first image data is represented within the second image data.

7. The computer-implemented method of any one of claims 1 to 6, further comprising obtaining second communication data representing one or more verbal or non-verbal communications from the first individual; in, The step of determining whether the determined region of interest is within the field of view of the second individual is only performed if the second communication data comprises verbal or non-verbal communication from the first individual regarding the region of interest.

8. A computer-implemented method according to any one of claims 1 to 7, wherein: The further information comprises an indicator of a recommended adjustment of the field of view of the second individual such that the determined region of interest will fall within the adjusted field of view.

9. The computer-implemented method of any one of claims 1 to 8, wherein: The further information comprises information identifying the determined region of interest and / or an image of the determined region of interest.

10. The computer-implemented method of any one of claims 1 to 9, wherein: In the event that the determined region of interest is occluded by an object within the field of view of the second individual, the further information comprises an indicator of the relative position of the occluding object within the field of view of the second individual.

11. The computer-implemented method of any one of claims 1 to 10, wherein: The computer-implemented method includes: in response to determining that the determined region of interest is not within the field of view of the second individual, providing additional information about the determined region of interest to the second individual; and The step of providing the further information comprises providing a visual representation of the further information to the second individual at a user interface.

12. A computer program product comprising computer program code means which, when executed on a computing device having a processing system, causes the processing system to perform all the steps of the method according to any one of claims 1 to 11.

13. A processing system (410) for assisting a second individual (120) in identifying a region of interest of a first individual (110), wherein: The first individual and the second individual are located in the same physical environment (100), and the processing system is configured to: determining (210) a region of interest of the first individual within a field of view of the first individual relative to an environment proximate to the first individual; determining (220) whether the determined region of interest is within the field of view of the second individual; In response to determining that the determined region of interest is not within the field of view of the second individual: providing (230, 240) to the first individual and / or the second individual an indicator that the determined region of interest is not within the second individual's field of view; and / or providing (230) further information about the determined region of interest to the second individual; and In response to determining that the determined area of ​​interest is within the field of view of the second individual, neither an indicator that the determined area of ​​interest is not within the field of view of the second individual is provided to the first individual and / or the second individual, nor is the additional information about the determined area of ​​interest provided to the second individual.

14. A system (400), comprising: The processing system (410) of claim 13; as well as A user interface (450) is configured to provide a visual representation of the indicator to the first individual and / or the second individual and / or to provide additional information to the second individual.

Images