Occlusion classification and feedback

Abstract

In one implementation, a method of detecting occlusion of an image sensor is performed at a device including one or more image sensors, one or more speakers, one or more processors, and a non-transitory memory. The method includes receiving an image of a physical environment from one or more image sensors. The method includes detecting an occlusion that at least partially occludes one or more image sensors based on an image of a physical environment. The method includes classifying an occlusion as one of a plurality of occlusion types. The method includes playing the occlusion notification via the one or more speakers based on the classification.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent No. 63 / 354,023, filed on June 21, 2022, which is hereby incorporated by reference in its entirety. Technical Field

[0003] The present disclosure generally relates to providing feedback that an image sensor is at least partially occluded. Background Art

[0004] In various implementations, electronic devices use image sensors to capture images of a physical environment and process the images to provide various user experiences, such as providing feedback about objects detected in the images. However, in various implementations, the image sensors may be obscured, such as by dirt or condensation, thereby reducing the effectiveness of the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] So that the present disclosure can be understood by those of ordinary skill in the art, a more detailed description may be obtained with reference to aspects of some exemplary implementations, some of which are illustrated in the accompanying drawings.

[0006] Figure 1 is a perspective view of a head mounted device according to some specific implementations.

[0007] Figure 2 is a block diagram of an exemplary operating environment according to some implementations.

[0008] Figure 3 Various fields of view are shown according to some specific implementations.

[0009] 4A to 4F The device field of view of a physical environment during various time periods according to some implementations is shown.

[0010] Figure 5 is a flowchart representation of a method of detecting occlusion of an image sensor according to some specific implementations.

[0011] As is common practice, the various features shown in the drawings may not be drawn to scale. Therefore, for clarity, the sizes of the various features may be arbitrarily expanded or reduced. In addition, some drawings may not depict all components of a given system, method, or device. Finally, throughout the specification and drawings, similar reference numerals may be used to represent similar features. Summary of the invention

[0012] Various embodiments disclosed herein include devices, systems, and methods for detecting occlusion of an image sensor. In various embodiments, the method is performed by a device including one or more image sensors, one or more speakers, one or more processors, and non-volatile memory. The method includes receiving an image of a physical environment from one or more image sensors. The method includes detecting an occlusion that at least partially occludes one or more image sensors based on the image of the physical environment. The method includes classifying the occlusion into one of a plurality of occlusion types. The method includes playing an occlusion notification via one or more speakers based on the classification.

[0013] According to some specific implementations, a device includes one or more processors, non-volatile memory, and one or more programs; one or more programs are stored in the non-volatile memory and are configured to be executed by one or more processors. One or more programs include instructions for executing or causing the execution of any method described herein. According to some specific implementations, a non-volatile computer-readable storage medium stores instructions that, when executed by one or more processors of the device, cause the device to execute or cause the execution of any of the methods described herein. According to some specific implementations, a device includes: one or more processors, non-volatile memory, and a component for executing or causing the execution of any of the methods described herein. DETAILED DESCRIPTION

[0014] Many details are described in order to provide a thorough understanding of the example implementations shown in the accompanying drawings. However, the accompanying drawings only illustrate some example aspects of the present disclosure and should not be considered limiting. One of ordinary skill in the art will appreciate that other effective aspects and / or variations do not include all of the specific details described herein. In addition, well-known systems, methods, components, devices, and circuits are not described in detail so as not to obscure more relevant aspects of the example implementations described herein.

[0015] In various implementations, electronic devices use image sensors to capture images of a physical environment and process the images to provide various user experiences. In various implementations, occlusion of the image sensor may reduce the effectiveness of the user experience.

[0016] In various embodiments, the electronic device includes a display that provides a pass-through video in which a captured image is displayed to a user, possibly with changes such as the addition of virtual objects. Thus, by viewing the image, the user can quickly determine that the image sensor is obscured. However, in various embodiments, the electronic device does not include a display. Thus, the electronic device determines that the image sensor is at least partially obscured and provides audio (or other non-visual) feedback to the user that the image sensor is at least partially obscured.

[0017] Additionally, in various implementations, the feedback provides information about the occlusion. For example, in various implementations, the feedback indicates the type of occlusion. For example, in various implementations, the feedback indicates whether the occlusion is partial occlusion or full occlusion. As another example, in various implementations, the feedback indicates whether the occlusion is occluding hair, occluding water, or occluding dirt.

[0018] Figure 1 A perspective view of a head mounted device 150 according to some specific implementations is shown. The head mounted device 150 includes a frame 151 including two earphones 152, each earphone being configured to abut a respective outer ear of a user. The frame 151 also includes a front component 154 configured to reside in front of the user's field of view. Each earphone 152 includes an inwardly facing speaker 160 (e.g., facing inwardly, facing outwardly, facing downwardly, etc.) and an outwardly facing imaging system 170. In addition, the front component 154 includes a display 181 for displaying images to the user, an eye tracker 182 for determining the gaze direction or gaze point of the user (which may include one or more rear-facing image sensors configured to capture images of at least one eye of the user), and a scene tracker 190 (which may include one or more forward-facing image sensors configured to capture images of the physical environment) that may supplement the imaging system 170 of the earphones 152.

[0019] In various implementations, the head mounted device 150 does not include the front member 154. Thus, in various implementations, the head mounted device is embodied as a headphone device that includes a frame 151 having two earphones 152 each configured to surround a corresponding outer ear of a user, and a headband coupled to the earphones 152 and configured to rest on top of the user's head. In various implementations, each earphone 152 includes an inward facing speaker 160 and an outward facing imaging system 170.

[0020] In various embodiments, the headset device is a headband. Therefore, in various embodiments, the headset 150 (or its headset 150) is embodied as one or more earplugs or earphones. For example, the earplug includes a frame configured to be inserted into the outer ear. Specifically, in various embodiments, the frame is configured to be inserted into the outer ear of a human, a person and / or a user of the earplug. The earplug includes a speaker 160 and an imaging system 170 coupled to the frame, the speaker being configured to output sound, and the imaging system being configured to capture one or more images of the physical environment in which the earplug exists. In various embodiments, the imaging system 170 includes one or more cameras (or image sensors). The earplug also includes one or more processors coupled to the frame. The speaker 160 is configured to output sound based on audio data received from the one or more processors, and the imaging system 170 is configured to provide image data to the one or more processors. In various embodiments, the audio data provided to the speaker 160 is based on image data obtained from the imaging system 170.

[0021] As described above, in various embodiments, the earplug includes a frame configured for insertion into the outer ear. Specifically, in various embodiments, the frame is sized and / or shaped to be inserted into the outer ear. The frame includes a surface that rests in the intertragic notch, thereby preventing the earplug from falling vertically downward. In addition, the frame includes surfaces that abut the tragus and anti-tragus, thereby holding the earplug horizontally in place. When inserted, the speaker 160 of the earplug is directed toward the ear canal, and the imaging system 170 of the earplug is directed outward and exposed to the physical environment.

[0022] Although the head mounted device 150 is an exemplary device that can perform one or more of the methods described herein, it should be understood that other wearable devices having one or more speakers and one or more cameras can also be used to perform the methods. The wearable audio device can be embodied in other wired or wireless form factors, such as a head mounted device, an in-ear device, a circum-ear device, an on-ear device, an open device, a closed device, a bone conduction device, or other audio devices.

[0023] Figure 2 is a block diagram of an operating environment 20 according to some implementations. The operating environment 20 includes a headset 200. In various implementations, the headset 200 corresponds to Figure 1 The headset 200 includes a frame 201. In various implementations, the frame 201 is configured for insertion into an outer ear. The headset 200 includes one or more processors 210 coupled to the frame 201 and in various implementations within the frame 201. The headset 200 includes a memory 220 (e.g., non-volatile memory) coupled to the frame 201 and in various implementations coupled to the one or more processors 210 within the frame 201.

[0024] The headset 200 includes a speaker 230 coupled to the frame 201 and configured to output sound based on audio data received from the one or more processors 210. The headset 200 includes an imaging system 240, which is coupled to the frame 201 and configured to capture an image of the physical environment in which the headset 200 exists and provide image data representing the image to the one or more processors 210. In various specific implementations, the imaging system 240 includes one or more cameras 241A, 241B. In various specific implementations, different cameras 241A, 241B have different fields of view. For example, in various specific implementations, the imaging system 240 includes a forward-facing camera and a rear-facing camera. In various specific implementations, at least one of the cameras 241A includes a fisheye lens 242, for example to increase the size of the field of view of the camera 241A. In various specific implementations, the imaging system 240 includes a depth sensor 243. Thus, in various embodiments, for each pixel of a plurality of pixels representing a location in a physical environment, the image data includes a color (or grayscale) value for the location representing the amount and / or wavelength of light detected at the location and a depth value representing the distance from the headset 200 to the location.

[0025] In various implementations, the headset 200 includes a microphone 250 coupled to the frame 201 and configured to generate ambient sound data indicative of sounds in the physical environment. In various implementations, the headset 200 includes an inertial measurement unit (IMU) 260 coupled to the frame 201 and configured to determine movement and / or orientation of the headset 200. In various implementations, the IMU 260 includes one or more accelerometers and / or one or more gyroscopes. In various implementations, the headset 200 includes a communication interface 270 coupled to the frame and configured to transmit data and receive data from other devices. In various implementations, the communication interface 270 is a wireless communication interface.

[0026] The headset 200 includes one or more communication buses 204 within the frame 201 for interconnecting the various components described above and / or additional components that may be included with the headset 200.

[0027] In various implementations, operating environment 20 includes a second earphone 280, which may include any or all of the components of earphone 200. In various implementations, frame 201 of earphone 200 is configured for insertion into one outer ear of a user, and the frame of second earphone 200 is configured for insertion into the other outer ear of the user (e.g., by being a mirrored version of frame 201).

[0028] In various implementations, the operating environment 20 includes a controller device 290. In various implementations, the controller device 290 is a smart phone, a tablet, a laptop, a desktop computer, a set-top box, a smart TV, a digital media player, or a smart watch. The controller device 290 includes one or more processors 291 coupled to a memory 292, a display 293, and a communication interface 294 via one or more communication buses 214. In various implementations, the controller device 290 includes additional components, such as any or all of the components described above with respect to the headset 200.

[0029] In various implementations, the display 293 is configured to display images based on display data provided by the one or more processors 291. In contrast, in various implementations, the earphone 200 (and similarly, the second earphone 280) does not include a display, or at least does not include a display within the user's field of view when inserted into the user's outer ear.

[0030] In various implementations, the one or more processors 210 of the headset 200 generate audio data that is provided to the speaker 230 based on the image data received from the imaging system 240. In various implementations, the one or more processors 210 of the headset send the image data to the controller device 290 via the communication interface 270, the one or more processors of the controller device 290 generate the audio data based on the image data, and the headset 200 receives the audio data via the communication interface 270. In any set of implementations, the audio data is based on the image data.

[0031] Figure 3 Various fields of view are shown according to some specific implementations. The user field of view 301 of the user 30 generally extends about 300 degrees, with varying degrees of visual perception within this range. For example, excluding far peripheral vision, the user field of view 301 is only about 120 degrees, and the user field of view 301 including only foveal vision (or central vision) is only about 5 degrees.

[0032] In contrast, the system ( Figure 1The head mounted device 150 of the embodiment of the present invention may have a device field of view that includes a field of view outside the user field of view 301 of the user 30. For example, the system may include a forward-facing and outward-facing camera that includes a fisheye lens with a 180-degree field of view close to each ear of the user 30, and may have a device with a forward field of view 302 of approximately 300 degrees. In addition, the system may also include a rearward-facing and outward-facing camera that includes a fisheye lens with a 180-degree field of view close to each ear of the user 30, and may also have a device with a rearward field of view 303 of approximately 300 degrees. In various specific implementations, a system including multiple cameras close to each ear of the user may have a full 360-degree device field of view (e.g., including a forward device field of view 302 and a rearward device field of view 303). It should be understood that in various specific implementations, a camera (or combination of cameras) may have a smaller or larger field of view than the above examples.

[0033] The above-described system can perform a variety of functions. For example, in various implementations, when audio (e.g., music or an audiobook) is playing via a speaker, in response to detecting a particular gesture in an image captured by an imaging system (even a gesture performed outside of the user's field of view), the system can alter the playback of the audio (e.g., by pausing or changing the volume of the audio). For example, in various implementations, in response to detecting a gesture performed by a user near the user's ear: closing an open hand into a clenched fist, the system pauses the playback of the audio via the speaker.

[0034] As another example, in various implementations, when audio is played via a speaker, in response to detecting in an image captured by an imaging system that a person is attempting to engage the user in a conversation or otherwise talk to the user (even if the person is outside the user's field of view), the system may alter the playback of the audio. For example, in various implementations, in response to detecting that a person behind the user is attempting to talk to the user, the system reduces the volume of the audio played via the speaker and stops executing the active noise cancellation algorithm.

[0035] As another example, in various implementations, in response to detecting an object or event of interest in the physical environment in an image captured by the imaging system, the system generates an audio notification. For example, in various implementations, in response to detecting a person in the user's vicinity or outside the user's field of view trying to get the user's attention (e.g., by waving the person's arm), the device plays a warning notification (e.g., a sound similar to a person saying "Hi!") via a speaker. In various implementations, the system plays the warning notification spatially via two or more speakers so that the user perceives the warning notification as coming from the direction of the detected object.

[0036] As another example, in various implementations, in response to detecting an object or event of interest in a physical environment in an image captured by an imaging system, the system stores in memory an indication that a particular object was detected (which may be determined using the image from the imaging system) in association with a location where the object was detected (which may also be determined using the image from the imaging system) and the time at which the object was detected. In response to a user query (e.g., a voice query detected via a microphone), the system provides an audio response. For example, in response to detecting a water bottle in a user's office, the system stores an indication that the water bottle was detected in the office, and in response to a user query at a later time, "Where's my water bottle?", the device may generate audio that approximates a person saying "in your office".

[0037] As another example, in various implementations, in response to detecting an object in the physical environment approaching the user in an image captured by the imaging system, the system generates an audio notification. For example, in various implementations, in response to detecting that a car is approaching the user at a speed exceeding a threshold, the system plays a warning notification (e.g., a sound similar to the beeping of a car horn) via a speaker. In various implementations, the system plays the warning notification spatially via two or more speakers so that the user perceives the warning notification as coming from the direction of the detected object.

[0038] For example, in various implementations, in response to detecting a gesture indicating an actionable item, the system performs an action associated with the actionable item. For example, in various implementations, in response to detecting that a user swipes over a phone number, the system calls the phone number.

[0039] 4A to 4F The device field of view 400 of an outdoor physical environment is shown during a series of time periods in various implementations. In various implementations, each time period is an instant, a fraction of a second, a few seconds, a few hours, a few days, or any length of time.

[0040] The device field of view 400 includes the user field of view 401 and a left portion 402A outside the user field of view 401 and a right portion 402B outside the user field of view 401. In various specific implementations, the device field of view 400 does not include all of the user field of view 401. For example, in various specific implementations, the device field of view 400 includes a first portion of the user field of view 401 and does not include a second portion of the user field of view 401. In various specific implementations, the device field of view 400 does not include any user field of view 401. The device field of view 400 includes a track 411 of the user walking in the user field of view 401.

[0041] Figure 4AThe device field of view 400 during a first time period is shown. During the first time period, a hat 421 worn by a user blocks the upper portion of the device field of view 400. The device detects occlusion of the device field of view 400 and classifies the occlusion. In various specific implementations, the device classifies the occlusion into one of a plurality of occlusion levels. In various specific implementations, the occlusion level includes a first level below the notification threshold, a second level above the notification threshold and below the processing threshold, and a third level above the processing threshold. When the occlusion level is below the notification threshold (e.g., the first level), the device does not automatically provide feedback about the occlusion to the user. When the occlusion level is above the notification threshold (e.g., the second level or the third level), the device provides feedback about the occlusion to the user, for example, by audibly playing an occlusion notification. When the occlusion level is below the processing threshold (e.g., the first level and the second level), the device continues to process the image of the physical environment to provide various user experiences. When the occlusion level is above the processing threshold (e.g., the third level), the device stops processing the image of the physical environment until the occlusion is cleared (at least to a level below the processing threshold).

[0042] In various implementations, the device classifies the occlusion based on the transmittance of the occlusion. For example, in various implementations, the device classifies the occlusion as an opaque occlusion (where at least a portion of the device field of view 400 is obscured by an opaque object (such as clothing, hair, or dirt)) or a transparent occlusion (where at least a portion of the device field of view 400 is distorted by a transparent object (such as fog, condensation, or water droplets)).

[0043] In various implementations, the device classifies occlusions based on the object causing the occlusion. For example, in various implementations, the device classifies occlusions as clothing occlusions, hair occlusions, dirt occlusions, or water occlusions. In various implementations, the device classifies occlusions based on one or more heuristics. For example, a transparent occlusion is more likely to be classified as a water occlusion. As another example, an occlusion that blocks an adjacent portion of the device's field of view 400 is more likely to be classified as a clothing occlusion. As another example, an occlusion that moves between images of a physical environment is more likely to be classified as a hair occlusion. In various implementations, the device uses a neural network to classify occlusions based on objects.

[0044] exist Figure 4A In the example, the device classifies the occlusion as a clothing occlusion with an occlusion level below the notification threshold. Therefore, the device does not automatically provide an occlusion notification to the user. However, in various implementations, in response to a user query (e.g., verbally asking "How is your view?"), the device plays a verbal response based on the classification of the occlusion (e.g., "Your hat partially blocks your view.").

[0045] Figure 4BThe device field of view 400 is shown during a second time period following the first time period. During the second time period, the user has pulled down hat 421. As a result, hat 421 obscures a majority of the device field of view 400. The device detects the occlusion and classifies the occlusion as a clothing occlusion with an occlusion level above a notification threshold. Thus, in response to detecting the occlusion, the device plays an occlusion notification. In various embodiments, the occlusion notification is a series of one or more tones or chimes. In various embodiments, the occlusion notification is a verbal notification (e.g., "Your hat is blocking your view.") In various embodiments, the verbal notification provides instructions to clear the occlusion (e.g., "Please adjust your hat.")

[0046] Figure 4C The device field of view 400 is shown during a third time period after the second time period. During the third time period, the user has removed the hat 421, but the user's hair 422 partially obscures the right portion 402B of the device field of view 400. The device detects the occlusion and classifies the occlusion as a hair occlusion with an occlusion level above the notification threshold. Therefore, in response to detecting the occlusion, the device plays an occlusion notification. In various embodiments, the occlusion notification is a series of one or more tones or sounds. In various embodiments, the occlusion notification is a verbal notification (e.g., "Your hair is blocking your view.") In various embodiments, the verbal notification provides instructions to clear the occlusion (e.g., "Please pull your hair back.")

[0047] In various implementations, in response to detecting an occlusion of the left portion 402A or the right portion 402B (corresponding to the left image sensor or the right image sensor) of the device field of view 400, the device plays the occlusion notification in the corresponding ear of the user (corresponding to the left speaker or the right speaker) without playing the occlusion notification in the other ear of the user. Figure 4C In various embodiments, in response to detecting an occlusion of the right side portion 402B of the device field of view 400, the device plays a verbal notification (e.g., "hair detected") using the right speaker in the user's right ear, rather than playing a verbal notification using the left speaker in the user's left ear.

[0048] Figure 4DThe device field of view 400 is shown during a fourth time period after the third time period. During the fourth time period, the user has tucked the user's hair 422 behind the user's ears, clearing all occlusions of the device field of view 400. The device detects that there is no occlusion of the device field of view. In response to detecting the occlusion of the device field of view 400 and subsequently detecting that there is no occlusion of the device field of view 400, the device detects the clearing of the occlusion. In various embodiments, in response to detecting the clearing of the occlusion, the device plays a clear notification. In various embodiments, the clear notification is a series of one or more tones or chimes. In various embodiments, the occlusion notification and the clear notification are different. However, in various embodiments, the occlusion notification and the clear notification are complementary. For example, in various embodiments, the occlusion notification is a descending arpeggio and the clear notification is an ascending arpeggio (e.g., the same tone). In various embodiments, the occlusion notification includes two tones in the minor interval and the clear notification includes two tones in the major interval. In various embodiments, the clear notification is a verbal notification (e.g., "The occlusion has been cleared")

[0049] Figure 4E The device field of view 400 is shown during a fifth time period following the fourth time period. During the fifth time period, dirt 423 obscures the device field of view 400. The device detects the occlusion and classifies the occlusion as a dirt occlusion with an occlusion level above a notification threshold. Thus, in response to detecting the occlusion, the device plays an occlusion notification. In various implementations, the occlusion notification is a series of one or more tones or chimes. In various implementations, the occlusion notification is a verbal notification (e.g., "The camera is dirty.") In various implementations, the verbal notification provides instructions to clear the occlusion (e.g., "Please clean the camera lens.")

[0050] Figure 4F The device field of view 400 is shown during a sixth time period following the fifth time period. During the sixth time period, the user has cleaned dirt 423 from the device field of view, but water 424 used in the cleanup obscures the device field of view 400. The device detects the occlusion and classifies the occlusion as a water occlusion with an occlusion level above a notification threshold. Therefore, in response to detecting the occlusion, the device plays an occlusion notification. In various embodiments, the occlusion notification is a series of one or more tones or sounds. In various embodiments, the occlusion notification is a verbal notification (e.g., "The camera is wet.") In various embodiments, the verbal notification provides instructions to clear the occlusion (e.g., "Please dry the camera lens.")

[0051] Figure 5 is a flowchart representation of a method 500 for detecting occlusion of an image sensor according to some specific implementations. In various specific implementations, the method 500 is performed by a device including one or more image sensors, one or more speakers, one or more processors, and non-transitory memory (e.g., Figure 1 Head mounted device 150 or Figure 2 In various embodiments, the method 500 is performed by a device without a display. In various embodiments, the method 500 is performed by a device with a display. In various embodiments, the method 500 is performed by a device with a display. Figure 1 Head mounted device 150 or Figure 2 The headset 200) is combined with a peripheral device (e.g., Figure 2 The method 500 is performed by a controller device 290 of the computer system. In various specific implementations, the method 500 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In various specific implementations, the method 500 is performed by a processor that executes instructions (e.g., code) stored in a non-transitory computer-readable medium (e.g., a memory).

[0052] Method 500 begins at block 510, where a device receives an image of a physical environment from one or more image sensors. 4A to 4F , the image of the physical environment has a device field of view 400, which includes a user field of view 401 and a left portion 402A outside the user field of view 401 and a right portion 402B outside the user field of view 401. In various implementations, the device field of view includes one or more areas outside the user field of view and may not include the user field of view or only include a portion of the user field of view.

[0053] Method 500 continues at block 520, where the device detects an occlusion that at least partially occludes one or more image sensors based on the image of the physical environment. Figure 4A and Figure 4B , the device detects that the hat 421 at least partially blocks the device field of view 400. As another example, in Figure 4C , the device detects that hair 422 at least partially obscures right portion 402B of device field of view 400. As another example, in Figure 4E In , the device detects that dirt 423 at least partially blocks the device field of view 400. As another example, in Figure 4F , the device detects that water 424 at least partially obscures the device field of view 400.

[0054] In various implementations, the device sends an image of the physical environment to a peripheral device, which detects an occlusion in the image of the physical environment (and, in various implementations, classifies the occlusion). In response to the detection, the peripheral device sends an indication of the detection (and, in various implementations, the classification) to the device. Thus, in various implementations, detecting the occlusion (and, in various implementations, classifying the occlusion) includes sending an image of the physical environment to the peripheral device, and receiving an indication of the detection (and, in various implementations, the classification) from the peripheral device.

[0055] For example, in various implementations, the indication is an audio signal to be played by the device. As another example, in various implementations, the indication includes a parameter indicating detection (and in various implementations, classification). In response to receiving the parameter indicating detection (and in various implementations, classification), the device generates an audio signal to be played by the device.

[0056] In various implementations, the image of the physical environment has a device field of view that is different from the user's field of view, and the occlusion is detected outside of the user's field of view. Figure 4C , the device detects occlusion caused by hair 422 in the right portion 402B of the device field of view 400 outside the user field of view 401.

[0057] In various implementations, the device determines that the occlusion is detected entirely outside of the user's field of view. Thus, in various implementations, determining that the occlusion is detected outside of an area of ​​the user's field of view includes determining that the occlusion is detected outside of the user's field of view. As another example, in various implementations, the device determines that the occlusion is detected within the user's field of view, but outside of a portion of the user's field of view, such as a foveal portion of the user's field of view or a peripheral portion of the user's field of view. Thus, in various implementations, determining that the occlusion is detected outside of an area of ​​the user's field of view includes determining that the occlusion is detected outside of a portion of the user's field of view.

[0058] In various implementations, to determine that the occlusion is detected outside the area of ​​the user's field of view, the device estimates the area of ​​the user's field of view and determines that the occlusion is detected outside the estimated area, e.g., not within the estimated area. Thus, in various implementations, determining that the occlusion is detected outside the area of ​​the user's field of view includes estimating the area of ​​the user's field of view.

[0059] In various implementations, the device includes an eye tracker. For example, Figure 1The head mounted device 150 includes an eye tracker 182. In various implementations, based on information from the eye tracker, the device estimates the area of ​​the user's field of view as an area around the location where the user is looking. For example, the foveal portion of the user's field of view can be an area of ​​approximately 5 degrees (or some other value) around the location where the user is looking. As another example, the near peripheral portion of the user's field of view can be approximately 30 degrees (or some other value) around the location where the user is looking. Therefore, in various implementations, estimating the area of ​​the user's field of view includes determining an area around the location where the user is looking.

[0060] In various implementations, the device includes multiple image sensors. For example, Figure 1 The head mounted device 150 includes two outward facing imaging systems 170. As another example, in combination with Figure 2 The headset 200 of the headset 280 includes two imaging systems 240. In various embodiments, an image of the physical environment is generated by combining two images of the physical environment from two image sensors, each of the two images having two different fields of view. In various embodiments, the device estimates the area of ​​the user's field of view as an area around a position (e.g., a center) in the area where the different fields of view overlap. For example, a near peripheral portion of the user's field of view may be approximately 30 degrees (or some other value) around the center of the area where the different fields of view overlap. Thus, in various embodiments, estimating the area of ​​the user's field of view includes determining an area around a position in the overlapping area of ​​the images of the physical environment.

[0061] In various embodiments, the device determines the area of ​​the user's field of view based on user feedback. For example, during a calibration procedure, the user may define the area of ​​the user's field of view by placing a finger at the edge of the user's field of view area. In response to detecting the user's finger, the device determines that the area of ​​the user's field of view is to the left or right of the finger. As another example, the user may define the user's field of view by placing a finger at the edge of the user's field of view. In response to detecting the user's finger, the device determines the middle peripheral portion of the user's field of view as half (or some other portion) of the image of the physical environment to the left or right of the finger. Therefore, in various embodiments, estimating the area of ​​the user's field of view includes receiving user feedback about the user's field of view.

[0062] The method 500 continues at block 530, where the device classifies the occlusion into one of a plurality of occlusion types. In various implementations, the plurality of occlusion types include a plurality of object type occlusion types. Each of the plurality of object type occlusion types is associated with an object type of an object causing the occlusion. In various implementations, the plurality of object type occlusion types include one or more of a clothing occlusion type, a hair occlusion type, a dirt occlusion type, or a water occlusion type. For example, in Figure 4C, the device classifies the occlusion caused by hair 422 as hair occlusion, e.g., occlusion having a hair occlusion type. As another example, in Figure 4F , the device classifies the occlusion 424 caused by water as a water occlusion, eg, an occlusion having a water occlusion type.

[0063] In various implementations, clothing shielding can be provided by a hat (such as Figure 4A or Figure 4B In various implementations, clothing occlusion may be caused by a collar of a shirt or jacket. In various implementations, dirt occlusion type may be caused by mud, sand, or other debris. In various implementations, water occlusion may be caused by water droplets, condensed water droplets, or mist.

[0064] In various implementations, classifying the occlusion (e.g., classifying the occlusion as one of a plurality of object type occlusion types) includes applying a neural network to the image of the physical environment. In various implementations, the neural network includes a set of interconnected nodes. In various implementations, each node includes an artificial neuron that implements a mathematical function, where each input value is weighted according to a set of weights, and the sum of the weighted inputs is passed through an activation function, typically a nonlinear function such as a sigmoid, piecewise linear function, or step function, to produce an output value. In various implementations, the neural network is trained using training data to set the weights.

[0065] In various implementations, the neural network comprises a deep learning neural network. Thus, in some implementations, the neural network comprises a plurality of layers (of nodes) between an input layer (of nodes) and an output layer (of nodes). In various implementations, the neural network receives as input an image of a physical environment. In various implementations, the neural network provides as output an object type occlusion type.

[0066] In various implementations, the plurality of occlusion types include an opaque occlusion type and a transparent occlusion type. Thus, in various implementations, classifying the occlusion as one of the plurality of object types includes classifying the occlusion as one of the opaque occlusion type or the transparent occlusion type. For example, in Figure 4B , the device classifies the occlusion caused by hat 421 as an opaque occlusion, e.g., an occlusion with an opaque occlusion type. As another example, in Figure 4F , the device classifies the occlusion caused by water 424 as a transparent occlusion, eg, an occlusion having a transparent occlusion type.

[0067] In various implementations, the plurality of occlusion types includes a first occlusion type associated with an occlusion level below a threshold and a second occlusion type associated with an occlusion level above the threshold. Figure 4A, the device classifies the occlusion caused by hat 421 as a first occlusion type associated with an occlusion level below a notification threshold. As another example, in Figure 4B , the device classifies the occlusion caused by hat 421 as a second occlusion type associated with an occlusion level above a notification threshold.

[0068] In various implementations, the one or more image sensors include a left image sensor and a right image sensor, and the plurality of occlusion types include a left image sensor occlusion type associated with occlusion of the left image sensor and a right image sensor occlusion type associated with occlusion of the right image sensor. Figure 4C , the device classifies the occlusion caused by hair 422 as a right image sensor occlusion, e.g., an occlusion with a right image sensor occlusion type.

[0069] The method 500 continues at block 540, where the device plays the occlusion notification via one or more speakers based on the classification. In various implementations, as described above, the plurality of occlusion types include a first occlusion type associated with an occlusion level below a threshold and a second occlusion type associated with an occlusion level above the threshold. In various implementations, playing the occlusion notification based on the classification includes playing the occlusion notification in response to classifying the occlusion as the second occlusion type. For example, in Figure 4A In response to classifying the occlusion caused by the hat 421 as the first occlusion type, the device does not automatically play the occlusion notification. Figure 4B In response to classifying the occlusion caused by the hat 421 as the second occlusion type, the device automatically plays an occlusion notification.

[0070] In various implementations, as described above, the plurality of occlusion types include a left image sensor occlusion type associated with occlusion of the left image sensor and a right image sensor occlusion type associated with occlusion of the right image sensor. In various implementations, the one or more speakers include a left speaker and a right speaker, and playing the occlusion notification based on the classification includes playing the occlusion notification in one of the left speaker or the right speaker (e.g., a speaker corresponding to the occluded image sensor). For example, in Figure 4C , in response to detecting that the occlusion is caused by hair 422 in the right portion 402B of the device's field of view 400, the device plays an occlusion notification via the right speaker in the user's right ear.

[0071] In various embodiments, the occlusion notification includes one or more tones. In various embodiments, the occlusion notification is a verbal notification. For example, Figure 4B In response to detecting the occlusion caused by hat 421, the device plays a verbal notification saying "Your hat is blocking your view." As another example, in Figure 4CIn response to detecting an occlusion caused by hair 422, the device plays a verbal notification saying "Your hair is blocking your view." In various implementations, the verbal notification includes instructions for clearing the occlusion. For example, in Figure 4B In response to detecting the occlusion caused by hat 421, the device plays a verbal notification saying "Please adjust your hat." As another example, in Figure 4C In response to detecting an occlusion caused by hair 422, the device plays a verbal notification saying "Please pull your hair back."

[0072] Therefore, different verbal notifications are played based on the classification.Therefore, in various specific implementations, playing the occlusion notification based on the classification includes playing the verbal notification based on the classification.

[0073] In various implementations, method 500 further includes, after playing the occlusion notification, in response to detecting the clearing of the occlusion, playing a clearing notification via one or more speakers. Thus, in various implementations, method 500 includes, after playing the occlusion notification, receiving a second image of the physical environment from the image sensor, detecting the clearing of the occlusion based on the second image of the physical environment, and in response to detecting the clearing of the occlusion, playing a clearing notification via one or more speakers.

[0074] In various implementations, the clear notification is a series of tones. In various implementations, the occlusion notification and the clear notification are distinct. However, in various implementations, the occlusion notification and the clear notification are complementary. For example, in various implementations, the occlusion notification is a descending arpeggio and the clear notification is an ascending arpeggio (e.g., the same tones). In various implementations, the occlusion notification includes two tones in the minor interval and the clear notification includes two tones in the major interval. In various implementations, the clear notification is a verbal notification (e.g., "occlusion cleared")

[0075] Although various aspects of specific implementations within the scope of the appended claims are described above, it should be apparent that the various features of the above-mentioned specific implementations can be embodied in a variety of forms, and any specific structures and / or functions described above are merely illustrative. Based on this disclosure, it should be understood by those skilled in the art that the aspects described herein can be implemented independently of any other aspects, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement a device and / or a method can be practiced. In addition, in addition to or different from one or more aspects set forth herein, other structures and / or functions can be used to implement such a device and / or such a method can be practiced.

[0076] It will also be understood that, although the terms "first", "second", etc. may be used to describe various elements in this article, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, a first node may be referred to as a second node, and similarly, a second node may be referred to as a first node, which changes the meaning of the description, as long as all occurrences of the "first node" are consistently renamed and all occurrences of the "second node" are consistently renamed. Both the first node and the second node are nodes, but they are not the same node.

[0077] The terms used herein are only for describing specific implementations and are not intended to limit the claims. As used in the description of this specific implementation and the appended claims, the singular forms of "one", "a kind of" and "the" are intended to also cover the plural forms, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" used herein refers to and covers any and all possible combinations of one or more items in the associated listed items. It will also be understood that the term "comprising" when used in this specification specifies the presence of stated features, integers, steps, operations, elements and / or parts, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts, and / or their groupings.

[0078] As used herein, the term “if” may be interpreted to mean “when the antecedent is true” or “when the antecedent is true” or “in response to determining” or “upon determining” or “in response to detecting” that the antecedent is true, depending on the context. Similarly, the phrase “if it is determined that [the antecedent is true]” or “if [the antecedent is true]” or “when [the antecedent is true]” is interpreted to mean “upon determining that the antecedent is true” or “in response to determining” or “upon determining” that the antecedent is true or “when detecting that the antecedent is true” or “in response to detecting” that the antecedent is true, depending on the context.

Claims

1. A method, comprising: At a device including one or more image sensors, one or more speakers, one or more processors, and non-transitory memory: receiving an image of a physical environment from the one or more image sensors; detecting an occlusion at least partially occluding the one or more image sensors based on the image of the physical environment; classifying the occlusion into one of a plurality of occlusion types; as well as An occlusion notification is played via the one or more speakers based on the classification. The method of claim 1 , wherein the plurality of occlusion types comprises a plurality of object type occlusion types. The method of claim 2 , wherein the plurality of object type occlusion types comprises a hair occlusion type. 4 . The method of claim 2 , wherein the plurality of object type occlusion types comprises at least one of a dirt occlusion type or a water occlusion type. 5 . The method according to claim 1 , wherein the plurality of occlusion types include an opaque occlusion type and a transparent occlusion type.

6. The method of any one of claims 1 to 5, wherein the plurality of occlusion types comprises a first occlusion type associated with an occlusion level below a threshold and a second occlusion type associated with an occlusion level above the threshold. 7 . The method of claim 6 , wherein playing the occlusion notification based on the classification comprises playing the occlusion notification in response to classifying the occlusion as the second occlusion type.

8. The method according to any one of claims 1 to 7, wherein the one or more image sensors include a left image sensor and a right image sensor, and wherein the multiple occlusion types include a left image sensor occlusion type associated with occlusion of the left image sensor and a right image sensor occlusion type associated with occlusion of the right image sensor.

9. The method of claim 8, wherein the one or more speakers include a left speaker and a right speaker, and wherein playing the occlusion notification based on the classification includes playing the occlusion notification in one of the left speaker or the right speaker.

10. The method of any one of claims 1 to 9, wherein the occlusion notification comprises one or more tones.

11. The method according to any one of claims 1 to 10, wherein the occlusion notification is a verbal notification. The method of claim 11 , wherein the verbal notification includes instructions for clearing the occlusion.

13. The method of any one of claims 1 to 12, wherein classifying the occlusion comprises applying a neural network to the image of the physical environment.

14. The method according to any one of claims 1 to 13, further comprising, after playing the occlusion notification: receiving a second image of the physical environment from the one or more image sensors; detecting the clearing of the occlusion based on the second image of the physical environment; as well as In response to detecting the clearing of the obstruction, a clearing notification is played via the one or more speakers.

15. The method of any one of claims 1 to 14, wherein the image of the physical environment has a device field of view that is larger than a user field of view, and wherein the occlusion is detected outside of a region of the user field of view.

16. A device, comprising: one or more image sensors; one or more speakers; one or more processors; non-transitory memory; and One or more programs stored in the non-transitory memory, which, when executed by the one or more processors, cause the device to perform any of the methods according to claims 1 to 15.

17. A non-volatile memory storing one or more programs which, when executed by one or more processors of a device comprising one or more image sensors and one or more speakers, cause the device to perform any of the methods of claims 1 to 15.

18. A device, comprising: one or more image sensors; one or more speakers; one or more processors; non-transitory memory; and Means for causing the apparatus to perform any of the methods according to claims 1 to 15.

19. A device, comprising: one or more image sensors; one or more speakers; non-transitory memory; and One or more processors configured to: receiving an image of a physical environment from the one or more image sensors; detecting an occlusion at least partially occluding the one or more image sensors based on the image of the physical environment; classifying the occlusion into one of a plurality of occlusion types; as well as An occlusion notification is played via the one or more speakers based on the classification.

Images