Assists visual motion perception in the extreme periphery

Abstract

Disclosed herein include systems, devices, computer-readable media, and methods for enhancing human visual motion perception in the periphery (e.g., far periphery or extreme periphery) or improving user safety using, for example, at least one auditory stimulus having a descending pitch.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application No. 62 / 826,540 filed on March 29, 2019, U.S. Provisional Patent Application No. 62 / 879,971 filed on July 29, 2019, and U.S. Provisional Patent Application No. 62 / 880,557 filed on July 30, 2019; the contents of each of which are incorporated herein by reference in their entirety. Technical Field

[0003] The present disclosure relates generally to the field of aiding visual motion perception, and more particularly to aiding visual motion perception in the periphery, such as the far periphery and extreme periphery. Background Art

[0004] The presence of auditory stimulation biases the perception of motion in the periphery more than in the fovea. Auditory perception becomes critical when visual reliability decreases. Detecting extreme peripheral visual motion from behind may be important because visual ambiguity is very high and detecting such motion is ecologically critical. Summary of the invention

[0005] Disclosed herein are embodiments of systems for improving visual motion perception. In some embodiments, the system includes: a non-transitory memory configured to store executable instructions; an audio output device; a hardware processor (or processors, such as virtual processors) in communication with the non-transitory memory, the image capture device, and the audio output device. A hardware processor programmed with executable instructions: receives a plurality of images captured by the image capture device. The processor may be programmed by the executable instructions to: determine the position and trajectory of an object relative to a user at a first time using the plurality of images. The position and trajectory of the object relative to the user at the first time may indicate that the object has a probability exceeding a probability threshold of passing the user at a passing time. At the first time and / or the passing time, the object may not be in the user's field of view, or may be in the far periphery of the user's field of view. The processor may be programmed by the executable instructions to: determine an auditory stimulus and an output time of the output auditory stimulus based on the position and trajectory of the object relative to the user at the first time. The output time may be the passing time, or the output time may be between the first time and the passing time. The feature of the auditory stimulus may be selected from a group including a descending tone. The processor may be programmed by the executable instructions to: cause the audio output device to output the auditory stimulus to the user at the output time. As a result of outputting the auditory stimulation to the user at the output time, the possibility that the user visually perceives an object located in the far periphery of the user's visual field at the output time or immediately after the output time may increase.

[0006] Embodiments of systems for improving visual motion perception disclosed herein. In some embodiments, the system includes: a non-transitory memory configured to store executable instructions; an audio output device; and a hardware processor (or processor, such as a virtual processor) in communication with the non-transitory memory, the image capture device, and the audio output device. The hardware processor can be programmed by executable instructions to: receive a plurality of images captured by the image capture device. The processor can be programmed by executable instructions to: use the plurality of images to determine that at a passing time, an object is passing a user, or has a probability exceeding a threshold of probability of passing a user. The object may not be in the user's field of view, or may be at the far periphery of the user's field of view when passing. The processor can be programmed by executable instructions to: determine an auditory stimulus having a feature selected from a group including a descending tone. The processor can be programmed by executable instructions to: cause the audio output device to output an auditory stimulus to the user at the output time. As a result of outputting the auditory stimulus to the user at the output time, the possibility that the user visually perceives an object located at the far periphery of the user's field of view at the output time or immediately after the output time can be increased.

[0007] In some embodiments, to receive the plurality of images, the hardware processor is programmed with executable instructions to receive sensory data comprising a plurality of images captured by one or more sensors. The one or more sensors may include an image capture device, a radio detection and ranging (radar), a light detection and ranging (lidar), an audio capture device, or a combination thereof. To determine the position and trajectory of an object relative to the user at a first time, the hardware processor may be programmed with executable instructions to use the sensor data to determine the position and trajectory of the object relative to the user at a first time. To determine that an object is passing by the user at the time of passing, or has a likelihood exceeding a likelihood threshold, the hardware processor may be programmed with executable instructions to use the sensor data to determine that an object is passing by the user, or has a likelihood exceeding a likelihood threshold.

[0008] In some embodiments, the system includes one or more sensors or one or more of the one or more sensors. The system may include an image capture device. In some embodiments, to receive sensor data, the hardware processor is programmed with executable instructions to: receive sensor data captured by one or more sensors from a second system. In some embodiments, the hardware processor is programmed with executable instructions to: transmit sensor data, the position and trajectory of an object relative to a user, and / or an indication that an object is passing the user or has a likelihood exceeding a threshold of likelihood of passing the user at a passing time to a third system at a first time. In some embodiments, a helmet, a car audio system, or a wearable speaker system or device is associated with, included in, or contained in the system or a portion thereof.

[0009] In some embodiments, the object is a first motor vehicle. The user may be in or may be riding in a second motor vehicle. In some embodiments, the object is at an extreme periphery of the user's field of view at a first time. In some embodiments, the far periphery of the user's field of view includes about 60° to about 110° temporarily away from the user's nose and toward the temple. The extreme periphery of the user's field of view may include about 90° to 110° temporarily away from the user's nose and toward the temple.

[0010] In some embodiments, the hardware processor is programmed with executable instructions to: determine, based on the position and trajectory of the object relative to the user at the first time, that the object has a likelihood of exceeding a threshold of likelihood of the user passing by at the passing time. In some embodiments, the threshold of likelihood of the user passing by the object at the passing time is at least 50%, 60%, 70%, 80%, 90%, 95%, or 99%. In some embodiments, the position and trajectory of the object relative to the user at the first time indicates that the object has a likelihood of exceeding the threshold of likelihood of the user passing by at the passing time within at most 1 meter, 2 meters, 5 meters, or 10 meters of the user.

[0011] In some embodiments, the characteristics of the auditory stimulation are selected from the group consisting of decreasing pitch, increasing loudness, higher number of frequencies, higher frequencies, regularity of angular differences, specificity of angular differences, sound shadowing, larger angular size, loudness asymmetry of the user's ears, head related transfer function of approximately 90°, vertical position below the user's horizontal line, or a combination thereof. In some embodiments, the change in the characteristics of the auditory stimulation is fastest or highest at or about the elapsed time. In some embodiments, the auditory stimulation comprises a narrow range of frequencies or pure tones. In some embodiments, the auditory stimulation comprises music, speech, or a combination thereof.

[0012] In some embodiments, the first time is the time when the plurality of images or a portion thereof are captured. The first time may be the time after the plurality of images or a portion thereof are captured. The first time may be the time immediately after the plurality of images or a portion thereof are captured. In some embodiments, the output time is the elapsed time. The output time may be immediately before the elapsed time. The output time may be between the first time and the elapsed time.

[0013] In some embodiments, the position and trajectory of the object relative to the user indicates that the object may pass by the user on the left or right side of the user when passing by. The audio output device may include a left audio output device and a right audio output device, which are configured to output auditory stimulation to the left ear and right ear of the user, respectively. In order to cause the audio output device to output the auditory stimulation to the user at the output time, the hardware processor may be programmed by the executable instructions to cause the left audio output device or the right audio output device to output the auditory stimulation to the left ear or right ear of the user, respectively, at the output time.

[0014] In some embodiments, auditory stimulation output to the user at the output time may cause the user to visually perceive objects in the far periphery of the user's field of view at or immediately after the output time by at least 10%, 20%, 30%, 40%, or 50% more likely.

[0015] Disclosed herein are embodiments of methods for improving visual motion perception. In some embodiments, the method is under the control of a hardware processor (or processor, such as a virtual processor), and includes: receiving sensor data captured by one or more sensors. The method may include: using the sensor data to determine the position and trajectory of an object relative to a subject at a first time. The position and trajectory of the object relative to the subject at a first time may indicate that the object has a probability of exceeding a threshold of probability of moving relative to the subject within a threshold distance of the subject at a second time. The object may not be in the subject's field of view, or may be in the periphery of the subject's field of view at the first time. The method may include: determining an auditory stimulus based on the position and trajectory of the object relative to the subject at the first time. The features of the auditory stimulus may correspond to the auditory features of the object moving relative to the subject within a threshold distance of the subject. The method may include: outputting the auditory stimulus to the subject at an output time. The method may result in an increased probability that the subject visually perceives an object in the periphery of the subject's field of view at the output time or immediately after the output time.

[0016] Disclosed herein are embodiments of methods for improving visual motion perception. In some embodiments, the method is under the control of a hardware processor (or processor, such as a virtual processor) and includes: receiving sensor data captured by one or more sensors. The method may include: using the sensor data, determining that an object is or has a probability above a probability threshold that the object is moving relative to the subject within a threshold distance of the subject. The method may include: determining an auditory stimulus whose characteristics correspond to auditory characteristics of an object moving relative to the subject within the threshold distance of the subject. The method may include: outputting the auditory stimulus to the subject at an output time. The method may result in an increased probability that the subject visually perceives an object in the periphery of the subject's field of view at the output time or a time immediately after the output time.

[0017] In some embodiments, one or more sensors include image capture devices, radio detection and ranging (radar), light detection and ranging (lidar), audio capture devices, or combinations thereof. In some embodiments, the sensor data includes the position, speed, and direction of the object captured by one or more telemetry and positioning sensors associated with the object. The one or more sensors may include one or more telemetry and positioning sensors associated with the object. In some embodiments, the sensor data includes the position, speed, and direction of the object captured by one or more telemetry and positioning sensors associated with the subject. The one or more sensors may include one or more telemetry and positioning sensors associated with the subject. The one or more telemetry and positioning sensors associated with the object may include a global positioning system (GPS) sensor associated with the object. The one or more telemetry and positioning sensors associated with the subject may include a global positioning system sensor (GPS) associated with the subject.

[0018] In some embodiments, the helmet, car audio system, or wearable speaker system or device is associated with, includes, or is contained in one or more sensors. Causing the auditory stimulus to be output may include causing an audio output device of the helmet, car audio system, or wearable speaker system or device to output the auditory stimulus at an output time.

[0019] In some embodiments, the object is a first motor vehicle. In some embodiments, the subject is in or riding in a second motor vehicle. In some embodiments, the object is at the far periphery or extreme periphery of the subject's field of view at the first time and / or the second time. In some embodiments, the far periphery of the user's field of view includes about 60° to about 110° away from the subject's nose and toward the temple in time. The extreme periphery of the user's field of view may include about 90° to 110° away from the subject's nose and toward the temple in time.

[0020] In some embodiments, the method includes: based on the position and trajectory of the object relative to the subject at the first time, determining that the object has a probability of exceeding a probability threshold of moving relative to the subject within a threshold distance of the subject at the second time. In some embodiments, the position and trajectory of the object relative to the subject at the first time can indicate that the object has a high probability of passing or hitting the subject at the second time. In some embodiments, the position and trajectory of the object relative to the subject indicates that the object may pass or hit the subject on the left or right side of the object at the second time. Causing the auditory stimulus to be output to the subject at the output time may include causing the auditory stimulus to be output to the left ear or right ear of the subject, respectively, at the output time.

[0021] In some embodiments, the threshold probability that the object moves relative to the subject within the threshold distance of the subject at the second time is at least 50%, 60%, 70%, 80%, 90%, 95% or 99%. In some embodiments, the threshold distance is at most 0 meters, 1 meter, 2 meters, 5 meters or 10 meters.

[0022] In some embodiments, the auditory stimulation is characterized by a descending pitch. In some embodiments, the auditory stimulation is characterized by a descending pitch, increased loudness, a higher number of frequencies, higher frequencies, regularity of angle differences, specificity of angle differences, sound shadows, larger angle sizes, loudness asymmetry of the subject's ears, a head-related transfer function of about 90 °, a vertical position below the subject's horizontal line, or a combination thereof. In some embodiments, the change in the auditory stimulation feature is the fastest or highest at or about the elapsed time. In some embodiments, the auditory stimulation comprises a narrow range of frequencies or pure tones. In some embodiments, the auditory stimulation comprises music, speech, or a combination thereof. In some embodiments, determining the auditory stimulation comprises determining the auditory stimulation and output time based on the position and trajectory of the object relative to the subject at the first time.

[0023] In some embodiments, the first time is the time when the sensor data or a portion thereof is captured. The first time may be a time after the sensor data or a portion thereof is captured. The first time may be a time immediately after the sensor data or a portion thereof is captured. In some embodiments, the output time is the second time. The output time may be immediately before the second time. The output time may be between the first time and the second time.

[0024] In some embodiments, the method results in at least a 10%, 20%, 30%, 40%, or 50% increase in the likelihood that the subject will visually perceive an object in the periphery of the user's field of view at or immediately after the output time.

[0025] Embodiments of systems for user safety disclosed herein. In some embodiments, the system includes: a non-transitory memory configured to store executable instructions; one or more sensors associated with a helmet or wearable device for capturing different types of sensor data; multiple audio output devices for outputting auditory stimuli that mimic sounds from (1a) horizontal and (1b) below the horizontal of the user and from (2a) to the left, (2b) directly behind, and (2c) to the right of the user; and a hardware processor in communication with the non-transitory memory. The hardware processor can be programmed by the executable instructions to: receive first sensor data captured by one or more sensors. The hardware processor can be programmed by the executable instructions to: determine a first position of an object relative to the user at a first time using the first sensor data. The first position of the object relative to the user can be within a first threshold distance of the user. The hardware processor can be programmed by the executable instructions to: determine one or more first auditory stimuli and corresponding one or more first audio output devices in a plurality of output devices for outputting the one or more first auditory stimuli using the first position of the object relative to the user at a first time. Features of the first auditory stimuli can be related to properties of the object. The hardware processor may be programmed by the executable instructions to cause each of the one or more first audio output devices to output a corresponding first auditory stimulus of the one or more first auditory stimuli.

[0026] In some embodiments, the characteristic of the first auditory stimulus is the pitch of the first auditory stimulus, wherein the characteristic of the first auditory stimulus is related to an attribute of the object including the size of the object, and wherein the pitch of the first auditory stimulus is positively or negatively correlated with the size of the object. In some embodiments, the characteristic of the first auditory stimulus is the loudness of the first auditory stimulus, wherein the characteristic of the first auditory stimulus is related to an attribute of the object including the speed of the object, and wherein the loudness of the first auditory stimulus is positively or negatively correlated with the speed of the object. In some embodiments, the characteristic of the first auditory stimulus is the pitch of the first auditory stimulus, wherein the characteristic of the first auditory stimulus is related to an attribute of the object including the size and / or speed of the object, wherein the pitch of the first auditory stimulus is positively or negatively correlated with the size of the object, and wherein the pitch of the first auditory stimulus is positively or negatively correlated with the speed of the object.

[0027] In some embodiments, the one or more first auditory stimuli and the corresponding one or more first audio output devices include two or more first auditory stimuli and corresponding two or more first audio output devices. The combination of the two or more first auditory stimuli output by the corresponding first audio output devices can imitate the sound of an object moving at a first position relative to the user at a first time that none of the multiple audio output devices can imitate alone. In some embodiments, the one or more first auditory stimuli include a narrow range of frequencies or pure tones. In some embodiments, the one or more first auditory stimuli include music, speech, or a combination thereof.

[0028] In some embodiments, the hardware processor is programmed with executable instructions to: receive second sensor data captured by one or more sensors. The hardware processor may be programmed with executable instructions to: determine a second position of the object relative to the user at a second time using the second sensor data. The second position of the object relative to the user may be within a second threshold distance of the user, wherein the second threshold distance is shorter or longer than the first threshold distance. The hardware processor may be programmed with executable instructions to: determine one or more second auditory stimuli and corresponding one or more second audio output devices in a plurality of output devices for outputting the one or more second auditory stimuli using the second position of the object relative to the user at the second time. Features of the second auditory stimuli may be related to properties of the object. Differences between features of the second auditory stimuli relative to features of the first auditory stimuli may be related to the second position of the object relative to the user at the second time. The hardware processor may be programmed with executable instructions to: cause each of the one or more second audio output devices to output a corresponding second auditory stimulus in the one or more second auditory stimuli.

[0029] In some embodiments, the difference between the features of the second auditory stimulation relative to the features of the first auditory stimulation is a higher or lower frequency number. In some embodiments, the difference between the features of the second auditory stimulation relative to the features of the first auditory stimulation is a greater loudness or a lower loudness.

[0030] In some embodiments, the hardware processor is programmed by executable instructions to receive third sensor data captured by one or more sensors. The hardware processor may be programmed by executable instructions to determine a third position of the object relative to the user at a third time using the third sensor data. The third position of the object relative to the user may be within a third threshold distance of the user. The third threshold distance may be shorter or longer than the second threshold distance. The hardware processor may be programmed by executable instructions to determine (i) one or more first auditory stimuli and one or more corresponding first audio output devices in a plurality of output devices for outputting the one or more first auditory stimuli, and (ii) one or more second auditory stimuli and one or more corresponding second audio output devices in a plurality of output devices for outputting the one or more second auditory stimuli using the third position of the object relative to the user at the third time. The hardware processor may be programmed by executable instructions to cause each of the one or more first and second audio output devices to output a corresponding first or second auditory stimulus in the one or more first and second auditory stimuli.

[0031] In some embodiments, the system is associated with, includes, or is contained in a helmet or wearable device, optionally with multiple audio output devices on, in, or within the helmet or wearable device. The multiple audio output devices can be positioned to output auditory stimuli that mimic sounds from (1a) horizontal and (1b) below the horizontal of the user and from (2a) to the left, (2b) directly behind, and (2c) to the right of the user.

[0032] In some embodiments, the plurality of audio output devices include: (i) a horizontal line level audio output device for outputting auditory stimulation that simulates the sound from the center point of the user's horizontal line when the user wears the helmet or wearable device, and (ii) a first left horizontal line audio output device and a first right horizontal line audio output device below the helmet or wearable device or on the first contour of the helmet or wearable device, for outputting auditory stimulation that simulates the sound from the first vertical level below the left and right horizontal lines of the user, respectively, when the user wears the helmet or wearable device. The plurality of audio output devices may include a first contour below the central horizontal line audio output device for outputting auditory stimulation that simulates the sound from below the horizontal line and the center point of the user.

[0033] In some embodiments, the plurality of audio output devices include (iii) an audio output device below the first left horizontal line and an audio output device below the first right horizontal line on the helmet or wearable device or on a second contour of the helmet or wearable device, for outputting auditory stimulations imitating sounds from a second vertical level below the user's horizontal line on the left and right sides of the user, respectively, when the helmet or wearable device is worn. The second vertical height may be lower than the first vertical height. The second contour may be lower than the first contour.

[0034] In some embodiments, the plurality of audio output devices include (ii) a second left horizontal line below audio output device and a second right horizontal line below audio output device of a first contour on or below the helmet or wearable device, for respectively outputting auditory stimulation imitating sounds from a first vertical level below the user's horizontal line when the helmet or wearable device is worn. The sound imitating the auditory stimulation output by the second left horizontal line below audio output device may be to the left of the sound imitating the auditory stimulation output by the first left horizontal line below audio output device of the first contour. The sound imitating the auditory stimulation output by the second right horizontal line below audio output device may be to the right of the sound imitating the auditory stimulation output by the first right horizontal line below audio output device of the first contour. The second left horizontal line below audio output device may be to the left of the first left horizontal line below audio output device of the first contour. The second right horizontal line below audio output device may be to the right of the first right horizontal line below audio output device of the first contour.

[0035] In some embodiments, the hardware processor is programmed with executable instructions to: receive fourth sensor data captured by one or more sensors. The hardware processor may be programmed with executable instructions to: use the fourth sensor data to determine that an object is passing by, or has a likelihood exceeding a likelihood threshold of passing by the user at a passing time, wherein the object may not be in the user's field of view or may be in the far periphery of the user's field of view at the passing time. The hardware processor may be programmed with executable instructions to: determine a fourth auditory stimulus having a feature corresponding to an auditory feature of an object passing by the user. The hardware processor may be programmed with executable instructions to: cause one or more of the plurality of audio output devices to output the fourth auditory stimulus to the user at an output time. The fourth auditory stimulus output to the user at the output time may result in an increased likelihood that the user visually perceives the object in the far periphery of the user's field of view at the output time or immediately after the output time.

[0036] In some embodiments, the one or more sensors include an image capture device, radio detection and ranging (radar), light detection and ranging (lidar), an audio capture device, or a combination thereof.

[0037] In some embodiments, the object is a first motor vehicle. In some embodiments, the user is in or riding in a second motor vehicle. In some embodiments, the object is at the extreme periphery of the user's field of view at the time of elapse. The far periphery of the user's field of view may include about 60° to about 110° away from the user's nose in time and toward the temple. The extreme periphery of the user's field of view may include about 90° to 110° away from the user's nose in time and toward the temple.

[0038] In some embodiments, the probability threshold of the user passing by the object is at least 50%, 60%, 70%, 80%, 90%, 95% or 99%. In some embodiments, the object is passing by or has a probability exceeding the probability threshold of passing by the user within a range of at most 1 meter, 2 meters, 5 meters or 10 meters of the user.

[0039] In some embodiments, the characteristic of the fourth auditory stimulus is selected from the group consisting of decreasing pitch, increasing loudness, higher number of frequencies, higher frequencies, regularity of angular differences, specificity of angular differences, sound shadowing, larger angular size, loudness asymmetry of the user's ears, a head-related transfer function of approximately 90°, a vertical position below the user's horizontal line, or a combination thereof. In some embodiments, the characteristic change of the fourth auditory stimulus is fastest or highest at or approximately at the elapsed time.

[0040] In some embodiments, the elapsed time is the time at which the fourth sensor data or a portion thereof is captured. The elapsed time may be the time after the fourth sensor data or a portion thereof is captured. The elapsed time may be the time immediately after the fourth sensor data or a portion thereof is captured.

[0041] In some embodiments, the object is passing or may pass by the user on the left or right of the user. In order to cause one or more of the plurality of audio output devices to output a fourth auditory stimulus to the user at the output time, the hardware processor may be programmed by executable instructions to cause the audio output device below the first left horizontal line or the audio output device below the first right horizontal line of the first contour and / or the audio output device below the first left horizontal line or the audio output device below the first right horizontal line of the second contour to output the fourth auditory stimulus at the output time.

[0042] In some embodiments, the fourth auditory stimulus output to the user at the output time may cause the user to visually perceive an object located in the far periphery of the user's field of view at the output time or immediately after the output time by at least 10%, 20%, 30%, 40%, or 50%.

[0043] Embodiments disclosed herein include a computer-readable medium comprising executable instructions that, when executed by a hardware processor of a computing system or device, cause the hardware processor or the computing system or device to perform any of the methods disclosed herein.

[0044] The details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the following description. Other features and advantages will become apparent from the description, drawings, and claims. Neither the present summary nor the following detailed description is intended to define or limit the scope of the subject matter of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 is a block diagram of a non-limiting exemplary system for enhancing visual motion perception in the far and extreme periphery.

[0046] Figure 2 is a flow chart illustrating an exemplary method for enhancing visual motion perception from behind.

[0047] Figures 3A-3D is a schematic illustration showing a non-limiting exemplary arrangement of an auditory output device (e.g., a speaker of a smart helmet or wearable device) of a system for enhancing visual motion perception in the far periphery and extreme periphery.

[0048] Figures 4A-4F is a schematic diagram illustrating non-limiting exemplary activation of an auditory output device of the system (e.g., a speaker of a smart helmet or wearable device) for enhancing visual motion perception in the far periphery and extreme periphery.

[0049] Figure 5 is a flow chart illustrating an exemplary method of improving user security.

[0050] Figure 6 is a block diagram of an illustrative computing system configured to implement any of the methods of the present disclosure.

[0051] Figure 7 is a non-limiting exemplary schematic diagram of an experimental setup used to determine whether auditory stimulation with descending pitch affects the detection of extreme peripheral visual motion from behind.

[0052] Figure 8 is a non-limiting exemplary schematic diagram of an experimental setup used to determine whether hearing influences the detection of extreme peripheral visual motion from behind.

[0053] Fig. 9 is a non-limiting exemplary schematic diagram of the experimental setup used to determine the effects of three types of auditory stimulation on the detection of extreme peripheral visual motion from behind.

[0054] Fig.10 is a non-limiting example graph illustrating the effects of three types of auditory stimulation on the detection of extreme peripheral visual motion from behind. DETAILED DESCRIPTION

[0055] In the following detailed description, reference is made to the accompanying drawings, which form a part of the description. In the drawings, similar symbols generally identify similar components unless the context dictates otherwise. The illustrative embodiments described in the detailed description, drawings, and claims are not meant to be limiting. Other embodiments may be utilized, and other changes may be made, without departing from the spirit or scope of the subject matter presented herein. It will be readily understood that aspects of the present disclosure, as generally described herein and shown in the accompanying drawings, may be arranged, substituted, combined, separated, and designed in a variety of different configurations, all of which are expressly contemplated herein and are a part of the present disclosure.

[0056] Embodiments of systems for improving visual motion perception disclosed herein. In some embodiments, the system includes: a non-transitory memory configured to store executable instructions; an audio output device; a hardware processor (or processor, such as a virtual processor) in communication with the non-transitory memory, the image capture device, and the audio output device. A hardware processor programmed by executable instructions: receives a plurality of images captured by the image capture device. The processor may be programmed by executable instructions to: use the plurality of images to determine the position and trajectory of an object relative to a user at a first time. The position and trajectory of the object relative to the user at the first time may indicate that the object has a probability of exceeding a probability threshold of passing the user at a passing time. The object may not be in the user's field of view, or may be at the far periphery of the user's field of view at the first time. The processor may be programmed by executable instructions to: determine an auditory stimulus and an output time of the output auditory stimulus based on the position and trajectory of the object relative to the user at the first time. The output time may be a passing time, or the output time may be between the first time and the passing time. The feature of the auditory stimulus may be selected from a group including a descending tone. The processor may be programmed by executable instructions to: cause the audio output device to output the auditory stimulus to the user at the output time. As a result of outputting the auditory stimulation to the user at the output time, the possibility that the user visually perceives an object located in the far periphery of the user's visual field at the output time or immediately after the output time may increase.

[0057] Disclosed herein are embodiments of systems for improving visual motion perception. In some embodiments, the system includes: a non-transitory memory configured to store executable instructions; an audio output device; and a hardware processor (or processor, such as a virtual processor) in communication with the non-transitory memory, the image capture device, and the audio output device. The hardware processor can be programmed by executable instructions to: receive a plurality of images captured by the image capture device. The processor can be programmed by executable instructions to: use the plurality of images to determine that an object is passing a user at a passing time, or has a probability exceeding a probability threshold of passing the user. The object may not be in the user's field of view, or may be at the far periphery of the user's field of view at the first time. The processor can be programmed by executable instructions to: determine an auditory stimulus having a feature selected from a group including a descending tone. The processor can be programmed by executable instructions to: cause the audio output device to output the auditory stimulus to the user at the output time. As a result of outputting the auditory stimulus to the user at the output time, the likelihood that the user visually perceives an object located at the far periphery of the user's field of view at the output time or immediately after the output time can be increased.

[0058] Disclosed herein are embodiments of methods for improving visual motion perception. In some embodiments, the method is under the control of a hardware processor (or processor, such as a virtual processor), and includes: receiving sensor data captured by one or more sensors. The method may include: using the sensor data to determine the position and trajectory of an object relative to a subject at a first time. The position and trajectory of the object relative to the subject at a first time may indicate that the object has a probability of exceeding a threshold of probability of moving relative to the subject within a threshold distance of the subject at a second time. The object may not be in the subject's field of view, or may be in the periphery of the subject's field of view at the first time. The method may include: determining an auditory stimulus based on the position and trajectory of the object relative to the subject at the first time. The features of the auditory stimulus may correspond to the auditory features of the object moving relative to the subject within a threshold distance of the subject. The method may include: outputting the auditory stimulus to the subject at an output time. The method may result in an increased probability that the subject visually perceives an object in the periphery of the subject's field of view at the output time or immediately after the output time.

[0059] Disclosed herein are embodiments of methods for improving visual motion perception. In some embodiments, the method is under the control of a hardware processor and includes: receiving sensor data captured by one or more sensors. The method may include: using the sensor data, determining that an object is or has a likelihood above a likelihood threshold that an object is moving relative to the subject within a threshold distance of the subject. The object may not be in the object's field of view, or may be at the periphery of the object's field of view at the first time. The method may include: determining an auditory stimulus whose characteristics correspond to auditory characteristics of an object moving relative to the subject within the subject's threshold distance. The method may include: outputting the auditory stimulus to the subject at an output time. The method may result in an increased likelihood that the subject visually perceives an object at the periphery of the subject's field of view at the output time or immediately after the output time.

[0060] Embodiments of systems for user safety disclosed herein. In some embodiments, the system includes: a non-transitory memory configured to store executable instructions; one or more sensors associated with a helmet or wearable device for capturing different types of sensor data; multiple audio output devices for outputting auditory stimuli that mimic sounds from (1a) horizontal and (1b) below the horizontal line of the user and from (2a) left, (2b) directly behind, and (2c) right of the user; and a hardware processor (or processor, such as a virtual processor) that communicates with the non-transitory memory. The hardware processor can be programmed by executable instructions to: receive first sensor data captured by one or more sensors. The hardware processor can be programmed by executable instructions to: use the first sensor data to determine a first position of an object relative to the user at a first time. The first position of the object relative to the user can be within a first threshold distance of the user. The hardware processor can be programmed by executable instructions to: use the first position of the object relative to the user at a first time to determine one or more first auditory stimuli and one or more corresponding first audio output devices in a plurality of output devices for outputting the one or more first auditory stimuli. Features of the first auditory stimuli can be related to attributes of the object. The hardware processor may be programmed by the executable instructions to cause each of the one or more first audio output devices to output a corresponding first auditory stimulus of the one or more first auditory stimuli.

[0061] Reference numerals may be repeated throughout the drawings to indicate corresponding relationships between referenced elements. The drawings are provided to illustrate the exemplary embodiments described herein and are not intended to limit the scope of the present disclosure.

[0062] Enhanced visual motion perception

[0063] Human visual perception is most accurate and sensitive at the fovea of ​​the eye (~2 degrees), with performance gradually decreasing through the parafovea (~5 degrees) and periphery (~40 degrees). Therefore, a person needs to direct his or her head and / or gaze at a moving object in his or her extreme periphery (60-100 degrees) (even if the object's motion is relative and caused by the perceiver's motion) in order to capture the object in the fovea. Visually perceiving moving objects in the extreme periphery is critical in a person's daily life, such as when driving. Many technologies have been developed, such as rearview mirrors, side mirrors, rear monitors, blind spot warning systems in side mirrors, or digital mirrors, for seeing moving objects that would otherwise not be captured in the fovea and parafovea of ​​a person's eyes. These technologies fall into one of two categories: displaying an image of an object closer to the fovea of ​​a person's retina optically (e.g., a mirror) and / or through image processing (e.g., a digital mirror); or, warning a person to direct his or her gaze toward an object (e.g., a blind spot warning system). Existing technologies require directing a person's gaze toward the object itself and / or its optical image on a device (e.g., a mirror). Existing technology causes people to drive without looking ahead, which is one of the most common causes of car accidents.

[0064] Disclosed herein include systems, devices, computer-readable media and methods for enhancing the human visual motion perception of the periphery. Using auditory cues or stimulation (e.g., simple auditory cues), people can perceive the moving objects around without directing his or her gaze to the object. In certain embodiments, the vision of people in far periphery or extreme periphery is enhanced, which can help the subject to perform tasks such as driving a car or riding a motorcycle. Auditory cues or stimulation can enable people to perceive far edge and extreme moving objects without directing people's gaze to the object. Enhanced perception can improve people's safety in daily life, such as when he or she drives a car. By presenting sound to people while manipulating one or more factors disclosed herein, people can activate or improve the visual perception of one or more moving objects in far periphery and extreme periphery. Presenting sound without manipulating any of these factors actually damages the visual perception of far periphery and extreme periphery.

[0065] Auditory stimulation, presentation, and cues

[0066] When a person is facing forward, an ecologically appropriate auditory stimulus, presentation, or cue corresponding to motion in the far periphery or extreme periphery, which is approximately 90 degrees to the side, can facilitate the perception of visual motion in the far periphery or extreme periphery. Auditory stimuli having one or more attributes or characteristics can make the far visual periphery or extreme periphery more salient than the non-ecological counterpart of the stimulus.

[0067] (a) tone – The pitch is falling (not rising).

[0068] (a') The pitch is not only decreasing, but it is decreasing at the fastest rate in its course. Consider the sigmoid function of the rate of change of pitch over time. When the function decreases at the fastest rate (i.e., the pitch decreases at the fastest rate), the human brain knows that the moving car (or moving object, such as a moving motorcycle or a moving truck) is at 90 degrees of the person's field of view and is likely to be physically closest to the person.

[0069] (b) Loudness – Sounds are loud because the corresponding sound source is nearby. Sounds can be loud enough to even vibrate the body (this can be considered a generalization of hearing).

[0070] (b') The loudness is not only high, but also at a peak. Loudness can be represented as an inverted U-shaped function. When the loudness flattens out, a person's brain knows that the object is approaching the periphery.

[0071] (c) Frequency range – When the object making the noise is close, people can hear a wider range of its transmitted frequencies, especially more high-frequency components. When the object making the noise is far away, people can hear more low-frequency components.

[0072] (c')c derivative.

[0073] (d) Regularity of angle differences – Even if a person is moving at zero velocity relative to the object, a moving object will sound regular in front of, behind, or to the side of the person.

[0074] Moving objects (like cars) are propelling themselves and making noise. For example, dirt is kicked up in the back instead of in the front. Humans are sensitive to the noise or sounds made.

[0075] Movers typically have a "heel-toe" sequence for each step or footstep, which is literal, rather than running on a point-sized tip. For objects with tires, there is also a heel-toe sequence, where the tire constantly rotates onto a new "toe," which becomes the heel as the tire rotates. The heel-toe sequence sounds different when the person is in front of the person than when the person is behind the person, even if the person is moving at zero velocity relative to the object. For example, when in front, the heel-toe sounds are more consistent in time (e.g., have a higher frequency), while when behind, the heel-toe sounds are more spaced apart (e.g., have a lower frequency).

[0076] (d') A derivative variant of (d).

[0077] (e) The peculiarity of the angle difference– For reasons that are not as general as in (d), real moving objects that make noise sound different in front of, behind, and to the side of a person. Consider a person driving in front of a car (at the same speed), opposite from the back of the car, and opposite from the side of the car, all at equal distances. There is a difference in the sound the person hears. The person might hear more motor sound when driving in front of the car (assuming the car has a front engine) and more exhaust and carburetor sound when driving behind the car. The difference in sound depends largely on the kind of object making the sound (e.g., car), which is something humans learn.

[0078] (e') A derivative variant of (e).

[0079] (f) Sound Shadow – Objects in the environment partially block the sound from things behind them. Therefore, even a very quiet moving object has a moving “sound shadow” that partially blocks the ambient sound behind the object. Moving objects can be associated with moving quietness, or have moving sound shadows that humans can perceive as moving sound, rather than moving sound.

[0080] (g) Larger angular dimensions – Moving sound-generating objects are not point-sized. Such objects are limited in size, and when closer, these objects emit sound from multiple directions. For example, when a train is far away, the train is a point source of sound. But when you are next to it, the brain can generate sound from almost 180 degrees. In fact, almost half a hemisphere, including above and below the horizontal line. Closer objects can have more "surround sound"

[0081] (g') Derivative variant of g.

[0082] (h) Asymmetrical ears – The closer a moving object is to a person, the greater the loudness asymmetry in the person’s ears.

[0083] (h')h derivative variant.

[0084] (i) Head Related Transfer Function (HRTF) – The closer a moving object is to a person, the more the HRTF indicates that the object is at a 90 degree angle to the person.

[0085] (i') A derivative variant of (i).

[0086] (k) Vertical Position– The strongest auditory cues for a moving creature can emanate directionally from far below the horizon, from the creature’s feet on the ground. So a creature in one’s immediate vicinity might range from 70 degrees below the horizon to 40 degrees above the horizon – and a total vertical range of 110 degrees – yet its sounds disproportionately tend to emanate from the bottom, at -70 degrees. An auditory cue at -70 degrees doesn’t necessarily mean the object is at -70 degrees. Instead, an auditory cue at -70 degrees can indicate an entire moving creature, whose feet (or wheels of a moving vehicle) happen to be down there, even though most creatures aren’t even near there. The rest of the creature’s body (or the vehicle’s body) might be completely silent. Most animals are actually like this – their joints make zero sounds when they move, and all the animal sounds come from the animal stomping on the ground below.

[0087] (k') One-to-one mapping from the height of an object's sound in a person's perceptual field to the distance between the object and the person – Objects can fill up a large amount of a person’s perceptual field, especially when the object is near the person. Objects can make sounds in many different ways, and from different parts of them. For moving creatures (or moving vehicles), there are some basic generalizations: Sounds made by a moving creature tend to come from his or her feet on the ground. From the vertical position of a sound made by a moving creature in the person’s perceptual field, the distance of the creature from the person can be determined. There is a one-to-one correspondence between vertical field position and distance below the horizontal line. Sounds coming from the horizontal line are infinitely far away. Sounds coming from or being perceived by a person from below his or her perceptual field are closer to his or her perceptual field. Sounds coming from below the person mean that the creature making the sound is essentially walking directly next to the person, with a distance of zero from the person.

[0088] System for enhancing visual motion perception

[0089] Figure 1 1 is a block diagram of a non-limiting exemplary system or device (e.g., wearable device) 100 for enhancing far peripheral and extreme peripheral visual motion perception. The system may include one or more sensors 104, a processor 108, a sound creator 112, and a sound generator 116. The one or more sensors 104 may include radar, a camera, and a microphone. The one or more sensors 104 may capture sensor data, such as a plurality of images.

[0090] The processor 108 may use the sensor data to determine properties and characteristics of a moving object (e.g., an object, such as a vehicle, approaching an object from behind, such as a user of the system), such as the size of the object, the distance from the object to the object, the speed of the object, the direction of movement of the object. The processor 108 may use the sensor data to determine the location and possible trajectory of the object. For example, the processor 108 may use the sensor data to determine the location and trajectory of the object relative to the subject at a first time. The location and trajectory of the object relative to the subject at a first time may indicate that the probability that the object is moving within a threshold distance (e.g., 5 meters) of the object relative to the object at a second time is higher than a probability threshold (e.g., 80%). The object may not be in (or is not in) the subject's field of view, or may be in (or is in) the periphery of the subject's field of view.

[0091] The sound creator 112 can convert the properties and characteristics of the object into the sound stimulation or prompt disclosed herein. The sound stimulation or prompt does not need to be or imitate the real / natural sound (e.g., engine sound) from the object. In some embodiments, the sound stimulation or prompt can be a very simple artificial sound that simulates or amplifies the essence of the sound from the moving object (e.g., a simple beep with a sudden pitch change). For example, the sound creator 112 can determine the auditory stimulation based on the position and trajectory of the object relative to the subject at the first time. The characteristics of the auditory stimulation can correspond to the auditory characteristics of the object moving relative to the object within a threshold distance of the object. For example, the auditory stimulation can have a descending pitch that can imitate a Doppler pitch shift.

[0092] The sound generator 116 may be an audio device such as a speaker or earphone. The sound generator 116 may output the determined sound stimulation or prompt. For example, the sound generator 116 may output an auditory stimulation to an object at an output time. As a result of the sound stimulation or prompt output by the sound creator 112, the extreme peripheral visual perception of the subject may be enhanced. At the output time or immediately after the output time (e.g., within 0.0000001 second, 0.000001 second, 0.00001 second, 0.00001 second, 0.00001 second, 0.0001 second, 0.001 second, 0.001 second, 0.01 second, 0.1 second, or 1 second), the possibility of the subject visually perceiving an object around the subject's field of vision increases.

[0093] System 100 or one or more components of system 100 can be associated with, can include, or can be included in, a helmet for a motorcycle or bicycle rider (e.g., a smart helmet), a helmet for a driver of a vehicle such as a car or recreational vehicle, and a helmet for an aviation pilot; a vehicle audio system (e.g., a car audio system); and a wearable speaker system for a pedestrian or wheelchair user.

[0094] Enhanced features of visual motion perception based on auditory stimulation

[0095] (1) Digital mirroring device . The digital mirror device and any method (or system) based on auditory presentation or stimulation disclosed herein is intended to help humans visually perceive moving objects in blind spots, such as the extreme periphery. However, the digital mirror device uses image processing, while the method disclosed herein uses auditory presentation. The digital mirror device also requires the person's gaze direction to move towards the visual image in the device, while the method disclosed herein does not require such a gaze direction because the method enables humans to directly perceive objects in the extreme periphery rather than the fovea.

[0096] (2) Rear Object Alert System . The rear object warning system and any method (or system) based on auditory presentation or stimulation disclosed herein is intended to help humans visually perceive moving objects in blind spots, such as extreme peripheries. However, the rear object warning system uses auditory and / or visual alarms to cause a person to direct his or her gaze toward an object, while the method disclosed herein uses auditory stimulation to enable a person to visually perceive an object without changing the direction of his or her gaze. The auditory stimulation may have features or characteristics disclosed herein (e.g., a descending pitch), while a simple alarm sound emitted by a rear object warning system cannot provide visual perception in extreme peripheries (and may even impair visual detection).

[0097] (3) Audio-tactile direction indicators . The audio-tactile direction indicator and any method (or system) based on auditory presentation or stimulation disclosed herein uses two modalities to present information. The audio-tactile direction indicator is based on auditory and tactile stimulation. The method disclosed herein is based on visual and auditory stimulation. The audio-tactile direction indicator is intended to provide directional information in a helmet primarily through auditory stimulation, and uses tactile stimulation as an aid. The method disclosed herein is intended to enable humans to visually perceive moving objects in extreme surroundings by presenting auditory stimulation having one or more characteristics or attributes disclosed herein (e.g., descending pitch).

[0098] (4) 3D audio or stereo technology . The 3D audio or stereo technology based on auditory presentation or stimulation disclosed herein and any method (or system) uses sound to allow humans to perceive the external world more clearly through auditory stimulation. These technologies aim to create more realistic, ultra-accurate sounds to replicate real environmental sounds. However, the methods disclosed herein only require simple sounds, while manipulating one or more factors or features disclosed herein, but do not require such high-quality sounds to be produced as these technologies. For example, a simple mono 1000 Hz beep with only a 10% reduction in pitch and an original fade-in and fade-out effect of loudness can enhance the perception of visual motion at far and extreme edges through inexpensive headphones.

[0099] Enhanced visual motion perception

[0100] Figure 2 is a flow chart illustrating an exemplary method 200 for enhancing or improving visual motion perception from behind. The method 200 may be embodied as a set of executable program instructions stored on a computer-readable medium (e.g., one or more disk drives) of a computing system or device. For example, Figure 6 The computing system 600 shown in FIG. 1 and described in more detail below may execute a set of executable program instructions to implement the method 200. When the method 200 is initiated, the executable program instructions may be loaded into a memory such as a random access memory and executed by one or more processors of the computing system 600. Although the description of the method 200 is relatively simple, the method 200 may be executed in a manner similar to that of the processor 100 of FIG. 1 . Figure 6 The depicted computing system 600 is intended to be illustrative only and not limiting. In some embodiments, the method 200 or portions thereof may be performed serially or in parallel by multiple computing systems.

[0101] In some embodiments, the computing system is associated with, includes, or is included in a helmet or wearable device. For example, the computing system can be attached to the helmet or wearable device. For example, the computing system can be part of the helmet or wearable device.

[0102] After method 200 starts at block 204, method 200 proceeds to block 208, where the computing system receives sensor data captured by one or more sensors. The one or more sensors may include an image capture device, radio detection and ranging (radar), light detection and ranging (lidar), an audio capture device, or a combination thereof. The sensor data may include a plurality of images. The computing system may receive a plurality of images captured by the image capture device.

[0103] In some embodiments, the sensor data includes the position, speed, and direction of the object captured by one or more telemetry and positioning sensors associated with the object. The one or more sensors may include one or more telemetry and positioning sensors associated with the object. The one or more telemetry and positioning sensors associated with the object may include a global positioning system (GPS) sensor associated with the object. For example, the computing system 200 of the first vehicle can use the sensor data captured by the one or more telemetry and positioning sensors of the second vehicle to determine the position and trajectory of the second vehicle relative to the first vehicle at the first time in block 212 (described in detail below).

[0104] For example, a first person is driving a first vehicle associated with or including a computing system 200. A second person is driving a second vehicle approaching the first vehicle from behind. The second vehicle is associated with or includes another computing system 200. The first vehicle may be associated with or include one or more telemetry and position sensors. The second vehicle may be associated with or include one or more telemetry and position sensors. The first vehicle may directly or indirectly transmit sensor data captured by one or more telemetry and position sensors of the first vehicle to the second vehicle. The second vehicle may directly or indirectly transmit sensor data captured by one or more telemetry and position sensors of the second vehicle to the first vehicle. For example, the computing system 200 of the first vehicle may use the sensor data captured by the one or more telemetry and position sensors of the second vehicle to determine the position and trajectory of the second vehicle at a first time relative to the first vehicle at block 212 (described in detail below). For example, the computing system 200 of the second vehicle may use the sensor data captured by the one or more telemetry and position sensors of the first vehicle to determine the position and trajectory of the first vehicle at another first time relative to the second vehicle at block 212 (described in detail below).

[0105] In some embodiments, the computing system includes one or more sensors or one or more of the one or more sensors. The computing system can be associated with one or more sensors or one or more of the one or more sensors (e.g., in electrical communication, wired communication, or wireless communication). In some embodiments, one or more sensors or one or more of the one or more sensors are associated with a second system, such as a sensor system. The computing system can be associated with the second system (e.g., in electrical communication, wired communication, or wireless communication). The computing system can receive sensor data captured by one or more sensors or one or more of the one or more sensors from the second system. In some embodiments, a helmet, a car audio system, or a wearable speaker system or device is associated with one or more sensors, including or contained in one or more sensors. For example, a computing system can be attached to a helmet and communicate electrically, wired, and / or wirelessly with one or more sensors. For example, one or more sensors can be attached to a helmet or a wearable device.

[0106] From block 208, method 200 proceeds to block 212, where the computing system uses the sensor data to determine that an object is moving, or has a likelihood above a likelihood threshold, relative to a subject (e.g., a user of the system) within a threshold distance of the subject (e.g., an object passing by the subject, or an object striking the subject). The computing system may use the sensor data to determine that the object is moving, or has a likelihood above a likelihood threshold, relative to the subject at a threshold distance of the subject at a second time. The computing system may use the sensor data to determine that at the time of the passing, the object is passing by the user, or has a likelihood above the likelihood threshold of passing by the user.

[0107] In some embodiments, the computing system may use the sensor data to determine the position and trajectory of the object relative to the subject at a first time. The position and trajectory of the object relative to the subject at the first time may indicate that the object is likely to pass by or hit the subject at a second time. The position and trajectory of the object relative to the user at the first time indicates that the object has a likelihood of exceeding a likelihood threshold of passing by the user at the passing time.

[0108] The computing system may determine, based on the position and trajectory of the object relative to the subject at the first time, that a likelihood that the object moved within a threshold distance of the subject relative to the subject at the second time is above a likelihood threshold. The computing system may determine, based on the position and trajectory of the object relative to the user at the first time, that the object has a likelihood exceeding a likelihood threshold at the elapsed time.

[0109] In some embodiments, the computing system uses one or more machine learning methods or models to determine that an object is moving relative to the subject within a threshold distance of the subject (e.g., at a second time), or has a likelihood above a likelihood threshold. The computing system may use one or more machine learning methods or models to determine that at a passing time, an object is passing the user, or has a likelihood above a passing likelihood threshold. The computing system may use one or more machine learning methods or models to determine the position and trajectory of an object relative to the subject at a first time.

[0110] The likelihood threshold may be, for example, at least 50%, 60%, 70%, 80%, 90%, 95%, or 99%. The threshold distance may be, for example, at most 0, 1, 2, 5, or 10 meters. The object may pass by the subject, for example, within at most 1, 2, 5, or 10 meters of the subject.

[0111] The object can be a first motor vehicle (e.g., a bicycle, motorcycle, car, or truck). The subject can be in a second motor vehicle. The subject can be riding in the second motor vehicle (e.g., a motorcycle or bicycle). The subject can be walking. The first motor vehicle can approach the target from behind.

[0112] In some embodiments, the computing system may transmit sensor data, the position and trajectory of the object relative to the user, and / or an indication that the object is passing the user or has a probability exceeding a probability threshold when passing the user to a third system at a first time. The computing system may be associated with the third system (e.g., in electrical communication, wired communication, or wireless communication). For example, the computing system may send sensor data, the position and trajectory of the object relative to the user, and / or an indication that the object is passing the user or has a probability exceeding a probability threshold when passing to an audio system of the vehicle or a control system of the vehicle at a first time.

[0113] The object may not be (or is not) in the subject's field of view at the first time, the second time, and / or the elapsed time. At the first time, the second time, and / or the elapsed time, the object may (or is) in the periphery (e.g., far periphery and / or extreme periphery) of the subject's field of view. At the first time, the second time, and / or the elapsed time, the object may (or is) in the far periphery of the subject's field of view. At the first time, the second time, and / or the elapsed time, it may (or is) in the extreme periphery of the subject's field of view. The far periphery of the user's field of view may include, for example, temporarily away from the subject's nose and toward the temple from about 60° to about 110°. The far periphery of the user's field of view may include, for example, from about 50°, about 55°, about 60°, about 65°, or about 70° to about 100°, about 105°, about 110°, about 115°, or about 120° temporarily away from the subject's nose and toward the temple. The outermost periphery of the user's field of view may include, for example, temporarily away from the subject's nose and toward the temple from about 90° to 110°. The far periphery of the user's field of view may include, for example, from approximately 80°, approximately 85°, approximately 90°, approximately 95°, or approximately 100° to approximately 100°, approximately 105°, approximately 110°, approximately 115°, or approximately 120° temporarily away from the subject's nose and toward the temple.

[0114] In some embodiments, the first time is the time at which the sensor data or the plurality of images or a portion thereof are captured. The first time may be the time after the sensor data or the plurality of images or a portion thereof are captured. The first time may be the time immediately after the sensor data or the plurality of images or a portion thereof are captured (e.g., within 0.0000001 seconds, 0.00001 seconds, 0.0001 seconds, 0.01 seconds, 0.1 seconds, or 1 second).

[0115] Method 200 proceeds from frame 212 to frame 216, in frame 216, the computing system determines an auditory stimulus whose features correspond to the auditory features of an object moving relative to the subject within a threshold distance of the subject. The computing system may determine the auditory stimulus based on the position and trajectory of the object relative to the subject at the first time. The features of the auditory stimulus may correspond to the auditory features of the object moving relative to the subject within the threshold distance of the subject. The computing system may determine an auditory stimulus having a feature selected from a group including a descending tone. The computing system may determine the auditory stimulus and the output time of the output auditory stimulus based on the position and trajectory of the object relative to the user at the first time. The output time may be an elapsed time, or the output time may be between the first time and the elapsed time. The computing system may determine the auditory stimulus that needs to be output based on the position and trajectory of the object relative to the user at the first time. The features of the auditory stimulus may be selected from a group including a descending tone. The auditory stimulus may be pre-selected. The features of the auditory stimulus (e.g., a descending tone) may be pre-selected.

[0116] In some embodiments, the output time is a second time (at which the object is determined to have a probability above a probability threshold to move within a threshold distance of the object relative to the object) or an elapsed time (at which the object has a probability above a probability threshold to pass by the user). The output time may be immediately before the second time or the elapsed time (e.g., 0.0000001 seconds, 0.000001 seconds, 0.0001 seconds, 0.001 seconds, 0.01 seconds, 0.1 seconds, or 1 second). The output time may be between the first and second times, or may be an elapsed time.

[0117] In certain embodiments, the auditory stimulation is characterized by a descending tone. For example, the auditory stimulation can be a 1000 Hz beep, and the tone is reduced or decreased by 10% or 100 Hz. The tone can be reduced or decreased by 1%, 5%, 10%, 15%, 20% or more of the auditory stimulation frequency, for example. The tone can be reduced or decreased by 10 Hz, 50 Hz, 100 Hz, 150 Hz, 200 Hz or more, for example. The reduction or decline of the auditory stimulation tone can occur in 0.00000001 second, 0.000001 second, 0.00001 second, 0.0001 second, 0.001 second, 0.01 second, 0.1 second or 1 second. The auditory stimulation can have a fade-in and fade-out effect. In some embodiments, the auditory stimulation comprises a narrow range of frequencies (e.g., 1 Hz, 10 Hz, or 100 Hz) or pure tones (e.g., 250 Hz, 500 Hz, 750 Hz, 800 Hz, 900 Hz, 1000 Hz, 1100 Hz, or 1200 Hz). In some embodiments, the auditory stimulation comprises music, speech, or a combination thereof.

[0118] In some embodiments, the characteristic of the auditory stimulation is selected from the group consisting of decreasing pitch, increasing loudness, higher number of frequencies, higher frequencies, regularity of angular differences, specificity of angular differences, sound shadowing, greater angular size, loudness asymmetry of the subject's (or user's) ears, a head-related transfer function of approximately 90°, a vertical position below the subject's (or user's) horizontal line, or a combination thereof. In some embodiments, the change in the auditory stimulation characteristic is fastest or highest at or about the elapsed time.

[0119] Method 200 proceeds from frame 216 to frame 220, wherein the computing system causes the auditory stimulus to be output to the subject at the output time. The computing system may cause one or more audio output devices to output the auditory stimulus at the output time. The computing system may be associated with, include, or be contained in one or more audio output devices. Another system may include one or more audio output devices, such as a helmet, a car audio system, or a wearable speaker system or device. The computing system may transmit the auditory stimulus and / or the output time to another system. The computing system may cause one or more audio output devices of another system to output the auditory stimulus at the output time.

[0120] Auditory stimulation can increase the possibility of visually perceiving an object at the periphery (e.g., far periphery and / or extreme periphery) of the subject's visual field at the time of output or immediately after the output. Auditory stimulation can increase the possibility of visually perceiving an object at the far edge and / or extreme of the user's visual field at the time of output or immediately after the time of output. The increase in possibility can be, for example, at least 10%, 20%, 30%, 40%, or 50%. The time immediately after the output time can be, for example, within 0.0000001 second, 0.000001 second, 0.00001 second, 0.0001 second, 0.001 second, 0.01 second, 0.1 second, or 1 second of the output time.

[0121] Left and right audio output devices. In some embodiments, the position and trajectory of the object relative to the subject at the first time indicates that the object may pass or hit the object to the left or right of the object at the second time (or the passing time). The computing system can cause the auditory stimulation to be output to the left ear or right ear of the subject at the output time, respectively.

[0122] In some embodiments, one or more audio output devices include a left audio output device and a right audio output device for outputting auditory stimulation to the left ear and / or right ear of the object. Earphones such as ear-hook earphones may include a left speaker and a right speaker for outputting audio stimulation to the left ear and / or right ear of the subject, respectively. The car audio system may include a left speaker and a right speaker. When the left speaker of the car audio system outputs audio stimulation, although the subject can hear the audio stimulation with both ears, the subject can first hear the audio stimulation with the left ear.

[0123] The position and trajectory of the object relative to the subject at the first time may indicate that the object may pass or strike the subject to the left of the user at the passing time or at the second time (at which time the object is determined to have a probability above a probability threshold to move relative to the subject within a threshold distance of the subject). The computing system may cause the left audio output device to output an auditory stimulus to the left ear of the user at the output time. The position and trajectory of the object relative to the subject may indicate that the object may pass or strike an object to the right of the user at the passing time or at the second time. The computing system may cause the right audio output device to output an auditory stimulus to the right ear of the user at the output time.

[0124] The method 200 ends at block 224 .

[0125] Systems for increased user safety

[0126] Speaker Configuration

[0127] Figures 3A-3D is a schematic diagram showing a non-limiting exemplary arrangement of an auditory output device (e.g., a speaker of a smart helmet or wearable device) of a system for improving user safety by, for example, enhancing visual motion perception in the far periphery and far periphery. Figure 3AThe back half 300 of the person's projection sphere is shown, i.e., behind the person. For most moving objects around a person (e.g., a driver or rider of a vehicle), the sound comes disproportionately from below the person's perceptual field of view. A person's perceptual field of view is at the level of the person's eyes and ears. Without being bound by theory, the sound of moving objects around a person comes from below the person's horizontal line because moving objects propel themselves on the ground and therefore generate considerable noise when interacting with the ground. In addition, the engine noise emitted by cars (and vehicles in general) also tends to be below the level of a person's eyes. Cars that follow a person far behind (and in the same direction) on the road tend to be located at the center point of the horizontal line (where the dotted line intersects 304c0). When cars approach a person (or a vehicle that a person is operating, such as driving or riding, or in a car), these cars tend to move downward and outward in the person's projection field. Cars that cross a few lanes of people will move outward more than downward (see the leftmost arrow or track 308). Cars crossing a lane will move more and more downward and less and less outward (see the arrow or track 312 at about 7 o'clock). Objects on a collision course to a person's rear end will move mostly downward and very little outward (see the area between the two lower arrows or tracks 316a, 316b).

[0128] Given that Figure 3A Placing speakers on a helmet or wearable device (e.g., a neck device) can present challenges because of the characteristic paths of objects in the projected field behind the person. If the speakers are placed at a constant height around the person’s head, then the speakers will track more, e.g. Figure 3B As shown by the horizontal dashed line 320 on the left side, unlike the car behind the driver, it moves in the same direction.

[0129] In some embodiments, the helmet is equipped with a two-dimensional array of speakers 324, 328a-328h, 332a-332h in the rear field of view, such as Figure 3C In some embodiments, the helmet is equipped with a two-dimensional speaker array, which includes a speaker 324 in the rear field of view, one or more speakers 328a-328h and / or one or more speakers 332a-332h. In some embodiments, the helmet is equipped with a two-dimensional array of speakers 324, 328a-328h, 332a-332h and one or more speakers for each radial line. For reference Figure 3A For the reasons discussed, for any position of the car behind the person, there is a unique point on the rear half of the projection sphere that corresponds to that position. Therefore, the activation of any particular speaker can uniquely cue (or indicate) the specific three-dimensional position of the person's car on the road behind the person.

[0130] For a neck device (or wearable device) having only a single profile or a single row of speakers, the device may include speakers 332a, 328a, 324, 328b, 332b, which are based on Figure 3D The device may be positioned according to the "car passing from behind, one lane passing" curve or pattern 340 shown. When the device is moved toward the back of the head of the person using (e.g., wearing) the device, the device may rise, or the device may cause the speakers to extend upward from the device accordingly. If the helmet (or wearable device) includes a single row of speakers, the speakers may be positioned according to the "car passing from behind, one lane passing" curve or pattern 336 on the back of the helmet instead of a straight line. The single row of speakers 332a, 328a, 324, 328b, 332b so positioned can be used to track typical passing cars. When a car will collide with a person wearing the helmet from behind, two or more speakers 332a, 328a, 324, 328b, 332b with the same eccentricity or different eccentricities may be activated. For example, a pair of speakers 332a, 332b may be activated. As another example, a pair of speakers 328a, 328b may be activated. The two or more speakers 332a, 328a, 324, 328b, 332b that are activated may have the same eccentricity or different eccentricities. The two or more speakers 332a, 328a, 324, 328b, 332b that are activated may output auditory stimuli with the same loudness or different loudness at corresponding times. In some embodiments, when a car is about to collide with a person wearing a helmet from behind, one of the speakers 332a, 328a, 324, 328b, 332b may be activated.

[0131] Speaker Activation

[0132] Figures 4A-4F is a schematic diagram illustrating non-limiting exemplary activation of an auditory output device of a system (e.g., a speaker of a smart helmet or wearable device) for improving user safety by, for example, enhancing visual motion perception in the far periphery and far periphery.

[0133] Figure 4AThe back half 400 of the person's projection sphere is shown, i.e., behind the person. Cars on the road that are far behind a person (and traveling in the same direction) tend to be located at the center point of the horizontal line in the projection field of view (0 degrees 404). When a car approaches a person (or a vehicle that the person is operating, such as driving or riding, or in a car), the car tends to move downward in the projection field and move outward in the person's projection field. For example, when a car approaches a person from a distance, the car can move from 0 degrees to 45 degrees in the person's projection field (represented by the 45-degree outline 404c1). The car can then move from 45 degrees to 90 degrees in the person's projection (represented by the 90-degree outline 404c2), and when the car is really close to the person (e.g., the car is behind the person, or when the car passes the person, the car is next to the person), the projection can be at the extreme periphery of the person. A car will move outward instead of downward when it passes several lanes above the person (e.g., see line or track 408). A car crossing a lane will move more downward and less outward (e.g., see line or track 412). An object on a collision course to a person's rear end will move mostly downward and very little outward (e.g., see line or track 416).

[0134] refer to Figure 4B1 and 4B2 , when an object (e.g., a car) comes from behind at a fixed distance (e.g., 50m), sound generation can begin. In some embodiments, the generated sound can have one or three tones: a low tone for a large object (e.g., a truck), a medium tone for a medium object (e.g., a car), and a high tone for a small object (e.g., a motorcycle). In some embodiments, the generated sound can have one or three tones: a low tone for a small object (e.g., a motorcycle), a medium tone for a medium object (e.g., a car), and a high tone for a large object (e.g., a truck). Each tone can have a narrow frequency range, or a pure tone (e.g., 250 Hz, 500 Hz, or 1000 Hz).

[0135] refer to Figure 4C , when an object is approaching from behind at a distance of, for example, 50 meters, speaker 0 may start to produce a single tone at a low loudness. The tone may reflect the size of the object: 250 Hz for a large truck, 500 Hz for a car, and 1000 Hz for a motorcycle. The tone may be adjusted according to the speed of the object: higher the speed, higher the tone, lower the speed, lower the tone. For example, a single tone of 1050 Hz may be produced for a motorcycle traveling at high speed. As another example, a single tone of 950 Hz may be produced for a motorcycle traveling at low speed.

[0136] refer to Figure 4D, when the object is behind the person at a distance of, for example, 25 meters, the speaker 1x (e.g., speaker 1a, 1b, or 1c) or the plurality of speakers 1x may start to produce a plurality of tones at a medium loudness. The tones may be as shown in the reference Figure 4C The generated tones are based on the size of the object (and possibly adjusted by the speed of the object) plus a narrow range of frequencies (e.g., ±1%, ±2%, ±5%, or ±10%, or ±10 Hz, ±20 Hz, ±50 Hz, or ±100 Hz). For example, if the tone generated when the object is 50 meters behind the person is 250 Hz, the multiple tones generated when the truck is 25 meters behind the person can be 250 Hz ±5%. Different speakers can be activated based on the trajectory of the object. Different trajectories will cause different speakers to be activated. For example, if the object approaches from behind with a trajectory of a collision course, speaker 1a can be activated. If the object approaches from behind with a dangerous approach trajectory, speaker 1b can be activated. If the object approaches from behind and the trajectory is dangerous when the person turns, speaker 1c can be activated. In some embodiments, if the object approaches from behind with a trajectory that is dangerous when the person turns right or left, respectively, speaker 1c' or 1c can be activated.

[0137] refer to Figure 4E , when the object is behind the person at a distance of, for example, 5 meters, the speaker 2x (e.g., speaker 2a, 2b, or 2c), or multiple speakers 2x in addition to the speaker 1x, may start to produce multiple tones at high loudness. The tones may be as described in reference Figure 4C The generated tones are based on the size of the object (and possibly adjusted by the speed of the object) plus a narrow range of frequencies. For example, if the tone generated when the object is 50 meters behind the person is 250 Hz, the multiple tones generated when the truck is 25 meters behind the person can be 250 Hz ± 5%. Different speakers can be activated according to the trajectory of the object. Different trajectories will cause different speakers to be activated. For example, if the object approaches from behind with a trajectory of a collision course, speakers 1a and 2a can be activated. If the object approaches from behind with a dangerous approach trajectory, speakers 1b and 2b can be activated. If the object approaches from behind with a dangerous trajectory when the person turns, speakers 1c and 2c can be activated. After two (or more) speakers are activated, a sound source at a larger angle can be imitated, the object, when the object is closer to the person. In some embodiments, if the object approaches from behind at a distance of, for example, 5 meters with a dangerous trajectory when the person turns right or left, respectively, speakers 1c' and 2c' or 1c and 2c can be activated.

[0138] refer to Figure 4F, when an object is at the outermost periphery of the person (e.g., from approximately 90 degrees) and the trajectory is dangerously close or dangerous when turning, speakers 1x and 2x (e.g., speakers 1b and 2b for the dangerously close trajectory) can produce a sudden pitch drop to have cross-modal enhancement of motion detection at the outermost periphery, as described herein.

[0139] Auditory stimulation

[0140] There are patterns in how the stimulus changes based on the behavior of the vehicle approaching the person. Here are some of the patterns.

[0141] A. There is a one-to-one mapping between the second half of the lower projection sphere and the car's position. Cars behind drivers travelling in the same direction can be located Figure 3A The driver can then position the vehicle on a contour in the road, depending on how far behind the car is and how many lanes are above the car. There tends to be a one-to-one mapping between the position on the lower half of the rear projection sphere and all possible three-dimensional (3D) positions of cars on the road behind the driver. For any 3D position of a car on the road behind the driver, the speaker corresponding to that 3D position can be activated. Interpolation of the speaker outputs helps to cue intermediate 3D positions in the absence of any corresponding speakers. For example, refer to Figure 3D , speakers 328a and 328b can be activated (e.g., with different loudness) to simulate the 3D position of the car, which corresponds to the position between the speakers in the back half of the projection sphere. As another example, Figure 3D Speakers 328a and 332a in can be activated to simulate the 3D position of the car, which corresponds to the position between the two speakers in the back half of the projection sphere.

[0142] B. The car tilts due to Doppler. The pitch of a car behind a person depends on their relative speeds. If the car is traveling at the same speed as the person, the car may have some baseline spacing, neither high nor low. The faster the car moves relative to the person (e.g., the faster the car approaches the person), the higher its pitch is above the baseline. The slower the car moves relative to the person (e.g., the car is falling behind), the more likely it is that its pitch will be below the baseline. For a tracked car, the pitch of the auditory sound or stimulus generated and output to the person corresponding to the car may vary (and in some embodiments be exaggerated) depending on the relative speed.

[0143] C. The loudness of the car. The loudness of the activated speakers corresponding to the car behind the person may correspond to the actual loudness of the car, given the actual linear distance of the car. The distance may be calculated directly from the position of the car in the lower rear projection sphere. When the car is physically closer, speakers closer to the horizontal line on the projection sphere (or helmet) may be quieter than speakers activated further along the projection sphere.

[0144] D. Use of tactile stimulation. Tactile stimulation can have two ecologically appropriate effects, the simultaneous relevance of which depends on the following two factors.

[0145] a. When a car behind a person is nearby, the sound made by the car may be loud enough to actually induce tactile stimulation. The tactile intensity of one or more stimulators (e.g., one or more speakers) of the helmet or necklace can vary as the inverse square of the distance between the person and the car. In addition, when nearby, the tactile stimulation can be over a wider area of ​​the stimulator (e.g., speaker), with the peak concentrated at the appropriate location on the rear projection sphere.

[0146] b. Tactile stimulation may also be caused by wind caused by a car approaching a person. Tactile stimulation of the auditory stimulation produced by one or more speakers of a helmet or necklace may be proportional to the Doppler shift above the baseline.

[0147] Improve user safety

[0148] Figure 5 is a flow chart illustrating an exemplary method 500 for improving user security. The method 500 may be embodied as a set of executable program instructions stored on a computer-readable medium (e.g., one or more disk drives) of a computing system or device. For example, Figure 1 As shown in the computing system 600 and described in more detail below, a set of executable program instructions may be executed to implement the method 500. When the method 500 is initiated, the executable program instructions may be loaded into a memory such as a random access memory and executed by one or more processors of the computing system 600. Although the method 500 is relatively simple, the method 500 may be executed by the processor 600. Figure 6 The illustrated computing system 600 is described for illustrative purposes only and is not intended to be limiting. In some embodiments, the method 200 or portions thereof may be executed serially or in parallel by multiple computing systems.

[0149] The computing system may include one or more sensors, or be associated with (e.g., in communication with, such as electrical communication, wired communication, or wireless communication with) one or more sensors. The one or more sensors may be associated with the helmet. The one or more sensors may capture different types of sensor data.

[0150] The computing system may include or be associated with multiple audio output devices (e.g., Figure 3D The speakers 332a, 328a, 324, 328b, 332 described, or referenced Figure 4B2The described speakers 0, 1c', 1b', 1a-1c, 2c', 2b', 2a-2c) are used to output auditory stimulation that imitates sounds from (1a) the user's horizontal line and (1b) below the horizontal line, and from (2a) the user's left side, (2b) directly behind, and (2c) to the right of the computing system. The system can be associated with a helmet or wearable device (such as an augmented reality or mixed reality device), can include a helmet or wearable device, or can be included in a helmet or wearable device. Multiple audio output devices can be on, in, or within the helmet or wearable device. Multiple audio output devices can be positioned to output auditory stimulation that imitates sounds from the user's horizontal line (1a) and (1b) below the horizontal line, and from the user's left side (2a), (2b), and (2c). Figure 3C , 3D , 4B1 and 4B2 and the accompanying descriptions show non-limiting exemplary schematic diagrams of the positioning of multiple audio output devices for emulating sounds from (1a) below the user's horizontal line and (1b) below the horizontal line and from (2a), (2b) and (2c) to the left of the user.

[0151] In some embodiments, the plurality of audio output devices include: (i) horizontal audio output devices (e.g., Figure 3D Speaker 3243D, or Figure 4B1 and 4B2 0) for outputting auditory stimulation that imitates the sound from the center point of the user's horizontal line when wearing a helmet or wearable device; and (ii) a first left horizontal line below audio output device and a first right horizontal line below audio output device (e.g., Figure 3D Speakers 328b, 328a or Figure 4B2 The plurality of audio output devices may include a first profile (e.g., a 45 degree profile) of the speakers 1b and 1b' in the helmet or wearable device, for outputting auditory stimulations imitating sounds from a first vertical level below the horizontal line of the user on the left and right sides of the user respectively when the helmet or wearable device is worn. The plurality of audio output devices may include a central horizontal line below the audio output device (e.g., Figure 4B2 The speaker 1a) in the figure is used to output auditory stimuli simulating sounds coming from below the user's horizontal line and center point.

[0152] In some embodiments, the plurality of audio output devices include (iii) a first left horizontal line below audio output device and a first right horizontal line below audio output device (eg, Figure 3D Speakers 332b, 332a or Figure 4B2A second profile (e.g., a 90 degree profile) of the speakers 2b and 2b') in the embodiment of the present invention is located at, or, when the helmet or wearable device is worn, the helmet or wearable device is used to output auditory stimulation that simulates sound from a second vertical level below the horizontal line of the user on the left and right sides of the user, respectively. The second vertical height can be lower than the first vertical height. The second profile can be lower than the first profile.

[0153] In some embodiments, the plurality of audio output devices include (ii) a second left horizontal line below audio output device and a second right horizontal line below audio output device (eg, Figure 4B2 The speaker 1c and 1c' in the helmet or wearable device, or, when the helmet or wearable device is worn, the helmet or wearable device is used to output auditory stimulation, which simulates the sound from the first vertical level below the user's horizontal line. The sound imitated by the auditory stimulation output by the second left horizontal line below the audio output device can be on the left side of the sound imitated by the auditory stimulation output by the first left horizontal line below the audio output device of the first contour. The sound imitated by the auditory stimulation output by the second right horizontal line below the audio output device can be on the right side of the sound imitated by the auditory stimulation output by the first right horizontal line below the audio output device of the first contour. The second left horizontal line below the audio output device can be on the left side of the first left horizontal line below the audio output device of the first contour. The second right horizontal line below the audio output device can be on the right side of the first right horizontal line below the audio output device of the first contour.

[0154] After method 500 begins at block 504, method 500 proceeds to block 508 where a computing system (e.g., Figure 6 The computing system 600 shown receives first sensor data captured by one or more sensors. In some embodiments, the one or more sensors include an image capture device, a radio detection and ranging (radar), a light detection and ranging (lidar), an audio capture device, or a combination thereof.

[0155] Method 500 proceeds from box 508 to box 512, where the computing system determines a first position of an object relative to a user at a first time using first sensor data (e.g., image data, radar data, lidar data, audio input, telemetry data, and position data). The first position of the object relative to the user at the first time may correspond to a position in the back half of the user's projection sphere, such as the center point of a horizontal line of 0 degrees. The computing system may use one or more machine learning methods or models to determine the first position of the object relative to the user at the first time. The first position of the object relative to the user may be within a first threshold distance of the user. The first threshold distance may be, for example, 1 meter, 2 meters, 5 meters, 10 meters, 20 meters, or 100 meters. For example, the object may be 50 meters behind the user. In some embodiments, the object is a first motor vehicle. In some embodiments, the user is in or riding in a second motor vehicle.

[0156] From block 512, method 500 proceeds to block 516, where the computing system determines one or more first auditory stimuli in a plurality of audio output devices and corresponding one or more first audio output devices for outputting the one or more first auditory stimuli using a first position of an object relative to a user at a first time. A feature of the first auditory stimulus may be associated with a property of the object. A feature of the first auditory stimulus may be associated with at least one property of the object.

[0157] In some embodiments, the one or more first auditory stimuli and the corresponding one or more first audio output devices include two or more first auditory stimuli and corresponding two or more first audio output devices. The combination of the two or more first auditory stimuli output by the corresponding first audio output devices can simulate the sound of an object moving at a first position relative to the user at a first time that none of the multiple audio output devices can simulate alone. For example, Figure 3D Speakers 328a and 328b in can be activated (e.g., at different loudnesses) to simulate the 3D position of the car, which corresponds to the position between the speakers in the back half of the projection sphere. As another example, Figure 3D Speakers 328a and 332a in can be activated to simulate the 3D position of the car, which corresponds to the position between the two speakers in the back half of the projection sphere.

[0158] In some embodiments, one or more first auditory stimuli include a narrow range of frequencies (e.g., 1 Hz, 10 Hz, or 100 Hz) or pure tones (e.g., 250 Hz, 500 Hz, 750 Hz, 800 Hz, 900 Hz, 1000 Hz, 1100 Hz, or 1200 Hz). In some embodiments, one or more first auditory stimuli include music, non-music, speech, non-speech, or a combination thereof.

[0159] In some embodiments, the first auditory stimulation is characterized by the pitch of the first auditory stimulation. The characteristic of the first auditory stimulation may be related to the property of the object, such as the size of the object. The pitch of the first auditory stimulation may be positively correlated or negatively correlated with the size of the object. In some embodiments, the first auditory stimulation is characterized by the loudness of the first auditory stimulation. The characteristic of the first auditory stimulation may be related to the property of the object (e.g., the property of the object at the first time), such as the speed of the object. The loudness of the first auditory stimulation may be positively correlated or negatively correlated with the speed of the object. In some embodiments, the first auditory stimulation is characterized by the pitch of the first auditory stimulation. The characteristic of the first auditory stimulation may be related to the property of the object, such as the size and speed of the object. The pitch of the first auditory stimulation may be positively correlated or negatively correlated with the size of the object. The pitch of the first auditory stimulation may be positively correlated or negatively correlated with the speed of the object.

[0160] From block 516, method 500 proceeds to block 520, where the computing system causes each of the one or more first audio output devices to output a corresponding first auditory stimulus in the one or more first auditory stimuli. For example, when an object approaches, for example, a distance of 50 meters from the user from behind, as shown in reference Figure 4C The speaker 0 starts to produce a single tone at a low volume.

[0161] Object approaching

[0162] In some embodiments, the computing system may receive second sensor data captured by one or more sensors. The computing system may use the second sensor data to determine a second position of the object relative to the user at a second time. The second position of the object relative to the user at the second time may correspond to a position in the back half of the user's projection sphere, such as a position on a 45-degree profile. The second position of the object relative to the user may be within a second threshold distance of the user. The second threshold distance may be, for example, 1 meter, 2 meters, 5 meters, 10 meters, 20 meters, or 100 meters. The second threshold distance may be shorter than (or longer than, or equal to) the first threshold distance. The computing system may determine one or more second auditory stimuli in a plurality of output devices and corresponding one or more second audio output devices for outputting one or more second auditory stimuli using the second position of the object relative to the user at the second time. The computing system may cause each of the one or more second audio output devices to output a corresponding second auditory stimulus in the one or more second auditory stimuli. For example, when an object comes from behind the user (e.g., at a distance of 25 meters), as shown in reference to Figure 4D One of the speakers 1a, 1b and 1c may start generating a plurality of tones at a medium volume according to the trajectory of the car.

[0163] The features of the second auditory stimulation may be related to a property of the object (e.g., a property of the object at the second time). The difference between the features of the second auditory stimulation relative to the features of the first auditory stimulation may be related to the second position of the object relative to the user at the second time. The difference between the features of the second auditory stimulation relative to the features of the first auditory stimulation may be related to the difference between the first position of the object relative to the user at the first time and the second position of the object relative to the user at the second time. The difference between the features of the second auditory stimulation relative to the features of the first auditory stimulation may be positively or negatively correlated with the difference between the first position of the object relative to the user at the first time and the second position of the object relative to the user at the second time.

[0164] In some embodiments, the difference between the features of the second auditory stimulus relative to the features of the first auditory stimulus is a higher or lower number of frequencies. For example, if the object is closer (or farther) from the user at the second time than at the first time, the second auditory stimulus may have a higher number of frequencies or a wider frequency range. As another example, if the object is closer (or farther) from the user at the first time than at the second time, the first auditory stimulus may have a higher number of frequencies or a wider frequency range. In some embodiments, the difference between the features of the second auditory stimulus relative to the features of the first auditory stimulus is a higher pitch or a lower pitch. For example, if the object is closer (or farther) from the user at the second time than at the first time, the second auditory stimulus may have a higher pitch compared to the first auditory stimulus. As another example, if the object is closer (or farther) from the user at the first time than at the second time, the first auditory stimulus may have a higher pitch compared to the second auditory stimulus. In some embodiments, the difference between the features of the second auditory stimulus relative to the features of the first auditory stimulus is a greater loudness or a lower loudness. For example, if the object is closer (or farther) from the user at the second time than at the first time, the second auditory stimulus may be louder than the first auditory stimulus. As another example, if the object is closer to (or farther away from) the user at a first time than at a second time, the first auditory stimulation may be louder than the second auditory stimulation.

[0165] Objects are very close

[0166] In some embodiments, the computing system may: receive third sensor data captured by one or more sensors. The computing system may use the third sensor data to determine a third position of the object relative to the user at a third time. The third position of the object relative to the user may be within a third threshold distance of the user. The second threshold distance may be, for example, 1 meter, 2 meters, 5 meters, 10 meters, 20 meters, or 100 meters. The third threshold distance may be shorter than (or longer than, or equal to) the second threshold distance. The computing system may determine (i) one or more first auditory stimuli and corresponding one or more first audio output devices from a plurality of output devices for outputting one or more first auditory stimuli, and (ii) one or more second auditory stimuli and corresponding one or more second audio output devices from a plurality of output devices for outputting one or more second auditory stimuli using the third position of the object relative to the user at a third time. The computing system may cause each of the one or more first and second audio output devices to output a corresponding first or second auditory stimulus from one or more first and second auditory stimuli. For example, when the object falls behind at a distance of 5 meters from the user, in addition to the speakers 1a, 1b, or 1c, as described in reference Figure 4E The loudspeaker 2a, 2b or 2c starts to produce a plurality of tones at a high volume.

[0167] Object passing by

[0168] In some embodiments, the computing system may receive fourth sensor data captured by one or more sensors. The computing system may use the fourth sensor data to determine that an object is passing by the user, or has a probability of passing by the user at a probability threshold of passing by the user at a passing time. In some embodiments, the probability threshold of the user passing by the object when passing by is at least 50%, 60%, 70%, 80%, 90%, 95%, or 99%. In some embodiments, the object is passing by the user, or has a probability of passing by the user at a probability threshold of passing by the user, within a range of at most 1 meter, 2 meters, 5 meters, or 10 meters of the user. The object may not be in or out of the user's field of view, or may be at or at the far edge of the user's field of view. In some embodiments, the passing time is the time when the fourth sensor data or a portion thereof is captured. The passing time may be the time immediately after the fourth sensor data or a portion thereof is captured (e.g., within 0.0000001 seconds, 0.00001 seconds, 0.0001 seconds, 0.001 seconds, 0.01 seconds, 0.1 seconds, or 1 second)

[0169] The computing system may determine a fourth auditory stimulus to output. The fourth auditory stimulus may have a characteristic corresponding to an auditory characteristic of an object passing by the user. In some embodiments, the characteristic of the fourth auditory stimulus is selected from a group including a decreasing pitch, an increased loudness, a higher number of frequencies, a higher frequency, a regularity of an angle difference, a specificity of an angle difference, a sound shadow, a larger angular size, a loudness asymmetry of the user's ears, a head-related transfer function of approximately 90°, a vertical position below the user's horizontal line, or a combination thereof. In some embodiments, the characteristic change of the second auditory stimulus is fastest or highest at or about the elapsed time.

[0170] The computing system may cause one or more of the plurality of audio output devices to output a fourth auditory stimulus to the user at the output time. The fourth auditory stimulus output to the user at the output time may cause the user to visually perceive the possibility of an object in the far periphery of the user's visual field at the output time or immediately after the output time. The fourth auditory stimulus output to the user at the output time may cause the user to visually perceive the possibility of an object in the far periphery of the user's visual field at the time immediately after the output time increases, such as within 0.00000001 second, 0.000001 second, 0.00001 second, 0.0001 second, 0.0001 second, 0.001 second, 0.001 second, 0.01 second, 0.1 second, or 1 second. The fourth auditory stimulus output to the user at the output time may cause the user to visually perceive the possibility of an object in the far periphery of the user's visual field at the output time or immediately after the output time, such as increasing by at least 10%, 20%, 30%, 40% or 50%. For example, when an object enters the far perimeter or extreme perimeter of the user on a dangerously close or dangerously turning trajectory, Figure 4F Speakers 1b and 2b or speakers 1c and 2c in the FIG. 5A respectively produce a sudden pitch drop to have cross-modal enhancement of motion detection at far periphery or extreme periphery.

[0171] In some embodiments, the object is at the extreme periphery of the user's field of view at the first time. The far periphery of the user's field of view may include about 60° to about 110° temporarily away from the nose and toward the user's temple. The extreme periphery of the user's field of view may include about 90° to 110° temporarily away from the nose and toward the user's temple.

[0172] In some embodiments, the object is passing or may pass by the user on the left side of the user. In order to make one or more of the multiple audio output devices output the fourth auditory stimulus to the user at the output time, the computing system may cause the audio output device below the first left horizontal line to output the second auditory stimulus at the output time. In some embodiments, the object is passing or may pass by the user on the right side of the user. In order to make one or more of the multiple audio output devices output the fourth auditory stimulus to the user at the output time, the computing system may cause the audio output device below the first right horizontal line of the first contour and / or the audio output device below the first right horizontal line of the second contour to output the second auditory stimulus at the output time.

[0173] Method 500 ends at block 524 .

[0174] Computer Vision, Machine Learning, and Neural Networks

[0175] Computer vision. One or more computer vision algorithms may be implemented to enhance visual motion perception and / or improve user safety (e.g., determining the location and trajectory of an object). Non-limiting examples of computer vision algorithms include: Scale Invariant Feature Transform (SIFT), Speeded Up Robust Features (SURF), Oriented Rapid Rotation Brief (ORB), Binary Robust Invariant Scalable Keypoints (BRISK), Fast Retinal Keypoints (FRANK), Viola-Jones algorithm, Eigenfaces method, Lucas Kanade algorithm, Horn-Schunk algorithm, mean shift algorithm, visual simultaneous localization and mapping (vSLAM) technology, sequential Bayesian estimator (e.g., Kalman filter, extended Kalman filter, etc.), bundle adjustment, adaptive threshold (and other thresholding techniques), iterative closest point (ICP), semi-global matching (SGM), semi-global block matching (SGBM), feature point histograms, various machine learning algorithms (e.g., support vector machine, k-nearest neighbor algorithm, naive Bayes, neural network (including convolutional or deep neural network) or other supervised / unsupervised models, etc.), etc. One or more of these computer vision algorithms may be used for trajectory prediction as described herein, for example, to determine a vehicle's trajectory relative to an object.

[0176] Machine learning. Enhancing visual motion perception and / or improving user safety may additionally or alternatively be performed by various machine learning algorithms. Once trained, the machine learning algorithms may be stored in a computing system (e.g., reference Figure 6Some examples of machine learning algorithms may include supervised or unsupervised machine learning algorithms, including regression algorithms (e.g., ordinary least squares regression), instance-based algorithms (e.g., learning vector quantization), decision tree algorithms (e.g., classification and regression trees), Bayesian algorithms (e.g., naive Bayes), clustering algorithms (e.g., k-means clustering), association rule learning algorithms (e.g., a-priori algorithms), artificial neural network algorithms (e.g., perceptrons), deep learning algorithms (e.g., deep Boltzmann machines or deep neural networks), dimensionality reduction algorithms (e.g., principal component analysis), integration algorithms (e.g., stacked generalization), and / or other machine learning algorithms. One or more machine learning algorithms may be used for trajectory prediction as described herein, for example, determining the trajectory of a vehicle relative to an object.

[0177] Neural Networks . A layer of a neural network (NN), such as a deep neural network (DNN), can apply a linear or nonlinear transformation to its input to generate its output. The neural network layer can be a normalization layer, a convolutional layer, a soft design layer, a corrected linear layer, a connection layer, a pooling layer, a loop layer, an initial layer, or any combination thereof. The normalization layer can normalize the brightness of its input to generate its output using, for example, L2 normalization. The normalization layer can, for example, normalize the brightness of multiple images relative to each other at once to generate multiple standardized images as its output. Non-limiting examples of methods for normalizing brightness include local contrast normalization (LCN) or local response normalization (LRN). Local contrast normalization can nonlinearly normalize the contrast of an image by normalizing a local area of ​​the image on a per-pixel basis to have a mean of 0 and a variance of 1 (or other values ​​of the mean and variance). Local response normalization can normalize an image on a local input area to have a mean of zero and a variance of one (or other values ​​of the mean and variance). The normalization layer can speed up the training process.

[0178] A convolutional layer can apply a set of kernels that convolve its input to generate its output. A soft-engineered layer can apply a soft-engineered function to its input. The soft design function (soft design (x) can be, for example, (x / (1+|x|)). The soft design layer may ignore the influence of outliers on each element. The rectified linear layer can be a rectified linear layer unit (ReLU) or a parameterized rectified linear layer unit (PReLU). The ReLU layer can apply a ReLU function to its input to generate its output. The ReLU function ReLU(x) can be, for example, maximum (0, x). The PReLU layer can apply a PReLU function to its input to generate its output. The PReLU function PReLU(x) can be, for example, x if x≥0, and ax if x<0, where a is a positive number. The concatenation layer can concatenate its input to generate its output. For example, the concatenation layer can concatenate four 5×5 images to generate a 20×20 image. The pooling layer can apply a pooling function that downsamples its input to generate its output. For example, the pooling layer can downsample a 20x 20 image to a 10x 10 image. Non-limiting examples of pooling functions include max pooling, average pooling, or min pooling.

[0179] At time point t, the recurrent layer can calculate the hidden state s(t), and the recurrent connection can provide the hidden state s(t) at time t to the recurrent layer as an input at a subsequent time point t+1. The recurrent layer can calculate its output at time t+1 based on the hidden state s(t) at time t. For example, the recurrent layer can apply a soft sign function to the hidden state s(t) at time t to calculate its output at time t+1. The hidden state of the recurrent layer at time t+1 has the hidden state s(t) of the recurrent layer at time t as its input. The recurrent layer can calculate the hidden state as its input. The recurrent layer can calculate the hidden state s(t+1) by applying, for example, a ReLU function to its input. The class initial layer may include one or more of a normalization layer, a convolutional layer, a soft design layer, a rectified linear layer such as a ReLU layer and a PReLU layer, a cascade layer, a pooling layer, or any combination thereof.

[0180] The number of layers in a NN can be different in different implementations. For example, the number of layers in a DNN can be 50, 100, 200, or more. The input types of a deep neural network layer can be different in different implementations. For example, a layer can receive the outputs of multiple layers as its input. The input of a layer can include the outputs of five layers. As another example, the input of a layer can include the 1% layer of a NN. The output of a layer can be the input of multiple layers. For example, the output of a layer can be used as the input of five layers. As another example, the output of a layer can be used as the input of the NN1% layer.

[0181] The input size or output size of a layer can be quite large. The input size or output size of a layer can be nxm, where n represents the width and m represents the height of the input or output. For example, n or m can be 11, 21, 31, or more. In different implementations, the channel size of the input or output of a layer can be different. For example, the channel size of the input or output of a layer can be 4, 16, 32, 64, 128, or more. In different implementations, the kernel size of a layer can be different. For example, the kernel size can be nxm, where n represents the width of the kernel and m represents the height of the kernel. For example, n or m can be 5, 7, 9, or more. The stride size of a layer can be different in different implementations. For example, the stride size of a deep neural network layer can be 3, 5, 7, or more.

[0182] In some embodiments, NN may refer to multiple NNs that together compute neural network outputs. Different NNs in the multiple NNs may be trained for different tasks. A processor (e.g., referring to Figure 6 The processor 610 described above may calculate the output of a NN in the plurality of NNs to determine the output of the NN. For example, the output of the NN in the plurality of NNs may include a likelihood score. The processor may determine the output of a NN including the plurality of NNs based on the likelihood scores of the outputs of different NNs in the plurality of NNs.

[0183] Execution Environment

[0184] Figure 6 Depicted is the general architecture of an example computing device 600 configured to implement the metabolite, annotation, and gene integration system disclosed herein. Figure 6 The general architecture of the computing device 600 depicted in FIG. 6 includes an arrangement of computer hardware and software components. The computing device 600 may include Figure 6 600. As shown in the figure, computing device 600 includes processing unit 610, network interface 620, computer readable medium drive 630, input / output device interface 640, display 650 and input device 660, all of which can communicate with each other through communication bus. Network interface 620 can provide connection to one or more networks or computing systems. Therefore, processing unit 610 can receive information and instructions from other computing systems or services via network. Processing unit 610 can also communicate with memory 670, and further provide output information for optional display 650 through input / output device interface 640. Input / output device interface 640 can also accept input from optional input device 660, such as keyboard, mouse, digital pen, microphone, touch screen, gesture recognition system, voice recognition system, game controller, accelerometer, gyroscope or other input devices.

[0185] Memory 670 may contain computer program instructions (grouped into modules or components in some embodiments) that are executed by processing unit 610 to implement one or more embodiments. Memory 670 typically includes RAM, ROM, and / or other persistent, auxiliary, or non-transitory computer-readable media. Memory 670 may store an operating system 672 that provides computer program instructions for use by processing unit 610 in general management and operation of computing device 600. Memory 670 may also include computer program instructions and other information for implementing aspects of the present disclosure.

[0186] For example, in one embodiment, the memory 670 includes a visual motion perception module 674 for enhancing visual motion perception, such as reference Figure 2 The memory 670 may additionally or alternatively include a user security module 676 for improving user security, such as reference Figure 5 Described method 500. In addition, memory 670 may include, or be in communication with, data storage 690 and / or one or more other data storages that store computer program instructions that memory 670 may store.

[0187] Example

[0188] Some aspects of the embodiments discussed above are disclosed in more detail in the following examples, which are in no way intended to limit the scope of the present disclosure.

[0189] Example 1

[0190] Enhances visual motion perception in the far and extreme periphery

[0191] The effects of auditory stimulation on the detection of extreme peripheral visual motion from behind were investigated.

[0192] It may be ecologically critical for a subject (e.g., a mammal or human) to be able to perceive the extreme periphery because the extreme periphery may be the subject's first point of contact with an object from out of sight. The extreme periphery may be considered the entrance into the subject's field of vision. When an object approaches the subject from behind, the subject's first point of contact with the object may be the subject's extreme periphery. When the subject turns his or her head toward an object behind the subject, similarly, the subject's first point of contact with the object may be the subject's extreme periphery.

[0193] Humans have developed many technologies to see objects in the extreme periphery. For example, a car may have two mirrors that allow the car driver to see objects in the extreme periphery of the driver. A car may have a blind spot warning system designed to facilitate the driver's vision in the extreme periphery. Facilitating extreme peripheral vision is very important for a person's daily life, such as driving a car or riding a motorcycle.

[0194] Surprisingly little is known about vision in the extreme periphery. One possible way that extreme peripheral vision is facilitated is through cross-modal modulation. Concurrent auditory stimulation may facilitate visual perception, such as detection. Cross-modal modulation may be stronger in the periphery than in the fovea. Concurrent auditory stimulation may facilitate vision in the extreme periphery.

[0195] Simultaneous auditory stimulation suppresses visual motion detection in the extreme periphery

[0196] To examine cross-modal effects in the extreme periphery, we investigated whether human subjects could perform a simple visual detection task in the extreme periphery with or without an auditory stimulus (e.g., a beep). In the experiment, a single flash of light was presented to the left and right periphery of the subject. The target location was determined individually and was located, on average, 96.8 degrees from the subject's fovea. Auditory stimuli consisted of simultaneous white noise, simultaneous brown noise, and simultaneous 500 Hz beeps. Baseline was a condition with no sound. Surprisingly, none of the three auditory stimuli promoted visual detection. Two of the three auditory stimuli (white noise and simultaneous 500 Hz beeps) suppressed detection. Similar suppression was observed with the beeps presented in monaural headphones.

[0197] Cross-modal suppression may be the result of auditory stimuli (such as beeps or white noise) being completely irrelevant to the visual stimulus and task. Simultaneous but irrelevant auditory stimuli can suppress visual perception. When visual reliability decreases, hearing becomes critical. Without being limited by theory, the observation of cross-modal suppression caused by irrelevant auditory stimuli may be the result of extreme peripheral vision being highly noisy and therefore susceptible to auditory input.

[0198] Simultaneous auditory stimulation facilitates visual motion detection in the extreme periphery

[0199] Auditory stimulation that is associated (e.g., highly correlated) with visual stimulation may be required to promote cross-modality in the extreme periphery. Certain actions are associated with unique sounds. Detecting passing objects from behind may be an important ecological emergency. Extreme periphery pass-by motion may be physically linked to a unique descending pitch shift. The pitch of the sound originates from the object and suddenly decreases at approximately 90 degrees of the subject's vision. This unique descending pitch shift occurs in the extreme periphery of the subject.

[0200] In an experiment to determine whether the detection of such forward motion could be facilitated by a unique descending pitch shift, a sequence of two 204 millisecond (ms) dots (306 ms stimulus onset asynchrony (SOA)) was presented at the extreme periphery of each subject. The extreme periphery was determined to be the maximum eccentricity with a detection rate of 75% for each subject. Each dot was located at two adjacent locations, 2° apart to allow for apparent forward motion, or at the same location. As auditory stimuli, a descending pitch beep was used simultaneously. The simultaneous delivery of a descending pitch beep can roughly simulate the Doppler pitch shift of a passing object. A simultaneous beep with an ascending pitch was used as a control in addition to another no-sound control. The results showed that the simultaneous delivery of a descending pitch beep improved the hit rate for motion detection compared to no sound and a simultaneous ascending pitch beep.

[0201] Figure 8 is a non-limiting exemplary schematic diagram of an experimental setup used to determine whether hearing affects the detection of extreme peripheral visual motion from behind. Preliminary testing was performed to determine the 75% detectable point on the left or right side of each subject. A first flash was presented at the subject's 75% detectable point, and then a second flash was presented either at the same location as the first flash (to simulate no motion) or two degrees forward from the first flash (to stimulate forward motion). If the second flash appeared at the same location as the first flash, then the subject should observe no motion. If the second flash appeared two degrees forward from the first flash, then the subject should observe forward motion.

[0202] Apparent forward motion is simulated by two consecutive nearby flashes 704a, 704b in the outermost periphery 708 of the subject. Figure 7 As shown, it is investigated whether detection of such forward motion can be achieved by a unique descending pitch shift 712. Motion Detection and Fig. 9 Combination of one of the three sounds shown, all three with separate pitch shift patterns. In the descending pitch shift condition, the pitch of the sound suddenly decreased from 1000 Hz to 900 Hz during the first and second flash presentations. For the control condition, the pitch of the sound suddenly increased from 900 Hz to 1000 Hz during the first and second flash presentations. For the other control, the pitch of the sound was set to 950 Hz. The three sounds had the same loudness modulation to simulate an object approaching and then moving away.

[0203] Fig.10is a non-limiting example diagram showing the effects of three types of auditory stimulation on the detection of extreme peripheral visual motion from behind. The results of cross-modal promotion are expressed as the difference d' between each sound condition and the no-sound baseline calculated using signal detection theory. The effect of auditory stimulation is F(1,17)=8.27, p.=0.010, where F(1,17) is the F value obtained from the analysis of variance (ANOVA) with 1 and 17 degrees of freedom. The cross-modal promotion in the descending pitch offset condition is significantly greater than that in the rising pitch offset condition and the constant pitch condition (p=0.031 and p=0.031, respectively). This cross-modal promotion is greater than zero only when the pitch offset decreases (p=0.010). The results clearly show that the descending pitch offset condition contributes to the detection of forward motion in the extreme periphery. Constant tone control with loudness changes of 90-degree peak has no promoting effect. The loudness changes that mimic passing objects may not be sufficient to promote motion detection in the extreme periphery. Reducing the spacing may be an important clue to detect "passing by" objects in the extreme periphery.

[0204] Each subject completed a post-hoc questionnaire. The post-hoc questionnaire was used to determine each subject's knowledge of the relationship between the sound the subject heard and the motion that elicited the sound the subject heard. The motion options included "moving toward you," "moving away from you," "passing by you," "moving around you at the same distance," "moving past you," and "none of the above." Surprisingly, almost no one could identify the correct motion. Only two subjects (11.8%) who heard the sound in the falling pitch shift condition correctly identified the motion of the object "passing by you" that elicited the sound heard. None of the subjects who heard the sound with the rising pitch condition correctly identified the "none of the above" motion that elicited the sound heard. Only one subject (5.9%) who heard the sound in the constant pitch condition correctly identified the "none of the above" motion that elicited the sound heard. The subjects' lack of knowledge about the motion that elicited the sound the subject heard means that the observed cross-modal facilitation cannot be explained by knowledge-based effects alone.

[0205] Concurrent cross-modal stimulation that is naturally and ecologically relevant to visual stimulation can facilitate extreme peripheral visual perception. Such ecologically relevant auditory stimulation can facilitate visual perception in the extreme periphery even if subjects cannot perceive the auditory stimulation correctly and / or subjects have no top-down knowledge about the association between auditory and visual stimulation. A descending pitch shift condition, but not a loudness change, can facilitate forward motion detection in the extreme periphery. Loudness changes do not necessarily indicate a passing object and may not be associated with cross-modal facilitation. Auditory stimulation that is unique to visual stimulation can facilitate cross-modal facilitation.

[0206] Overall, these data suggest that incorporating a descending pitch offset into sound can facilitate forward visual motion detection in the extreme periphery.

[0207] Other considerations

[0208] In at least some of the previously described embodiments, one or more elements used in one embodiment may be used interchangeably in another embodiment, unless such replacement is technically infeasible. It will be appreciated by those skilled in the art that various other omissions, additions, and modifications may be made to the above methods and structures without departing from the scope of the claimed subject matter. All of these modifications and variations are intended to fall within the scope of the subject matter defined by the appended claims.

[0209] Those skilled in the art will appreciate that for this and other processes and methods disclosed herein, the functions performed in the processes and methods may be implemented in different orders. In addition, the steps and operations outlined are provided only as examples, and some steps and operations may be optional, combined into fewer steps and operations, or expanded into additional steps and operations without departing from the essence of the disclosed embodiments.

[0210] With regard to the use of substantially any plural and / or singular terms herein, those skilled in the art may translate the plural into the singular and / or the singular into the plural, depending on the context and / or application. For the sake of clarity, various singular / plural arrangements may be explicitly set forth herein. As used in this specification and the appended claims, the singular forms "one", "an" and "the" include plural references unless the context clearly dictates otherwise. Therefore, phrases such as "configured to" are intended to include one or more enumerated devices. Such one or more enumerated devices may also be configured together to perform the enumeration. For example, a processor configured to perform enumeration A, B, and C may include a first processor configured to perform enumeration A, and work in conjunction with a second processor configured to perform enumeration B and C. Unless otherwise noted, any reference to "or" in this document is intended to include "and / or".

[0211] Those skilled in the art will understand that, in general, the terms used herein, and particularly in the appended claims (e.g., the subjects of the appended claims), are generally intended to be “open” terms (e.g., the term “including” should be interpreted as “including but not limited to,” the term “having” should be interpreted as “having at least,” the term “comprising” should be interpreted as “including but not limited to,” etc.). Those skilled in the art will further understand that if a specific number of claim recitations is intended to be introduced, such intent will be explicitly recited in the claim, and in the absence of such recitation, no such intent is present. For example, to aid understanding, the following appended claims may contain the use of the introductory phrases “at least one” and “one or more” to introduce claim recitations. However, the use of such phrases should not be interpreted as implying that the introduction of a claim recitation by the indefinite article “a” or “an” limits any particular claim containing such introduced claim recitation to containing only one embodiment of such recitation, even when the same claim includes the introductory phrases “one or more” or “at least one” and an indefinite article such as “a” or “an” (e.g., “a” and / or “an” should be interpreted to mean “at least one” or “one or more”); the use of definite articles to introduce claim recitations should also be interpreted as meaning “at least one” or “one or more”. This is the case. Furthermore, even if a specific number of an introduced claim enumeration is explicitly recited, one skilled in the art will recognize that such recitation should be interpreted to mean at least the recited number (e.g., a simple recitation of "two enumerations" without other modifiers means at least two enumerations, or two or more enumerations). Furthermore, in the case of a convention similar to "at least one of A, B, and C", "a system having at least one of A, B, and C" would include, but is not limited to, a system having A alone, B alone, C alone, A and B together, A and C together, B and C together, and / or A, B, and C together, etc. In a convention similar to "at least one of A, B, and C", "a system having at least one of A, B, and C" would include, but is not limited to, a system having A alone, B alone, C alone, A and B together, A and C together, B and C together, and / or A, B, and C together, etc. In the case of the convention of "is one of A, B or C", "a system having at least one of A, B or C" will include but is not limited to systems having A alone, B alone, C alone, A and B together, A and C together, B and C together, and / or A, B and C together, etc. Those skilled in the art will further understand that almost any discrete word and / or phrase presenting two or more alternative terms, whether in the specification, claims or drawings, should be understood to include the possibility of one, either or both of these terms. For example, the phrase "A or B" will be understood to include the possibility of "A" or "B" or "A and B".

[0212] In addition, where features or aspects of the disclosure are described in terms of Markush groups, those skilled in the art will recognize that the disclosure is also thereby described in terms of any individual member or subgroup of members of the Markush group.

[0213] As will be understood by those skilled in the art, for any and all purposes, such as in providing a written description, all ranges disclosed herein also include any and all possible sub-ranges and combinations of sub-ranges thereof. Any listed range can be easily identified as fully describing and capable of decomposing the same range into at least equal halves, thirds, quarters, fifths, tenths, etc. As a non-limiting example, each range discussed here can be easily decomposed into a lower third, a middle third, and an upper third, etc. It will also be understood by those skilled in the art that all languages, such as "up to", "at least", "greater than", "less than", etc., include the numbers listed and refer to ranges that can be subsequently decomposed into sub-ranges as described above. Finally, as will be understood by those skilled in the art, a range includes each individual member. Thus, for example, a group with 1-3 clauses refers to a group with 1, 2, or 3 clauses. Similarly, a group with 1-5 clauses refers to a group with 1, 2, 3, 4, or 5 clauses, and so on.

[0214] It should be understood that various embodiments of the present disclosure have been described herein for illustrative purposes, and that various modifications may be made without departing from the scope and spirit of the present disclosure. Therefore, the various embodiments disclosed herein are not intended to be limiting, and the true scope and spirit are indicated by the appended claims.

[0215] It should be understood that not all objects or advantages may be achieved according to any particular embodiment described herein. Thus, for example, those skilled in the art will recognize that certain embodiments may be configured to operate in a manner that achieves or optimizes one advantage or group of advantages as taught herein without necessarily achieving other objects or advantages as taught or suggested herein.

[0216] All processes described herein may be contained in and fully automated by a software code module executed by a computing system including one or more computers or processors. The code module may be stored in any type of non-transitory computer-readable medium or other computer storage device. Some or all methods may be implemented in dedicated computer hardware.

[0217] According to the present disclosure, many other changes other than those described here will be apparent. For example, according to an embodiment, some actions, events or functions of any algorithm described here can be performed in different orders, can be added, merged or completely omitted (for example, not all described actions or events are necessary for the practice of the algorithm). In addition, in some embodiments, actions or events can be performed simultaneously, such as by multithreading, interrupt processing or multiple processors or processor cores or on other parallel architectures, rather than sequential execution. In addition, different tasks or processes can be performed by different machines and / or computing systems that can work together.

[0218] The various illustrative logic blocks and modules described in conjunction with the embodiments disclosed herein may be implemented or executed by a machine, such as a processing unit or processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic, a discrete hardware component or any combination thereof designed to perform the functions described herein. The processor may be a microprocessor, but alternatively, the processor may be a controller, a microcontroller or a state machine, a combination thereof, and the like. The processor may include a circuit configured to process computer executable instructions. In another embodiment, the processor includes an FPGA or other programmable device that performs logical operations without processing computer executable instructions. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, a combination of one or more microprocessors and a DSP core, or any other such configuration. Although described primarily with respect to digital technology, the processor may also include primarily analog components. For example, some or all of the signal processing algorithms described herein may be implemented in analog circuits or analog and digital hybrid circuits. The computing environment may include any type of computer system, including but not limited to a microprocessor-based computer system, a mainframe computer, a digital signal processor, a portable computing device, a device controller, or a computing engine within a device, and the like.

[0219] Any process description, element or block in the flowcharts described herein and / or depicted in the accompanying drawings should be understood to potentially represent a module, segment or portion of code that includes one or more executable instructions for implementing a specific logical function or element in the process. Alternative implementations are included within the scope of the embodiments described herein, in which elements or functions may be deleted, performed or discussed out of the order shown, including being performed substantially simultaneously or in reverse order, depending on the functions involved, as understood by those skilled in the art.

[0220] It should be emphasized that many changes and modifications may be made to the above-described embodiments, and the elements of these embodiments should be understood as being in other acceptable examples. All such modifications and changes are intended to be included within the scope of this disclosure and protected by the accompanying claims.

Claims

1. A system for improving visual motion perception of an object by a user of the system, comprising: a non-transitory memory configured to store executable instructions; Audio output devices; as well as A hardware processor in communication with a non-transitory memory, an image capture device, and an audio output device, the hardware processor being programmed with executable instructions to: receiving a plurality of images captured by the image capture device; determining a position and trajectory of an object relative to a user at a first time using the plurality of images; determining, based on the position and trajectory of the object relative to the user at the first time, that a likelihood of the object passing the user at the elapsed time is above a likelihood threshold, wherein the object is not in a field of view of the user at the first time and / or the elapsed time or is in a far periphery of the field of view of the user, and wherein the far periphery of the field of view of the user includes approximately 60° to approximately 110° away from the user's nose and toward the temple in time; determining an auditory stimulus and an output time of outputting the auditory stimulus based on the position and trajectory of the object relative to the user at the first time, wherein the output time is an elapsed time or the output time is between the first time and the elapsed time, and wherein a feature of the auditory stimulus is selected from the group consisting of a descending pitch; and The audio output device is caused to output auditory stimulation to the user at the output time, thereby increasing the likelihood that the user will visually perceive an object in the far periphery of the user's field of view at or immediately after the output time.

2. The system according to claim 1, wherein: In order to receive the multiple images, the hardware processor is programmed by the executable instructions to: receive sensor data including the multiple images captured by one or more sensors, wherein the one or more sensors include image capture devices, radio detection and ranging (radar), light detection and ranging (lidar), audio capture devices or a combination thereof, wherein in order to determine the position and trajectory of the object relative to the user at the first time, the hardware processor is programmed by the executable instructions to: use the sensor data to determine the position and trajectory of the object relative to the user at the first time, or wherein, in order to determine that the object is passing by the user at a passing time, or has a probability above a probability threshold of passing the user, the hardware processor is programmed by the executable instructions to: use the sensor data to determine that the object is passing by the user at a passing time, or has a probability above a probability threshold of passing the user.

3. The system according to claim 2, wherein: The system comprises the one or more sensors or one or more of the one or more sensors.

4. The system according to claim 1, wherein: The system includes the image capture device.

5. The system according to claim 2, wherein: To receive the sensor data, the hardware processor is programmed by the executable instructions to receive sensor data captured by the one or more sensors from a second system.

6. The system according to any one of claims 2, 3 and 5, wherein: The hardware processor is programmed by the executable instructions to transmit the sensor data, the position and trajectory of the object relative to the user at the first time, and / or an indication that the object is passing the user or has a likelihood above a likelihood threshold of passing the user at the passing time to a third system.

7. The system according to any one of claims 1 to 5, wherein: A helmet, car audio system, or wearable speaker system or device is associated with, includes, or is contained in the system or a portion thereof.

8. The system according to any one of claims 1 to 5, wherein: The system includes: multiple audio output devices for outputting auditory stimuli that simulate sounds from (1a) the horizon and (1b) below the horizon of the user, and from (2a) the left side, (2b) directly behind, and (2c) the right side of the user.

9. The system according to any one of claims 1 to 5, wherein: The object is a first motor vehicle.

10. The system according to any one of claims 1 to 5, wherein: The user is in or riding in a second motor vehicle.

11. The system according to any one of claims 1 to 5, wherein: The hardware processor may be programmed by the executable instructions to determine, based on the position and trajectory of the object relative to the user at a first time, that the object has a likelihood above a likelihood threshold of passing the user at the passing time.

12. The system according to any one of claims 1 to 5, wherein: The threshold probability that the object passes the user at the passing time is at least 50%, 60%, 70%, 80%, 90%, 95% or 99%.

13. The system according to any one of claims 1 to 5, wherein: The position and trajectory of the object relative to the user at the first time indicates that the object is within a range of at most 1 meter, 2 meters, 5 meters or 10 meters of the user, with a likelihood above a likelihood threshold of passing the user at the passing time.

14. The system according to any one of claims 1 to 5, wherein: The object is at the extreme periphery of the user's field of view at the first time.

15. The system of claim 14, wherein: The extreme periphery of the user's field of view includes approximately 90° to 110° in time away from the user's nose and toward the temple.

16. The system according to any one of claims 1 to 5, wherein: The characteristics of the auditory stimulation are selected from the group consisting of decreasing pitch, increasing loudness, higher number of frequencies, higher frequencies, regularity of angular differences, specificity of angular differences, sound shadowing, larger angular size, loudness asymmetry of the user's ears, a head-related transfer function of approximately 90°, a vertical position below the user's horizontal line, or a combination thereof.

17. The system according to any one of claims 1 to 5, wherein: The characteristic changes of the auditory stimulus are fastest or highest at or about the elapsed time.

18. The system according to any one of claims 1 to 5, wherein: The auditory stimuli include a narrow frequency range or pure tones.

19. The system according to any one of claims 1 to 5, wherein: The auditory stimulation includes music, speech or a combination thereof.

20. The system according to any one of claims 1 to 5, wherein: The first time is the time when the multiple images or a portion thereof are captured, wherein the first time is the time after the multiple images or a portion thereof are captured, or wherein the first time is the time immediately after the multiple images or a portion thereof are captured.

21. The system according to any one of claims 1 to 5, wherein: The position and trajectory of the object relative to the user indicate that the object may pass by the user on the left or right side of the user at the passing time, wherein the audio output device includes a left audio output device and a right audio output device configured to output the auditory stimulation to the left ear and the right ear of the user, respectively, and wherein, in order to cause the audio output device to output the auditory stimulation to the user at the output time, the hardware processor is programmed by executable instructions to: cause the left audio output device or the right audio output device to output the auditory stimulation to the left ear or the right ear of the user, respectively, at the output time.

22. The system according to any one of claims 1 to 5, wherein: The output time is an elapsed time, the output time is immediately before the elapsed time, or the output time is between the first time and the elapsed time.

23. A system according to any of claims 1-5, thereby increasing the likelihood that the user will visually perceive objects in the far periphery of the user's field of view at the output time or immediately after the output time by at least 10%, 20%, 30%, 40% or 50%.

24. A method for improving visual motion perception of an object by a subject, comprising: Under the control of the hardware processor: receiving sensor data captured by one or more sensors; using the sensor data to determine a position and trajectory of an object relative to a subject at a first time; determining that a likelihood that the object moved relative to the subject within a threshold distance of the subject at a second time is above a likelihood threshold, wherein the object was not within the subject's field of view or was at the periphery of the subject's field of view at a first time; determining an auditory stimulus based on the position and trajectory of the object relative to the subject at the first time, wherein a characteristic of the auditory stimulus is selected from the group consisting of a descending pitch; and The auditory stimulation is caused to be output to the subject at the output time, thereby increasing the possibility that the subject visually perceives an object in the periphery of the subject's visual field at the output time or immediately after the output time.

25. The method according to claim 24, wherein: The one or more sensors include image capture devices, radio detection and ranging (radar), light detection and ranging (lidar), audio capture devices, or combinations thereof.

26. The method according to any one of claims 24-25, wherein: The helmet, car audio system, or wearable speaker system or device is associated with, includes, or is included in the one or more sensors, and wherein causing the auditory stimulus to be output includes causing an audio output device of the helmet, car audio system, or wearable speaker system or device to output the auditory stimulus at an output time.

27. The method according to claim 26, wherein: The helmet, car audio system or wearable speaker system or device includes: multiple audio output devices for outputting auditory stimuli, which imitate sounds from the user's (1a) horizon and (1b) below the horizon and from the user's (2a) left side, (2b) directly behind, and (2c) right side.

28. The method according to any one of claims 24-25, wherein: The sensor data includes a position, velocity, and orientation of the object captured by one or more telemetry and positioning sensors associated with the object, and wherein the one or more sensors include one or more telemetry and positioning sensors associated with the object.

29. The method according to any one of claims 24-25, wherein: The sensor data includes the position, velocity, and orientation of the subject captured by one or more telemetry and positioning sensors associated with the object, and wherein the one or more sensors include one or more telemetry and positioning sensors associated with the subject.

30. The method according to any one of claims 24-25, wherein: The one or more telemetry and positioning sensors associated with the object include a global positioning system (GPS) sensor associated with the object, and wherein the one or more telemetry and positioning sensors associated with the subject include a global positioning system sensor (GPS) associated with the subject.

31. The method according to any one of claims 24-25, wherein: The object is a first motor vehicle.

32. The method according to any one of claims 24-25, wherein: The subject is in or riding in a second motor vehicle.

33. The method according to any one of claims 24-25, comprising: Based on the position and trajectory of the object relative to the subject at the first time, it is determined that the object has a likelihood exceeding a threshold likelihood of moving relative to the subject within a threshold distance of the subject at the second time.

34. The method according to any one of claims 24-25, wherein: The position and trajectory of the object relative to the subject at a first time indicates that the object has a high probability of passing over or striking the subject at a second time.

35. The method according to any one of claims 24-25, wherein: The threshold likelihood that the object is moving relative to the subject within the threshold distance of the subject at the second time is at least 50%, 60%, 70%, 80%, 90%, 95% or 99%.

36. The method according to any one of claims 24-25, wherein: The threshold distance is at most 0m, 1m, 2m, 5m or 10m.

37. The method according to any one of claims 24-25, wherein: The object is at the far periphery or extreme periphery of the subject's visual field at a first time and / or a second time.

38. The method of claim 37, wherein: The far periphery of the subject's visual field includes about 60° to about 110° temporally away from the subject's nose and toward the temple, and / or wherein, The extreme periphery of the subject's visual field includes about 90° to about 110° in time away from the subject's nose and toward the temple.

39. The method according to any one of claims 24-25, wherein: The characteristics of the auditory stimulus are selected from the group consisting of decreasing pitch, increasing loudness, higher number of frequencies, higher frequencies, regularity of angular differences, specificity of angular differences, sound shadowing, larger angular size, loudness asymmetry in the subject's ears, a head-related transfer function of approximately 90°, a vertical position below the subject's horizontal line, or a combination thereof.

40. The method according to any one of claims 24-25, wherein: Determining the auditory stimulus includes determining the auditory stimulus and the output time based on the position and trajectory of the object relative to the subject at the first time.

41. The method according to any one of claims 24-25, wherein: The characteristic changes of auditory stimuli are fastest or highest at or about the elapsed time.

42. The method according to any one of claims 24-25, wherein: The auditory stimuli include a narrow range of frequencies or pure tones.

43. The method according to any one of claims 24-25, wherein: The auditory stimulation includes music, speech or a combination thereof.

44. The method according to any one of claims 24-25, wherein: The first time is a time when the sensor data or a portion thereof is captured, wherein the first time is a time after the sensor data or a portion thereof is captured, or wherein the first time is a time immediately after the sensor data or a portion thereof is captured.

45. The method according to any one of claims 24-25, wherein: The position and trajectory of the object relative to the subject indicates that the object may pass by or hit the subject on the left or right side of the subject at the second time, and wherein outputting the auditory stimulation to the subject at the output time includes outputting the auditory stimulation to the left ear or right ear of the subject, respectively, at the output time.

46. ​​The method according to any one of claims 24-25, wherein: The output time is the second time, the output time is immediately before the second time, or is between the first time and the second time.

47. A method according to any of claims 24-25, thereby increasing the likelihood that a subject will visually perceive objects in the periphery of the user's field of view at or immediately after the output time by at least 10%, 20%, 30%, 40% or 50%.

Images