Object perception method and device based on binaural hearing signal representation and hearing aid

Abstract

The application provides an object perception method and device based on binaural hearing signal expression and a hearing aid, and belongs to the technical field of hearing aids. The object perception method based on binaural hearing signal expression integrates radar perception, object analysis, target identification, spatial sound effect simulation and hearing aid broadcasting and the like, realizes intuitive perception of a target object to a surrounding environment in a vision-limited or hearing-limited scene, and enables the target object to understand the dynamics of the surrounding environment in real time without relying on visual information, thereby enhancing the auxiliary ability of a wearable device to a user in a complex scene. Compared with traditional visual display or voice prompt, the feedback mode of the hearing signal is more suitable for the needs of vision-limited users, hearing-limited users or in a specific complex scene, and can effectively improve the use experience of the target object.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of hearing aids, in particular to an object perception method and device based on binaural hearing signal expression and a hearing aid. BACKGROUND

[0002] With the continuous progress of science and technology, wearable devices are increasingly popular in daily life and work. Using these devices to help users perceive obstacles in the surrounding environment is of great significance for specific groups or application scenarios. The core technology of obstacle perception is to measure and feedback the position and movement information of surrounding objects. Currently, radar ranging technology has been widely used in automatic driving, unmanned aerial vehicles and intelligent wearable devices, especially in frequency-modulated continuous wave (FMCW) millimeter wave technology. Unlike traditional pulse radar systems, frequency-modulated continuous wave radar can measure the distance, angle and speed of multiple objects simultaneously, and these three parameters are sufficient to reflect the spatial position and dynamic information of the object.

[0003] A complete millimeter wave radar system includes transmitting and receiving radio frequency components, analog circuits and digital circuits. Traditional radar systems are usually built using discrete components, which not only increases power consumption and system cost, but also poses great challenges in design due to the complexity and high frequency of the system. In general, existing single-chip radar sensor solutions have achieved high integration, miniaturization and low power consumption, and can be widely used in personal wearable devices.

[0004] However, traditional radar signals are usually presented in digital or image form. If a wearable device relies on visual display to deliver this information, this approach is not user-friendly for visually impaired users or users in certain specific scenarios. In addition, the auditory feedback method based on voice, although effective, is often not intuitive enough to meet the needs of complex scenarios. SUMMARY

[0005] The present application provides an object perception method and device based on binaural hearing signal expression and a hearing aid to solve the problem of the lack of intuitive feedback information based on simple voice in complex scenarios in the prior art, and to achieve a more adaptive effect for visually impaired users, hearing impaired users or in specific complex scenarios.

[0006] The present application provides an object perception method based on binaural hearing signal expression, comprising:

[0007] Obtaining the position information of the objects around the target object by the radar perception device carried by the target object; the position information of the objects includes the distance and angle of the objects from the target object;

[0008] analyzing the position information of the object, determining the type and movement trend of the object;

[0009] determining the target object from the object based on the type, movement trend, distance and angle of the object relative to the target object;

[0010] generating a target auditory signal for simulating the perception of the state information of the target object by the binaural ears of the person at the position of the target object based on the type, movement trend, distance and angle of the target object relative to the target object; the state information of the target object includes type, distance, angle and movement trend;

[0011] playing the target auditory signal to the target object through the hearing aid worn by the target object, so that the target object obtains the state information of the target object.

[0012] According to the object perception method based on binaural auditory signal expression provided by the application, the target auditory signal for simulating the perception of the state information of the target object by the binaural ears of the person at the position of the target object is generated based on the type, movement trend, distance and angle of the target object relative to the target object, and includes:

[0013] determining the timbre of the target auditory signal based on the type and distance of the target object relative to the target object; determining the intensity, binaural intensity difference and binaural time difference of the target auditory signal based on the distance and angle of the target object relative to the binaural ears of the target object; determining the frequency characteristics of the target auditory signal based on the movement trend of the target object;

[0014] generating the target auditory signal based on the determined timbre, intensity, binaural intensity difference, binaural time difference and frequency characteristics.

[0015] According to the object perception method based on binaural auditory signal expression provided by the application, the frequency characteristics of the target auditory signal are determined based on the movement trend of the target object, and include:

[0016] In the case that the movement speed of the target object is less than a speed threshold, the movement trend of the target object is determined to be a stationary state;

[0017] The frequency characteristics of the target auditory signal are determined to be a fixed-frequency pulse tone.

[0018] According to the object perception method based on binaural auditory signal expression provided by the application, the frequency characteristics of the target auditory signal are determined based on the movement trend of the target object, and include:

[0019] In a case that the moving speed of the target object is greater than or equal to a speed threshold, the moving trend of the target object is determined as a moving state;

[0020] The carrier frequency of the target auditory signal is determined as an initial frequency, and a frequency modulation signal of the target auditory signal is determined based on the moving state of the target object to adjust the initial frequency; wherein the frequency modulation signal adjusts the size of the carrier frequency in association with the moving speed of the target object, and the size of the adjusted carrier frequency is associated with the distance between the target object and the target object after the target object moves;

[0021] The frequency feature of the target auditory signal is determined based on the frequency modulation signal and the initial frequency.

[0022] According to the object perception method based on the binaural auditory signal expression provided by the application, the timbre of the target auditory signal is determined based on the type of the target object and the distance between the target object and the target object, and the method comprises the following steps:

[0023] Based on the type of the target object, a basic timbre model corresponding to the target object is determined;

[0024] According to the distance of the target object, the attenuation rate of the sound is calculated by an attenuation function, the high-frequency component and the low-frequency component in the spectrum of the basic timbre model corresponding to the target object are adjusted, and the timbre of the target auditory signal is determined.

[0025] According to the object perception method based on the binaural auditory signal expression provided by the application, the target object is determined from the objects based on the type of the object, the moving trend of the object, and the distance and angle between the object and the target object, and the method comprises the following steps:

[0026] Based on the distance and angle between the object and the target object, the objects in the target range are determined;

[0027] In a case that the number of the objects in the target range is greater than a target number, it is determined whether the objects in the target range are still in the target range after a target time period based on the moving trend of the objects in the target range;

[0028] Based on the first number of the objects in the target range that are still in the target range after the target time period and a preset priority, a second number of objects are selected from the objects in the target range that are in a stationary state based on the moving trend of the objects in the target range;

[0029] The first number of the objects in the target range that are in a moving state and the second number of the objects in the target range that are in a stationary state are determined as the target object.

[0030] According to the object perception method based on binaural hearing signal expression provided by the application, the target hearing signal is broadcast to the target object through a hearing aid worn by the target object, and the method comprises the following steps:

[0031] When the number of the target objects is greater than one, the corresponding target hearing signal is broadcast to the target object through the hearing aid worn by the target object at intervals according to the sound-emitting priority corresponding to each target object; the sound-emitting priority is determined based on the type, motion trend, distance and angle of the target object.

[0032] The application further provides an object perception device based on binaural hearing signal expression, which comprises the following components:

[0033] The acquisition module is used to acquire the position information of the objects around the target object through the radar perception device carried by the target object; the position information of the objects comprises the distance and angle of the objects from the target object.

[0034] The analysis module is used to analyze the position information of the objects and determine the type and motion trend of the objects.

[0035] The first processing module is used to determine the target object from the objects based on the type, motion trend, distance and angle of the objects from the target object.

[0036] The second processing module is used to generate the target hearing signal for simulating the binaural perception of the state information of the target object by a person at the position of the target object based on the type, motion trend, distance and angle of the target object from the target object; the state information of the target object comprises the type, distance, angle and motion trend.

[0037] The feedback module is used to broadcast the target hearing signal to the target object through the hearing aid worn by the target object, so that the target object acquires the state information of the target object.

[0038] The application further provides a hearing aid, which comprises a microphone, an audio processing module, a communication module, a power module and a loudspeaker; the hearing aid broadcasts the target hearing signal to the target object by using the object perception method based on binaural hearing signal expression according to any one of the above-mentioned methods, so that the target object acquires the state information of the target object.

[0039] The application further provides an electronic device, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor; when the processor executes the program, the object perception method based on binaural hearing signal expression according to any one of the above-mentioned methods is realized.

[0040] The application further provides a non-transitory computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the object perception method based on binaural hearing signal expression of any of the above.

[0041] The application further provides a computer program product comprising a computer program which, when executed by a processor, implements the object perception method based on binaural hearing signal expression of any of the above.

[0042] The object perception method based on binaural hearing signal expression, the device and the hearing aid provided by the application realize intuitive perception of a target object to the surrounding environment in a vision-limited or hearing-limited scene by integrating radar perception, object analysis, target recognition, spatial sound effect simulation and hearing aid broadcasting, etc., so that the target object can understand the dynamics of the surrounding environment in real time without relying on visual information, and the auxiliary ability of a wearable device to a user in a complex scene is enhanced. Compared with traditional visual display or voice prompt, the feedback mode of the hearing signal is more suitable for the needs of vision-limited users, hearing-limited users or in a specific complex scene, and can effectively improve the use experience of the target object. BRIEF DESCRIPTION OF DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are some embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0044] Figure 1 is one of the flowcharts of the object perception method based on binaural hearing signal expression provided by the application;

[0045] Figure 2 is the second flowchart of the object perception method based on binaural hearing signal expression provided by the application;

[0046] Figure 3 is the structural schematic diagram of the object perception device based on binaural hearing signal expression provided by the application;

[0047] Figure 4 is the structural schematic diagram of the electronic device provided by the application. DETAILED DESCRIPTION

[0048] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below in combination with the drawings in the present application. Obviously, the described embodiments are part of, but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0049] The present application will be described below in combination with Figures 1-4 The object perception method, device and hearing aid based on binaural hearing signal expression of the present application are described.

[0050] As Figure 1 shown, the object perception method based on binaural hearing signal expression of the embodiment of the present application mainly includes steps 110, 120, 130, 140 and 150.

[0051] In step 110, the position information of the objects around the target object is acquired by the radar perception device carried by the target object.

[0052] The position information of the object includes the distance and angle of the object from the target object.

[0053] The target object refers to the user carrying the system, which can be a visually impaired or hearing impaired individual or other individual in need.

[0054] The radar perception device is a smart device containing a sensor, which is usually an ultrasonic radar, millimeter wave radar or laser radar, for detecting objects in the surrounding environment and acquiring their position information. By emitting signals and receiving the returned reflected waves, the radar device can estimate the distance, angle and relative motion information of the object.

[0055] The radar perception device can be a separate wearable device, can be integrated into other user terminals, or integrated into the user's hearing aid, and can communicate with the hearing aid, and the form of the radar perception device is not limited here.

[0056] In this embodiment, the radar sensing device can include a 77GHz RFCMOS millimeter wave radar sensor. In recent years, with the wide application of radar technology in the fields of automobiles, medical treatment, etc., the 77GHz RFCMOS millimeter wave sensor product has been very mature. Such a sensor integrates all radio frequency and analog functions, such as VCO, PLL, PA, LNA, mixer, and ADC of multiple TX / RX channels, and also integrates digital components such as MCU, accelerator, or digital signal processor. Unlike traditional SiGe-based solutions, the radio frequency front end and digital back end of the 77GHz millimeter wave radar sensor are more flexible and programmable, and can be customized according to requirements to maximize the extraction of useful data in the scene. Through a single frequency-modulated signal and real-time adjustment of the processing back end, higher measurement range, speed, resolution, and specific algorithm requirements can be met. Compared with 24GHz radar, the 7GHz sensor has higher resolution in distance and speed, and its volume is smaller, with an antenna size of about one-third of the original.

[0057] The radar device worn by the target object can scan the surrounding environment in real time and detect all objects within the detection range. The information of these objects, including distance, angle, and possible motion state, will be transmitted to the processing unit through the radar device.

[0058] It can be understood that through the radar device, the user can perceive the position of the surrounding objects without visual input, increasing his perception of the environment.

[0059] Step 120, analyzing the position information of the object to determine the type and motion trend of the object.

[0060] According to the characteristics of the radar echo, the nature of the object can be analyzed, such as a stationary object, a pedestrian, a vehicle, etc.

[0061] According to the change in the position of the object relative to the target object, it can be analyzed whether the object is stationary, approaching, moving away, or moving laterally.

[0062] The position information of the object sensed by the radar (including distance, angle) can be processed. Combined with the motion trajectory of the object, the type of the object (such as a person, a vehicle, an obstacle) and its motion trend can be inferred. For example, the object is approaching the target object, moving away from the target object, or remaining stable.

[0063] In a complex environment, the motion information of multiple objects can be comprehensively judged to identify the most important target object.

[0064] It can be understood that the most important objects and their dynamic states to the target object are identified, which helps to generate accurate feedback signals subsequently.

[0065] The following explains how to measure distance, angle and velocity using a frequency-modulated continuous wave (FMCW) millimeter wave radar as an example.

[0066] In a FMCW radar signal, the frequency increases linearly with time. This type of signal is also called a chirp. A FMCW radar system transmits a chirp signal and captures the signal reflected by a target in its path. A simplified radar architecture includes a synthesizer, a transmit antenna, a receive antenna and a mixer. The synthesizer generates the chirp signal, which is transmitted by the transmit antenna and captured by the receive antenna as a reflection from an object. The mixer mixes the receive (Rx) and transmit (Tx) signals to produce an intermediate frequency (IF) signal.

[0067] The mixer's role can also be understood as a function of time in terms of the Tx and Rx chirp frequencies. For an object at a distance d from the radar, the mixer's output IF signal will be a sinusoid with a constant frequency. If the radar detects multiple objects, the different received chirps are converted to multiple IF frequency components, each with a constant frequency. This IF signal, composed of multiple frequency components, can be processed using a Fourier transform to separate the different frequency components. The Fourier transform processing will produce a frequency spectrum with different peaks for different frequency components, each peak indicating the presence of an object at a particular distance.

[0068] To measure velocity, a FMCW radar transmits two chirps separated by a time interval Tc. Each reflected signal is processed by a fast Fourier transform (FFT) to detect the target's distance. The distance corresponding to each signal will have a peak at the same location but with a different phase.

[0069] If there are multiple moving objects with different velocities and the two objects are at the same distance from the radar, to measure the velocity, the radar system must send more than two chirps. It sends a set of n equally spaced chirps, which is called a chirp frame.

[0070] A FMCW radar system can also estimate the angle of the reflected signal in the horizontal plane, which is also called the angle of arrival (AOA). The angle estimation is based on observing a small change in the distance to the object resulting in a change in the phase of the distance FFT value or the Doppler FFT peak. This result is used to estimate the angle using at least two RX antennas, the differential distance from the object to each antenna results in a change in the phase of the FFT peak, which enables the AOA to be estimated.

[0071] At step 130, the target object is determined from the objects based on the type of object, the motion trend and the distance and angle to the target object.

[0072] The target object refers to an object that is identified and considered important to the target subject, usually an object that has potential danger or needs attention.

[0073] The target object can be filtered according to the type of object, motion trend, and distance, angle, etc. data relative to the target subject. This filtering process can determine which objects are most important to the target subject and which objects need to be prioritized through an algorithm.

[0074] For example, if an object in front of the target subject is approaching quickly, this object will be determined as the target object.

[0075] Of course, the number of target subjects can not be limited to one, and multiple target objects can be focused on at the same time.

[0076] It can be understood that irrelevant objects can be filtered out to ensure that the target subject can focus on important objects, especially in complex scenes, such as in places with many people and many cars, which can help the user better identify potential dangerous objects or other objects that need attention.

[0077] Step 140, based on the type of target object, motion trend, and distance and angle relative to the target subject, generate a target auditory signal for simulating the binaural hearing of the target subject at the target object's location to perceive the state information of the target object.

[0078] The state information of the target object includes type, distance, angle, and motion trend.

[0079] The target auditory signal represents the state of the target object through specific audio signals or audio patterns. The characteristics of the sound, such as pitch, volume, directionality, etc., can be used to convey the type, distance, angle, and motion trend of the object.

[0080] According to the type, distance, angle, and motion trend of the target object, a set of specific auditory signals can be generated. For example: if the object is close, the volume will increase. If the object is on the left, the sound source will come from the left ear, simulating spatial position perception. If the object is close, the frequency of the sound can be accelerated to represent the speed of the object. If the object moves quickly, the pitch can be higher to simulate the speed of the object. The generated auditory signal simulates the binaural hearing experience and simulates the sound perception of the target subject at different locations.

[0081] It can be understood that through the auditory signal, the target subject can perceive the type, direction, distance, and motion state of the object, and even be able to judge the threat level of the object through the characteristics of the sound, which is more intuitive than traditional voice feedback or image display, especially in complex environments, the target subject can react faster.

[0082] At step 150, the target auditory signal is played to the target object through a hearing aid worn by the target object, so that the target object obtains the state information of the target object.

[0083] The hearing aid is an electronic device worn in the ear of the target object, mainly used for enhancing hearing. It can adjust the volume and frequency as needed to help the user hear the external sound.

[0084] The generated target auditory signal can be transmitted to the hearing aid worn by the target object, and the hearing aid can also adjust the volume, frequency, directivity, etc. according to the characteristics of the signal, to ensure that the target object can clearly hear the state information related to the surrounding objects.

[0085] For different objects, different object information can be delivered through different sound characteristics such as pitch, volume, and duration.

[0086] In this embodiment, the target object can obtain the position information of the objects in the surrounding environment in real time through the auditory signal. Especially in the case where the user cannot directly rely on visual information or the auditory signal of his own ear, the binaural sound effect provided by the hearing aid can help the user identify the direction, distance, and speed of the object, thereby improving its self-protection ability in a complex environment.

[0087] The object perception method based on binaural auditory signal expression provided by the embodiment of the application integrates radar perception, object analysis, target recognition, spatial sound effect simulation, and hearing aid broadcasting technologies, realizes intuitive perception of the target object to the surrounding environment in a vision-limited or hearing-limited scene, and the target object can also understand the dynamics of the surrounding environment in real time without relying on visual information, thereby enhancing the auxiliary ability of the wearable device to the user in a complex scene. The feedback mode of the auditory signal is more suitable for the needs of vision-limited users, hearing-limited users, or in a specific complex scene, and can also effectively improve the use experience of the target object.

[0088] In some embodiments, based on the type, motion trend, distance from the target object, and angle of the target object, the target auditory signal for simulating the binaural perception of the target object by a person at the position of the target object is generated, including: determining the timbre of the target auditory signal based on the type of the target object and the distance from the target object; determining the intensity, binaural intensity difference, and binaural time difference of the target auditory signal based on the distance and angle of the target object from the binaural of the target object; determining the frequency characteristics of the target auditory signal based on the motion trend of the target object; and generating the target auditory signal based on the determined timbre, intensity, binaural intensity difference, binaural time difference, and frequency characteristics.

[0089] It can be understood that the target auditory signal can be generated in a manner simulating the human auditory system, so that the target object can perceive and analyze the state information of the surrounding target objects.

[0090] Different types of objects (such as people, cars, animals, etc.) will produce different sound characteristics. For example, the sound of a car and the sound of a human being are different in timbre. The farther the distance, the more the sound propagation will be attenuated, and at the same time, it will also affect the change of timbre. For example, a sound far away may sound more low-pitched or blurred.

[0091] Therefore, the generation of timbre will be adjusted according to the type of object and the distance from the target object, so as to simulate the sound perception heard in the real world.

[0092] The intensity of the sound is closely related to the distance of the target object. The closer the target object is to the listener, the louder the sound; on the contrary, the farther the sound is smaller.

[0093] The interaural level difference (ILD) refers to the difference in sound intensity received by the human ear through both ears to determine the direction of the sound. The closer the object is to one ear, the greater the difference in sound intensity.

[0094] The interaural time difference (ITD) refers to the time difference in sound arrival received by the human ear through both ears to determine the position of the sound. When the sound comes from one side, the other ear will receive the sound slightly later, so this time difference is also used to determine the direction of the sound source.

[0095] Through these parameters, the human binaural system can be simulated to determine the direction and distance of the sound according to the difference in sound intensity and time.

[0096] The motion state of the target object (such as stationary or moving) will affect the frequency of the sound it emits. If the target object is approaching or moving away from the listener, for example, a car approaching will hear a higher frequency, and moving away will hear a lower frequency, which is called the Doppler effect. The frequency of a moving object will change with the change in distance.

[0097] In this embodiment, the frequency characteristics of the sound can also be adjusted by simulating the motion trend of the target object, so as to realize dynamic auditory perception of the target object.

[0098] The timbre, intensity, binaural intensity difference, binaural time difference, and frequency characteristics are combined to generate a comprehensive auditory signal that simulates the location, type, and movement state of the target object. In other words, the generated target auditory signal is a multi-dimensional sound signal that contains perception information such as direction, distance, object type, and movement trend, ultimately allowing people to hear the existence, orientation, and movement state of the object.

[0099] The intensity of the sound can be set according to the distance. The distance from 1 to 20 meters can be mapped to the intensity range of 80-40 decibels with a resolution of 1 meter, and the resolution is 2 decibels, that is, the distance increases by 1 meter, and the sound intensity of the prompt sound decreases by 2 decibels.

[0100] For binaural intensity difference and binaural time difference, the 180-degree range in front can be divided into 12 positions with a resolution of 15 degrees. There are 6 positions on the left and right, and the ITD time of each position decreases from the maximum 0.62ms at 0 and 180 degrees to 0ms at 90 degrees in front, with a decrease of about 0.1ms per position. The ILD of each position decreases from the maximum 25dB at 0 and 180 degrees to 0dB at 90 degrees in front, with a decrease of about 4dB per position.

[0101] In some embodiments, based on the movement trend of the target object, the frequency characteristics of the target auditory signal are determined, including: in the case that the movement speed of the target object is less than a speed threshold, determining that the movement trend of the target object is to remain stationary; and determining that the frequency characteristics of the target auditory signal are a fixed-frequency pulse tone.

[0102] The movement speed of the target object can be determined first. The threshold is a criterion for distinguishing whether the object is in a "stationary" state. For example, if the speed of the object is very small (such as almost stationary), the object can be considered to be in a stationary state. If the threshold is set to 1 meter / second, the speed below the threshold is a stationary object, and the speed above the threshold is a moving object.

[0103] When the movement speed of the target object is less than the threshold, it can be inferred that the object has no obvious movement trend and is considered to be in a stationary or very slow movement state.

[0104] If it is determined that the target object is in a stationary state, a fixed-frequency pulse tone can be generated. The "pulse tone" here refers to a periodic sound signal with a constant frequency and a certain time interval (pulse) for repeated appearance. For example, a stationary object selects a 1000Hz pulse tone as a prompt sound.

[0105] It can be understood that in this way, the frequency characteristics of the sound can be determined according to the motion state of the target object (whether it is moving or the speed of movement is fast enough). If the target object remains stationary (or moves very slowly), a constant frequency pulse tone can be used to represent it.

[0106] In this way, dynamic objects and stationary objects can be effectively distinguished, and reasonable sound output can be made when simulating auditory perception.

[0107] In some embodiments, the frequency characteristics of the target auditory signal are determined based on the motion trend of the target object, including the following processes.

[0108] The motion trend of the target object can be determined as a motion state when the speed of the target object is greater than or equal to a speed threshold.

[0109] Then, the carrier frequency of the target auditory signal can be determined as an initial frequency, and a frequency modulation signal of the target auditory signal can be determined based on the motion state of the target object to adjust the initial frequency; wherein the size of the frequency modulation signal adjusting the carrier frequency is associated with the speed of the target object, and the size of the adjusted carrier frequency is associated with the distance between the target object and the target object after the target object moves. Finally, the frequency characteristics of the target auditory signal are determined based on the frequency modulation signal and the initial frequency.

[0110] It should be noted that it can be checked whether the speed of the target object exceeds a preset speed threshold. If the speed of the target object is greater than or equal to the threshold, it is considered that the target object is in a motion state.

[0111] An initial frequency can be set for the target auditory signal, which is usually a base frequency used as the starting carrier frequency of the signal. For example, a 1000 Hz narrowband noise is selected as the carrier for a moving object.

[0112] In the frequency modulation signal, the carrier frequency refers to the base frequency of the sound signal, which usually changes according to the frequency modulation signal. In simple terms, this initial frequency is a reference value.

[0113] The frequency of the signal can be adjusted according to the motion state of the target object. This is achieved by a frequency modulation signal that adjusts the carrier frequency according to the speed of the object.

[0114] The adjustment of the frequency modulation signal is closely related to the speed of the target object. The faster the object moves, the greater the impact of the frequency modulation signal on the carrier frequency, that is, the greater the frequency change of the signal.

[0115] The frequency modulation signal is not only related to the speed, but also to the distance between the target object and the target subject. Generally, in practical applications, if the object is far away from the observer, the frequency may need to be adjusted according to the relative position of the target object (e.g., using the Doppler effect). In other words, the size of the carrier frequency is related to the distance between the object after moving and the target subject. Generally, when the object is close to the observer, the frequency increases; when it is far away, the frequency decreases. For example, the speed can be fast or slow in the range of 500Hz to 2000Hz, and the frequency modulation information can form a Doppler effect to assist people in feeling the movement trend of the object approaching or moving away and the speed of the movement.

[0116] Finally, the frequency characteristics of the final target auditory signal can be determined by combining the initial frequency and the adjustment result of the frequency modulation signal. This means that based on the speed of the object, the distance between the object and the target subject, and the initial set frequency, a frequency characteristic that matches these factors will be finally generated.

[0117] It should be noted that the frequency characteristics of the generated auditory signal can be dynamically adjusted according to the motion state of the target object (whether it is moving, how fast it is moving, and the distance between the object and the target). Specifically, first, it is determined whether the object is moving, and a base frequency is set. Then, based on the speed of the object and the distance to the target subject, the base frequency is adjusted using the frequency modulation signal to generate the final auditory signal.

[0118] In this embodiment, this approach simulates the sound changes in reality, such as the Doppler effect (change in frequency when the object is moving), so that the target auditory signal more accurately reflects the dynamic state of the object.

[0119] In some embodiments, based on the type of the target object and the distance to the target subject, the timbre of the target auditory signal is determined, including: based on the type of the target object, determining a base timbre model corresponding to the target object; according to the distance of the target object, calculating the attenuation rate of the sound through an attenuation function, adjusting the high frequency component and the low frequency component in the frequency spectrum in the base timbre model corresponding to the target object, and determining the timbre of the target auditory signal.

[0120] Timbre is the quality or characteristic of a sound that allows the target subject to distinguish different sound sources. Specifically, it is embodied through the spectral characteristics of the sound, such as the proportion of high frequency components and low frequency components.

[0121] Each object will have its own "timbre", which can be modeled as a base timbre model. This model describes the spectral characteristics of the sound emitted by the object.

[0122] For example, if the target object is a car, its base timbre model may contain strong low frequency components; if the target object is a bird, its base timbre model may contain more high frequency components.

[0123] A base timbre model is a preliminary audio feature description for different objects or sound sources. The base timbre model mainly reflects the spectral characteristics of the sound, i.e., the intensity distribution of different frequency components in the sound.

[0124] For example, the base timbre model for a person's voice may exhibit more prominent mid-high frequencies and fewer low frequency components, while the base timbre model for a truck may include stronger low frequency components.

[0125] Sound attenuates during propagation, especially high frequency components attenuate faster than low frequency components. This is because high frequency sound waves have shorter wavelengths and are more easily absorbed and scattered by air.

[0126] An attenuation function is a mathematical tool used to simulate the attenuation effect during sound propagation. This function is usually related to distance, the farther the distance, the more obvious the attenuation of sound.

[0127] For example, when the target object is close, the components of each frequency in the sound will be clear; when the target object is far away, the high frequency components of the sound will be attenuated more, making the sound heard more "low".

[0128] According to the distance of the object, we can calculate the attenuation rate of the sound through the attenuation function. This attenuation rate will affect the base timbre model of the target object, especially the intensity of high and low frequency components.

[0129] As the distance increases, high frequency components attenuate faster, so the high frequency components in the base timbre model need to be reduced. That is, if the target object is far away, the high frequency components can be reduced in the timbre, making the sound sound more low.

[0130] Low frequency components attenuate slower than high frequency components, so as the distance of the object increases, low frequency components are usually relatively retained more. Even if the distance is far away, the low frequency will be clearer.

[0131] It can be understood that through the above attenuation function and distance calculation, the timbre of the target object can be adjusted according to the distance of the target object. The timbre of the target object will change with the distance, especially the proportion of high and low frequency components will be adjusted accordingly.

[0132] For example, when the target object is very close, the timbre may be brighter (high frequency is stronger); when the target object is far away, the timbre may become more low, i.e., the high frequency components are weakened, and the low frequency components are retained more.

[0133] To achieve a more realistic spatial and distance perception effect, the sound level difference, time difference and timbre difference can be combined to conduct comprehensive coding. For example, a dynamic mapping algorithm is designed to convert distance data into corresponding auditory signal parameters. Then, by adjusting the volume, delay and spectral characteristics of the left and right channels, a binaural auditory signal with spatial and depth perception is generated.

[0134] Due to the differences in auditory perception ability and preference of users, the encoding parameters can be adaptively adjusted and optimized in actual application. This can be achieved by collecting feedback data of users and dynamically adjusting the encoding strategy according to the feedback results to improve user experience and accuracy.

[0135] In some embodiments, as shown in Figure 2 The target object is determined from the objects based on the type of the objects, the motion trend, and the distance and angle from the target object, mainly including steps 210, 220, 230, and 240.

[0136] Step 210, determining the objects in the target range based on the distance and angle of the objects from the target object;

[0137] Step 220, in the case that the number of objects in the target range is greater than the target number, judging whether the objects are still in the target range after the target time based on the motion trend of the objects in the target range;

[0138] Step 230, based on the first number of objects still in the target range after the target time and the preset priority, screening a second number of objects from the objects in the target range with a stationary state from the motion trend of the objects in the target range;

[0139] Step 240, determining the first number of objects with a motion state and the second number of objects with a stationary state as target objects.

[0140] First, it is necessary to determine which objects are located within a predetermined target range according to the distance and angle of the objects from the target object.

[0141] Suppose there are multiple objects located within the area of the target range, it is necessary to judge whether these objects are within the target range by calculating the distance and angle between the objects and the target object. For example, a virtual "circular or three-dimensional range" can be set, and when the distance and angle of the objects from the target object meet the range requirements, it will be considered to be within the target range.

[0142] The size and shape of the target range can depend on the specific scenario, for example, different target objects have different sensitivities, so the size of the target range is related to the sensitive area of the target object. Alternatively, the target range is also related to the environment in which the target object is located. If it is in a busy and complex environment such as a street, the target range can be smaller.

[0143] If the number of objects is greater than the number of targets, it means that there are too many objects and too much feedback information for the target object, which is not conducive to the target object obtaining effective information. When there are multiple objects in the target range, it is necessary to further determine whether these objects remain in the target range after a predetermined target duration.

[0144] The target duration refers to a set time window to check whether the object remains in the target range within this time. For example, it may be necessary to determine whether an object remains in the target range in the next few seconds to ensure that it is still an effective target.

[0145] Based on the objects that remain in the target range after the target duration, the next step is to further filter the target objects according to the motion trend of the objects. In the target range, first filter out the objects in motion. The number of these objects is the first number.

[0146] The preset priority refers to the fact that each object can be sorted according to priority, for example, some objects have a higher priority and may have a stronger relevance to the target. The priority can be defined according to different characteristics, such as the type and importance of the target object.

[0147] From the stationary objects in the target range, select a second number of objects. The selection of stationary objects can be based on whether the object has moved little within a certain time.

[0148] Finally, combine the two types of objects selected to determine the final target object. That is, the first number of moving objects and the second number of stationary objects.

[0149] These two types of objects are combined to determine the final target object. In other words, the target object can be selected according to the motion state of the object, whether it is in motion or stationary, and it can become a target.

[0150] In this embodiment, when there are many objects, important target objects can be selected according to whether the objects are moving and the priority level to provide feedback to the target object, which can effectively reduce the disturbance of too much information to the target object.

[0151] In some embodiments, the target auditory signal is broadcast to the target object through a hearing aid worn by the target object, including: in the case where the number of target objects is greater than one, the corresponding target auditory signal is broadcast to the target object through the hearing aid worn by the target object in intervals according to the sound-emitting priority of the corresponding target object; the sound-emitting priority is determined based on the type, motion trend, and distance and angle from the target object of the target object.

[0152] If there are multiple target objects in the environment, such as multiple potential dangerous objects, these target objects may all need to generate auditory signals related to them for the target object (a person wearing a hearing aid) to identify.

[0153] In some cases, there is more than one target object. For example, if there are multiple vehicles and pedestrians on a city street, a person wearing a hearing aid may need to receive a sound signal containing multiple types of information.

[0154] The sound-emitting priority refers to the ordering of the sound signals of each target object according to their priority, with higher priority signals being broadcast first. For example, dangerous objects or objects moving at higher speeds can be given higher priority.

[0155] Since there are multiple target object sound signals, each target object signal can be broadcast in order (in intervals) according to priority, rather than all signals being broadcast simultaneously. This ensures that the wearer can clearly hear each signal and react accordingly based on its importance.

[0156] The nature of the target object can affect its priority. For example, if the target object is a certain threat (such as a fast-moving car), it will have a high priority; if it is a stationary object or a distant object, it will have a lower priority.

[0157] The motion trend of the object (whether it is approaching quickly, slowing down, stopping, etc.) can affect its priority. For example, an object that is approaching quickly can be given a higher priority so that the wearer can be aware of it in time and react.

[0158] The distance and angle between the target object and the target object are also key factors in determining the sound-emitting priority. If the target object is very close to the wearer or is moving in his direction, it will have a higher priority. If the target object is outside the wearer's line of sight, the priority can be lower.

[0159] Suppose a person wearing a hearing aid is walking on a street. If there are multiple objects (such as cars, pedestrians, roadblocks, etc.) approaching him, the car can be given a higher priority based on its speed and distance from the pedestrian, and the pedestrian will be notified of the car's approach first.

[0160] If another pedestrian is far away and stationary, the car information can be announced first, and the pedestrian information can be announced later.

[0161] This intelligent scheduling ensures that the delivery of auditory signals is more efficient and safe, avoids the interference of multiple signals, and helps the wearer to obtain the most important information first.

[0162] In some embodiments, the detected target object can also be classified into different objects according to distance, angle and speed, and the prompt sound is emitted in the order of from far to near, from left to right, and from movement to stillness, and the interval time between each prompt sound is equal.

[0163] After a scan and the prompt are completed, a prompt sound different from the other prompt sounds representing the detected object is emitted to indicate that the period is completed and the device is in working condition, regardless of whether there is a detected object or not, such as a 500Hz beat sound.

[0164] The object perception device based on binaural auditory signal expression provided by the present application is described below, and the object perception device based on binaural auditory signal expression described below can be referred to in correspondence with the object perception method based on binaural auditory signal expression described above.

[0165] As shown in Figure 3 The object perception device based on binaural auditory signal expression of the embodiment of the present application mainly includes an acquisition module 310, an analysis module 320, a first processing module 330, a second processing module 340 and a feedback module 350.

[0166] The acquisition module 310 is used to acquire the position information of the object around the target object through the radar perception device carried by the target object; the position information of the object includes the distance and angle of the object from the target object;

[0167] The analysis module 320 is used to analyze the position information of the object to determine the type and movement trend of the object;

[0168] The first processing module 330 is used to determine the target object from the object based on the type, movement trend, distance and angle of the object from the target object;

[0169] The second processing module 340 is used to generate a target auditory signal for simulating the perception of the state information of the target object by a person at the position of the target object based on the type, movement trend, distance and angle of the target object from the target object; the state information of the target object includes type, distance, angle and movement trend;

[0170] The feedback module 350 is used to broadcast the target auditory signal to the target object through the hearing aid worn by the target object, so that the target object acquires the state information of the target object.

[0171] The object perception device based on binaural hearing signal expression provided by the embodiment of the present application integrates radar perception, object analysis, target recognition, spatial sound simulation, and hearing aid broadcasting, and realizes intuitive perception of the target object to the surrounding environment in a vision-limited or hearing-limited scene, so that the target object can understand the dynamic of the surrounding environment in real time without relying on visual information, and the auxiliary ability of the wearable device to the user in a complex scene is enhanced. The feedback mode of the hearing signal is more suitable for the needs of the vision-limited user, the hearing-limited user, or in a specific complex scene compared with the traditional visual display or voice prompt, and the use experience of the target object can be effectively improved.

[0172] The embodiment of the present application also provides a hearing aid, which comprises a microphone, an audio processing module, a communication module, a power module, and a loudspeaker. The hearing aid adopts the object perception method based on binaural hearing signal expression to broadcast the target hearing signal to the target object, so that the target object can obtain the state information of the target object.

[0173] Figure 4 An example of a schematic diagram of the physical structure of an electronic device is shown in Figure 4 The electronic device can include a processor 410, a communication interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communication interface 420, and the memory 430 can communicate with each other through the communication bus 440. The processor 410 can call the logical instructions in the memory 430 to execute the object perception method based on binaural hearing signal expression, which includes: obtaining the position information of the object around the target object through the radar perception device carried by the target object; the position information of the object includes the distance and angle of the object from the target object; analyzing the position information of the object to determine the type and motion trend of the object; determining the target object from the object based on the type, motion trend, distance from the target object, and angle of the object; generating a target hearing signal for simulating the binaural hearing of a person at the position of the target object to perceive the state information of the target object based on the type, motion trend, distance from the target object, and angle of the target object; the state information of the target object includes the type, distance, angle, and motion trend; broadcasting the target hearing signal to the target object through the hearing aid worn by the target object to make the target object obtain the state information of the target object.

[0174] In addition, the logic instructions in the memory 430 described above can be implemented in the form of software functional units and sold or used as independent products, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions for making a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0175] In another aspect, the present application also provides a computer program product, which comprises a computer program, the computer program can be stored on a non-transitory computer readable storage medium, and the computer program can be executed by a processor to enable a computer to execute the object perception method based on binaural hearing signal expression provided by the above-mentioned method, the method comprising: acquiring position information of objects around a target object by a radar perception device carried by the target object; the position information of the object includes the distance and angle of the object from the target object; analyzing the position information of the object to determine the type and motion trend of the object; determining a target object from the object based on the type, motion trend, distance from the target object and angle of the object; generating a target hearing signal for simulating the perception of the state information of the target object by a person at the position of the target object based on the type, motion trend, distance from the target object and angle of the target object; the state information of the target object includes type, distance, angle and motion trend; and playing the target hearing signal to the target object through a hearing aid worn by the target object to enable the target object to acquire the state information of the target object.

[0176] In yet another aspect, the present application also provides a non-transitory computer readable storage medium having stored thereon a computer program, which, when executed by a processor, implements a method for object perception based on binaural hearing signal representation as provided by any of the above methods, the method comprising: obtaining position information of objects around a target object by a radar perception device carried by the target object; the position information of the objects including distances and angles of the objects from the target object; analyzing the position information of the objects to determine types and motion trends of the objects; determining a target object from the objects based on the types, motion trends, distances and angles of the objects from the target object; generating a target hearing signal for simulating a person at a position of the target object perceiving state information of the target object based on the types, motion trends, distances and angles of the target object from the target object; the state information of the target object including the types, distances, angles and motion trends; and playing the target hearing signal to the target object through a hearing aid worn by the target object to enable the target object to obtain the state information of the target object.

[0177] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0178] From the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be implemented by means of software and necessary universal hardware platforms, and of course can also be implemented by hardware. Based on such understanding, the above technical solutions can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes a number of instructions to make a computer device (which can be a personal computer, server, or network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0179] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for object perception based on binaural hearing signal representation, characterized in that, The method comprises: acquiring position information of objects around the target object through a radar sensing device carried by the target object; the position information of the objects comprises a distance and an angle of the objects from the target object; analyzing the position information of the objects to determine a type and a motion trend of the objects; determining a target object from the objects based on the type, the motion trend, the distance and the angle of the objects from the target object, comprising: determining objects within a target range based on the distance and the angle of the objects from the target object; in a case where a number of the objects within the target range is greater than a target number, judging whether the objects are still within the target range after a target time based on the motion trend of the objects within the target range; based on a first number of the objects within the target range that are still within the target range after the target time and a preset priority, screening a second number of objects from the objects within the target range that are in a stationary state based on the motion trend; determining the target object as the first number of the objects in a motion state and the second number of the objects in the stationary state; the size of the target range is related to a sensitivity of the target object; generating a target auditory signal for simulating binaural perception of state information of the target object by a person at a position of the target object based on the type, the motion trend, the distance and the angle of the target object from the target object; the state information of the target object comprises the type, the distance, the angle and the motion trend; wherein, based on the motion trend of the target object, determining a frequency characteristic of the target auditory signal, comprising: in a case where a motion speed of the target object is less than a speed threshold, determining that the motion trend of the target object is to remain in a stationary state; determining that the frequency characteristic of the target auditory signal is a fixed-frequency pulse tone and the pulse tone repeatedly occurs at a time interval; the motion trend is inferred based on the position information and a motion trajectory of the object; based on the motion trend of the target object, determining the frequency characteristic of the target auditory signal, comprising: in a case where the motion speed of the target object is greater than or equal to the speed threshold, determining that the motion trend of the target object is in a motion state; determining that a carrier frequency of the target auditory signal is an initial frequency, and determining a frequency modulation signal of the target auditory signal based on the motion state of the target object to adjust the initial frequency; wherein, the frequency modulation signal adjusts the size of the carrier frequency in association with the motion speed of the target object, and the size of the adjusted carrier frequency is associated with a distance of the target object from the target object after the motion; determining the frequency characteristic of the target auditory signal based on the frequency modulation signal and the initial frequency; playing the target auditory signal to the target object through a hearing aid worn by the target object, so that the target object acquires the state information of the target object; after a scan and a prompt are completed, issuing a prompt sound different from other prompt sounds representing the measured object to indicate that a scan period is completed and the device is in a working state.

2. The binaural hearing signal representation based object perception method according to claim 1, wherein, The target auditory signal is generated based on the type, motion trend, distance and angle of the target object, and the target auditory signal is used to simulate the perception of the state information of the target object by the binaural ears of a person at the target object position, and the target auditory signal comprises: The timbre of the target auditory signal is determined based on the type and distance of the target object, and the intensity, binaural intensity difference and binaural time difference of the target auditory signal are determined based on the distance and angle of the target object from the binaural ears of the target object; The target auditory signal is generated based on the determined timbre, intensity, binaural intensity difference, binaural time difference and frequency characteristics.

3. The binaural hearing signal representation based object perception method according to claim 2, wherein, The timbre of the target auditory signal is determined based on the type and distance of the target object, and the intensity, binaural intensity difference and binaural time difference of the target auditory signal are determined based on the distance and angle of the target object from the binaural ears of the target object; The target auditory signal is generated based on the determined timbre, intensity, binaural intensity difference, binaural time difference and frequency characteristics. The target auditory signal is generated based on the determined timbre, intensity, binaural intensity difference, binaural time difference and frequency characteristics.

4. The binaural hearing signal representation based object perception method of claim 1, wherein, The target auditory signal is generated based on the determined timbre, intensity, binaural intensity difference, binaural time difference and frequency characteristics.

5. An object perception device based on binaural auditory signal expression, characterized in that: An acquisition module is configured to acquire position information of objects around a target object by a radar perception device carried by the target object, wherein the position information of the objects comprises distance and angle of the objects from the target object; An analysis module is configured to analyze the position information of the objects to determine types and motion trends of the objects; A first processing module is configured to determine target objects from the objects based on the types, motion trends, distance and angle of the objects from the target object; The first processing module is further configured to determine objects in a target range based on the distance and angle of the objects from the target object; In a case where the number of objects in the target range is greater than a target number, whether the objects are still in the target range after a target time period is determined based on the motion trends of the objects in the target range; A second number of objects are selected from the objects in the target range in a stationary state based on a first number of objects in the target range that are still in the target range after the target time period and a preset priority, and the target objects are determined as the first number of objects in a motion state and the second number of objects in the stationary state; the size of the target range is related to the sensitivity of the target object. ​ The second processing module is configured to generate a target auditory signal for simulating binaural perception of state information of the target object by a person at the target object position based on a type of the target object, a motion trend, and a distance and an angle from the target object; the state information of the target object includes the type, the distance, the angle, and the motion trend; the second processing module is further configured to determine a frequency characteristic of the target auditory signal based on the motion trend of the target object; the second processing module is further configured to determine that the motion trend of the target object is a stationary state when a motion speed of the target object is less than a speed threshold, and determine that the frequency characteristic of the target auditory signal is a fixed-frequency pulse tone and the pulse tone repeatedly occurs at a time interval; the motion trend is inferred based on position information and a motion trajectory of the object; the second processing module is further configured to determine that the motion trend of the target object is a motion state when the motion speed of the target object is greater than or equal to the speed threshold, and determine a carrier frequency of the target auditory signal as an initial frequency and determine a frequency modulation signal of the target auditory signal based on a motion state of the target object to adjust the initial frequency; wherein the frequency modulation signal adjusts a size of the carrier frequency in association with the motion speed of the target object, and a size of the adjusted carrier frequency is associated with a distance from the target object after the target object moves; the frequency characteristic of the target auditory signal is determined based on the frequency modulation signal and the initial frequency. The feedback module is configured to broadcast the target auditory signal to the target object through a hearing aid worn by the target object, so that the target object obtains the state information of the target object; after one scan and prompting are completed, a prompt sound different from other prompt sounds representing the measured object is emitted to indicate that a scan period is completed and the device is in a working state.

6. A hearing aid, characterized in that The hearing aid comprises a microphone, an audio processing module, a communication module, a power module, and a loudspeaker, and the hearing aid broadcasts the target auditory signal to the target object by using the object perception method based on binaural auditory signal expression according to any one of claims 1 to 4, so that the target object obtains the state information of the target object. The processor implements the object perception method based on binaural auditory signal expression according to any one of claims 1 to 4 when executing the program.

7. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, ​

Images