Object sensing method and device based on binaural auditory signal expression and hearing aid

Abstract

The invention provides an object sensing method and device based on binaural auditory signal expression and a hearing aid, and belongs to the technical field of hearing aids. According to the object sensing method based on binaural auditory signal expression, radar sensing, object analysis, target identification, spatial sound effect simulation, hearing aid broadcasting and other technologies are integrated; visual perception of the target object to the surrounding environment in a vision-limited or hearing-limited scene is achieved, the target object can know the dynamic state of the surrounding environment in real time without depending on visual information, the auxiliary ability of the wearable device to a user in a complex scene is enhanced, and compared with traditional visual display or voice prompt, the feedback mode of the hearing signal has the advantage that the user experience is improved. The method is more suitable for the requirements of vision-limited users, hearing-limited users or in specific complex scenes, and the use experience of the target object can be effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of hearing aids, and in particular to an object perception method, device and hearing aid based on binaural auditory signal expression. Background Art

[0002] With the continuous advancement of science and technology, the application of wearable devices in daily life and work is becoming more and more popular. Using these devices to help users perceive obstacles in the surrounding environment is of great significance for specific groups or application scenarios. The core technology of obstacle perception is to measure and feedback the position and movement information of surrounding objects. At present, radar ranging technology has been widely used in fields such as autonomous driving, drones and smart wearable devices, especially frequency modulated continuous wave (FMCW) millimeter wave technology. Unlike traditional pulse radar systems, FMCW radar can measure the distance, angle and speed of multiple objects at the same time by continuously transmitting frequency modulated signals. These three parameters are sufficient to reflect the spatial position and dynamic information of objects.

[0003] A complete millimeter wave radar system includes transmitting and receiving RF components, analog circuits, and digital circuits. Traditional radar systems are usually built using discrete components, which not only increases power consumption and system cost, but also leads to huge design challenges due to the complexity and high frequency of the system. In general, existing single-chip radar sensor solutions have achieved high integration, miniaturization, and low power consumption, and can be widely used in personal wearable devices.

[0004] However, traditional radar signals are usually presented in digital or image form. If wearable devices rely on visual displays to convey this information, this approach is not user-friendly for users with limited vision or in certain scenarios. In addition, although voice-based auditory feedback has a certain effect, it is often not intuitive enough and cannot meet the needs of complex scenarios. Summary of the invention

[0005] The present invention provides an object perception method, device and hearing aid based on binaural auditory signal expression, which are used to solve the defect that feedback information based on simple voice in complex scenes in the prior art is not intuitive enough, and to achieve the effect of being more adaptable to the needs of users with limited vision, limited hearing or in specific complex scenes.

[0006] The present invention provides an object perception method based on binaural auditory signal expression, comprising:

[0007] Acquiring position information of objects around the target object through a radar sensing device carried by the target object; the position information of the object includes the distance and angle between the object and the target object;

[0008] Analyze the location information of the object to determine the type and movement trend of the object;

[0009] Determine a target object from the objects based on the type of the object, the movement trend, and the distance and angle from the target object;

[0010] Based on the type, movement trend, distance and angle of the target object, a target auditory signal is generated to simulate the perception of state information of the target object by the two ears of a person at the position of the target object; the state information of the target object includes the type, distance, angle and movement trend;

[0011] The target auditory signal is broadcast to the target object through the hearing aid worn by the target object, so that the target object obtains the status information of the target object.

[0012] According to an object perception method based on binaural auditory signal expression provided by the present invention, based on the type, motion trend, distance and angle of the target object, generating a target auditory signal for simulating the binaural perception of the state information of the target object by a person at the position of the target object, comprising:

[0013] Determine the timbre of the target auditory signal based on the type of the target object and the distance from the target object; determine the intensity, binaural intensity difference and binaural time difference of the target auditory signal based on the distance and angle between the target object and the binaural ears of the target object; determine the frequency characteristics of the target auditory signal based on the movement trend of the target object;

[0014] The target auditory signal is generated based on the determined timbre, intensity, interaural intensity difference, interaural time difference and frequency characteristics.

[0015] According to an object perception method based on binaural auditory signal expression provided by the present invention, determining the frequency characteristics of the target auditory signal based on the motion trend of the target object includes:

[0016] When the moving speed of the target object is less than a speed threshold, determining that the moving trend of the target object is to remain stationary;

[0017] The frequency characteristic of the target auditory signal is determined to be a pulsed pure tone with a fixed frequency.

[0018] According to an object perception method based on binaural auditory signal expression provided by the present invention, determining the frequency characteristics of the target auditory signal based on the motion trend of the target object includes:

[0019] When the moving speed of the target object is greater than or equal to a speed threshold, determining that the moving trend of the target object is a moving state;

[0020] Determine the carrier frequency of the target auditory signal as the initial frequency, and determine the frequency modulation signal of the target auditory signal based on the motion state of the target object to adjust the initial frequency; wherein the magnitude of the carrier frequency adjusted by the frequency modulation signal is associated with the motion speed of the target object, and the magnitude of the adjusted carrier frequency is associated with the distance between the target object and the target object after the motion;

[0021] A frequency characteristic of the target auditory signal is determined based on the frequency modulation signal and the initial frequency.

[0022] According to an object perception method based on binaural auditory signal expression provided by the present invention, the timbre of the target auditory signal is determined based on the type of the target object and the distance from the target object, including:

[0023] Based on the type of the target object, determining a basic timbre model corresponding to the target object;

[0024] According to the distance of the target object, the attenuation rate of the sound is calculated by using an attenuation function, and the high-frequency components and low-frequency components in the spectrum of the basic timbre model corresponding to the target object are adjusted to determine the timbre of the target auditory signal.

[0025] According to an object perception method based on binaural auditory signal expression provided by the present invention, the target object is determined from the objects based on the type of the object, the movement trend, and the distance and angle from the target object, including:

[0026] Determining objects within a target range based on the distance and angle of the objects from the target object;

[0027] When the number of objects within the target range is greater than the target number, judging whether the objects are still within the target range after the target time based on the movement trend of the objects within the target range;

[0028] Based on the first number of objects still within the target range after the target time and the preset priority, a second number of objects are screened out from objects within the target range whose motion trend is a stationary state;

[0029] The first number of objects whose movement trends are in a moving state and the second number of objects in a stationary state are determined as the target objects.

[0030] According to an object perception method based on binaural auditory signal expression provided by the present invention, the target auditory signal is broadcast to the target object through a hearing aid worn by the target object, comprising:

[0031] When the number of target objects is greater than one, the corresponding target auditory signals are broadcast to the target objects at intervals through the hearing aids worn by the target objects according to the sound emission priorities corresponding to the target objects; the sound emission priorities are determined based on the type of the target objects, the movement trend, and the distance and angle from the target objects.

[0032] The present invention also provides an object perception device based on binaural auditory signal expression, comprising:

[0033] An acquisition module is used to acquire the position information of objects around the target object through a radar sensing device carried by the target object; the position information of the object includes the distance and angle between the object and the target object;

[0034] An analysis module, used to analyze the location information of the object and determine the type and movement trend of the object;

[0035] A first processing module, configured to determine a target object from the objects based on the type of the object, the movement trend, and the distance and angle from the target object;

[0036] A second processing module is used to generate a target auditory signal for simulating the perception of state information of the target object by two ears of a person at the position of the target object based on the type, motion trend, distance and angle of the target object from the target object; the state information of the target object includes type, distance, angle and motion trend;

[0037] A feedback module is used to broadcast the target auditory signal to the target object through the hearing aid worn by the target object, so that the target object obtains the status information of the target object.

[0038] The present invention also provides a hearing aid, comprising: a microphone, an audio processing module, a communication module, a power module and a speaker, wherein the hearing aid uses the object perception method based on binaural auditory signal expression as described in any of the above items to broadcast the target auditory signal to the target object, so that the target object obtains the status information of the target object.

[0039] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the object perception method based on binaural auditory signal expression as described in any one of the above is implemented.

[0040] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the object perception method based on binaural auditory signal expression as described in any one of the above is implemented.

[0041] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned object perception methods based on binaural auditory signal expression.

[0042] The object perception method, device and hearing aid based on binaural auditory signal expression provided by the present invention integrate radar perception, object analysis, target recognition, spatial sound simulation and hearing aid broadcasting technologies to achieve the target object's intuitive perception of the surrounding environment in scenarios with limited vision or limited hearing. The target object does not need to rely on visual information and can also understand the dynamics of the surrounding environment in real time, thereby enhancing the auxiliary ability of wearable devices to users in complex scenarios. Compared with traditional visual display or voice prompts, the feedback method of auditory signals is more adapted to the needs of users with limited vision, limited hearing or in specific complex scenarios, and can also effectively improve the user experience of the target object. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0044] Figure 1 This is one of the flow charts of the object perception method based on binaural auditory signal expression provided by the present invention;

[0045] Figure 2 This is the second flow chart of the object perception method based on binaural auditory signal expression provided by the present invention;

[0046] Figure 3 It is a structural schematic diagram of an object perception device based on binaural auditory signal expression provided by the present invention;

[0047] Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] Combine the following Figure 1-Figure 4 The object perception method, device and hearing aid based on binaural auditory signal expression of the present invention are described.

[0050] like Figure 1 As shown, the object perception method based on binaural auditory signal expression according to the embodiment of the present invention mainly includes step 110, step 120, step 130, step 140 and step 150.

[0051] Step 110, obtaining position information of objects around the target object through a radar sensing device carried by the target object.

[0052] The position information of an object includes the distance and angle between the object and the target object.

[0053] The target audience refers to the user who carries the system, which may be individuals with limited vision or hearing loss or other needs.

[0054] Radar sensing device is an intelligent device that contains a sensor, usually an ultrasonic radar, millimeter wave radar or laser radar, which is used to detect objects in the surrounding environment and obtain their location information. By transmitting signals and receiving returning reflected waves, radar equipment can estimate the distance, angle and relative motion information of objects.

[0055] The radar sensing device can be a separate wearable device, can be integrated into other user terminals, or can be integrated into the user's hearing aid, and can communicate with the hearing aid. There is no restriction on the setting form of the radar sensing device.

[0056] In this embodiment, the radar sensing device may include a 77GHz RFCMOS millimeter wave radar sensor. In recent years, with the widespread application of radar technology in the fields of automobiles, medical treatment, etc., 77GHz RFCMOS millimeter wave sensor products have become very mature. Such sensors integrate all RF and analog functions, such as VCO, PLL, PA, LNA, mixers and ADCs for multiple TX / RX channels, and also integrate digital components such as MCU, accelerator or digital signal processor. Unlike traditional SiGe-based solutions, the RF front end and digital back end of the 77GHz millimeter wave radar sensor are more flexible and programmable, and can be customized according to needs to maximize the extraction of useful data in the scene. Through the instant adjustment of a single frequency modulation signal and the processing back end, the requirements of higher measurement range, speed, resolution and specific algorithms can be met. Compared with 24GHz radar, 7GHz sensors have higher resolution in distance and speed, and are smaller in size, with antenna size about one-third of the original.

[0057] The radar device worn by the target object will scan the surrounding environment in real time and detect all objects within the detection range. The information of these objects, including distance, angle and possible motion status, will be transmitted to the processing unit through the radar device.

[0058] It can be understood that through radar equipment, users can perceive the location of surrounding objects without visual input, increasing their awareness of the environment.

[0059] Step 120: Analyze the location information of the object to determine the type and movement trend of the object.

[0060] Based on the characteristics of the radar echo, the nature of the object can be analyzed, such as a stationary object, a pedestrian, a vehicle, etc.

[0061] Based on the change in the object's position relative to the target object, analyze whether the object is stationary, approaching, moving away, or moving laterally.

[0062] Data processing can be performed based on the position information (including distance and angle) of the object sensed by the radar. Combined with the object's motion trajectory, the type of object (for example, a person, a car, an obstacle) and its movement trend can be inferred. For example, whether the object is approaching the target object, moving away from the target object, or remaining stable.

[0063] In complex environments, the motion information of multiple objects can be comprehensively judged to identify the most important target objects.

[0064] It can be understood that identifying the objects that are most important to the target object and their dynamic states can help generate accurate feedback signals later.

[0065] The following uses frequency modulated continuous wave millimeter wave radar as an example to illustrate the working principle of how to measure distance, angle and speed.

[0066] In the signal of an FMCW radar, the frequency increases linearly with time. This type of signal is also called a chirp. An FMCW radar system transmits a chirp signal and captures the signal reflected by the target in its path. A simplified radar structure consists of a synthesizer, a transmitting antenna, a receiving antenna, and a mixer. The synthesizer generates a chirp signal, which is emitted by the transmitting antenna and captured by the receiving antenna. The mixer mixes the receive Rx and transmit Tx signals to generate an intermediate frequency (IF) signal.

[0067] The role of the mixer can also be understood in terms of the chirp frequencies of Tx and Rx expressed as a function of time. For an object at a distance d from the radar, the IF signal at the mixer output will be a sine wave with a constant frequency. If the radar detects multiple objects, the different receive chirps are converted into multiple IF frequency components, each with a constant frequency. This IF signal consists of multiple frequency components and can be processed using a Fourier transform to separate the different frequency components. The Fourier transform process will produce a spectrum with different peaks for different frequency components, each peak indicating the presence of an object at a specific distance.

[0068] To measure speed, an FMCW radar transmits two chirped signals separated by a time interval Tc. Each reflected signal is processed by a Fast Fourier Transform (FFT) to detect the distance to the target. The distance corresponding to each signal will have a peak at the same position, but with a different phase.

[0069] If there are multiple moving objects with different speeds when measuring, and the distance between two objects and the radar is the same, in order to measure the speed, the radar system must send more than two chirps. It sends a group of n equally spaced chirp signals, which is called a chirp frame.

[0070] FMCW radar systems can also estimate the angle of the reflected signal on the horizontal plane, which is also called the angle of arrival (AOA). Angle estimation is based on the observation that small changes in the distance to an object result in a phase change in the range FFT value or Doppler FFT peak. This result is used to perform angle estimation using at least two RX antennas. The differential distance from the object to each antenna results in a phase change in the FFT peak. The phase change can estimate the AOA.

[0071] Step 130 , determining a target object from the objects based on the type of the object, the movement trend, and the distance and angle from the target object.

[0072] A target object refers to an object that is identified and considered to be of great significance to the target object, usually an object that is potentially dangerous or requires attention.

[0073] The target objects can be screened out based on the type of object, movement trend, distance and angle relative to the target object, etc. This screening process can determine which objects are most important to the target object and which objects need to be processed first through algorithms.

[0074] For example, if an object directly in front of the target object approaches quickly, the object will be determined as the target object.

[0075] Of course, the number of target objects is not limited to only one, and multiple target objects can be focused on at the same time.

[0076] It is understandable that irrelevant objects can be filtered out to ensure that the target subject can focus on important objects, especially in complex scenes, such as traveling in crowded places with many people and vehicles, which can help users better identify potential dangerous objects or other targets that need attention.

[0077] Step 140 , based on the type of the target object, the movement trend, and the distance and angle from the target object, a target auditory signal is generated to simulate the perception of the state information of the target object by the two ears of a person at the position of the target object.

[0078] The status information of the target object includes type, distance, angle and motion trend.

[0079] The target auditory signal indicates the state of the target object through a specific audio signal or audio pattern. The characteristics of the sound, such as pitch, volume, directionality, etc., can be used to convey the type, distance, angle and movement trend of the object.

[0080] Depending on the type, distance, angle, and movement of the target object, a specific set of auditory signals can be generated. For example: if the object is closer, the volume will increase. If the object is on the left, the source of the sound will come from the left ear, simulating a sense of spatial location. If the object is close, the frequency of the sound can be accelerated to indicate the speed of the object's approach. If the object is moving faster, the pitch may be higher to simulate the speed of the object's movement. The generated auditory signals mimic the binaural hearing experience and simulate the perception of sound at different locations of the target object.

[0081] It is understandable that through auditory signals, the target object can perceive the type, direction, distance and motion state of the object, and can even judge the threat level of the object by the characteristics of the sound. This is more intuitive than traditional voice feedback or image display, especially in complex environments, and the target object can react faster.

[0082] Step 150: broadcast the target auditory signal to the target object through the hearing aid worn by the target object, so that the target object obtains the status information of the target object.

[0083] Hearing aids are electronic devices worn in the ears of target subjects, mainly used to enhance hearing. It can adjust the volume and frequency as needed to help users hear external sounds.

[0084] The generated target auditory signal can be transmitted to the hearing aid worn by the target object. The hearing aid can also adjust the volume, frequency, directionality, etc. according to the characteristics of the signal to ensure that the target object can clearly hear the status information related to the surrounding objects.

[0085] For different objects, different object information can be conveyed through different sound characteristics such as pitch, volume, duration, etc.

[0086] In this embodiment, the target object can obtain the position information of objects in the surrounding environment in real time through auditory signals. Especially when the user cannot directly rely on visual information or auditory signals from his own ears, the binaural sound effects provided by the hearing aid can help the user identify the direction, distance and speed of the object, thereby improving his self-protection ability in complex environments.

[0087] According to the object perception method based on binaural auditory signal expression provided by the embodiment of the present invention, by integrating radar perception, object analysis, target recognition, spatial sound simulation, and hearing aid broadcasting technologies, the target object can realize the intuitive perception of the surrounding environment in scenarios with limited vision or limited hearing. The target object does not need to rely on visual information, but can also understand the dynamics of the surrounding environment in real time, thereby enhancing the auxiliary ability of wearable devices to users in complex scenarios. Compared with traditional visual display or voice prompts, the feedback method of auditory signals is more adapted to the needs of users with limited vision, limited hearing or in specific complex scenarios, and can also effectively improve the user experience of the target object.

[0088] In some embodiments, based on the type of the target object, the movement trend, and the distance and angle from the target object, a target auditory signal is generated to simulate the perception of the state information of the target object by the ears of a person at the position of the target object, including: determining the timbre of the target auditory signal based on the type of the target object and the distance from the target object; determining the intensity, binaural intensity difference, and binaural time difference of the target auditory signal based on the distance and angle between the target object and the ears of the target object; determining the frequency characteristics of the target auditory signal based on the movement trend of the target object; and generating the target auditory signal based on the determined timbre, intensity, binaural intensity difference, binaural time difference, and frequency characteristics.

[0089] It is understandable that the target auditory signal can be generated by simulating the human auditory system, so that the target object can perceive and analyze the status information of the surrounding target objects.

[0090] Different types of objects (such as people, cars, animals, etc.) produce different sound characteristics. For example, the sound of a car and the sound of a human being are different in timbre. The farther away, the attenuation of the sound will occur, which will also affect the change of timbre. For example, a sound that is far away may sound deeper or fuzzy.

[0091] Therefore, the timbre generation is adjusted according to the object type and the distance to the target object, thus simulating the perception of sound heard in the real world.

[0092] The intensity of a sound is closely related to the distance of the target object. The closer the target object is to the listener, the louder the sound; conversely, the farther away the object is, the quieter the sound.

[0093] Interaural Level Difference (ILD) refers to the ability of the human ear to determine the direction of a sound by the difference in sound intensity received by the two ears. The closer an object is to one ear, the greater the difference in sound intensity.

[0094] Interaural Time Difference (ITD) refers to the fact that the human ear can also determine the location of a sound by the time difference in the arrival of the sound received by the two ears. When the sound comes from one side, the other ear will receive the sound slightly later, so this time difference is also used to determine the direction of the sound source.

[0095] Through these parameters, it is possible to simulate how human ears judge the direction and distance of sound based on the intensity difference and time difference of the sound.

[0096] The motion of an object (i.e., stationary or moving) affects the frequency of the sound it emits. If an object is moving closer or farther away from the listener, such as a car approaching, you will hear a higher frequency, and a car moving away, you will hear a lower frequency. This change is called the Doppler effect. The frequency of a moving object will change as the distance changes.

[0097] In this embodiment, the movement trend of the target object can be simulated to adjust the frequency characteristics of the sound, thereby achieving dynamic auditory perception of the target object.

[0098] Factors such as timbre, intensity, binaural intensity difference, binaural time difference and frequency characteristics will be combined to generate a comprehensive auditory signal, which can simulate the location, type, motion state and other information of the target object. In other words, the generated target auditory signal will be a multi-dimensional sound signal, containing perceptual information such as direction, distance, object type and motion trend, so that people can finally hear the existence, position and motion status of the object.

[0099] The intensity of the sound can be set according to the distance. The distance from 1 to 20 meters can be mapped to the intensity range of 80-40 decibels with a resolution of 1 meter and a resolution of 2 decibels, that is, when the distance increases by 1 meter, the intensity of the prompt sound decreases by 2 decibels.

[0100] For interaural intensity difference and interaural time difference, the 180-degree range in front can be divided into 12 directions with a resolution of 15 degrees. For the 6 directions on the left and right, the ITD time of each direction decreases from the maximum of 0.62ms at 0 degrees and 180 degrees to 0ms at 90 degrees in front, and decreases by about 0.1ms at each direction. The ILD of each direction decreases from the maximum of 25dB at 0 degrees and 180 degrees to 0dB at 90 degrees in front, and decreases by about 4dB at each direction.

[0101] In some embodiments, based on the movement trend of the target object, the frequency characteristics of the target auditory signal are determined, including: when the movement speed of the target object is less than a speed threshold, determining the movement trend of the target object to remain stationary; and determining the frequency characteristics of the target auditory signal to be a pulsed pure tone of a fixed frequency.

[0102] You can first determine whether the target object's speed is lower than a preset speed threshold. This threshold is a criterion for distinguishing whether an object is in a "stationary" state. For example, if the object's speed is very low (such as almost motionless), the object can be considered to be in a stationary state. If the threshold is set at 1 meter per second, objects with speeds lower than this threshold are considered stationary objects, and objects with speeds higher than this threshold are considered moving objects.

[0103] When the target object's moving speed is less than this threshold, it can be inferred that the object has no obvious movement trend and is considered to be stationary or moving very slowly.

[0104] If the target object is determined to be stationary, a fixed-frequency pulse pure tone can be generated. Here, "pulse pure tone" refers to a periodic sound signal, the frequency of the audio remains constant and repeats at a certain time interval (pulse). For example, a stationary object selects a 1000Hz pulse pure tone as a prompt tone.

[0105] It is understandable that in this way, the frequency characteristics of the sound can be determined according to the motion state of the target object (whether it is moving or whether the speed of movement is fast enough). If the target object remains stationary (or moves very slowly), it can be represented by a pulse pure tone with a constant frequency.

[0106] Through this method, dynamic objects and stationary objects can be effectively distinguished, and reasonable sound output can be made when simulating auditory perception.

[0107] In some embodiments, based on the movement trend of the target object, the frequency characteristics of the target auditory signal are determined, including the following process.

[0108] When the moving speed of the target object is greater than or equal to a speed threshold, the moving trend of the target object may be determined to be a moving state.

[0109] Then, the carrier frequency of the target auditory signal can be determined as the initial frequency, and the frequency modulation signal of the target auditory signal can be determined based on the motion state of the target object to adjust the initial frequency; wherein the magnitude of the carrier frequency adjusted by the frequency modulation signal is associated with the motion speed of the target object, and the magnitude of the adjusted carrier frequency is associated with the distance between the target object and the target object after the motion. Finally, the frequency characteristics of the target auditory signal are determined based on the frequency modulation signal and the initial frequency.

[0110] It should be noted that it is possible to check whether the moving speed of the target object exceeds a preset speed threshold. If the speed of the target object is greater than or equal to the threshold, it is considered that the target object is in a moving state.

[0111] An initial frequency can be set for the target auditory signal, which is usually a fundamental frequency and used as the starting carrier frequency of the signal. For example, a moving object selects 1000Hz narrowband noise as the carrier.

[0112] In an FM signal, the carrier frequency refers to the fundamental frequency of the sound signal, which usually changes according to the FM signal. In simple terms, this initial frequency is a reference value.

[0113] The frequency of this signal can be adjusted based on the motion of the target object. This is achieved by using a frequency modulated signal, which adjusts the carrier frequency based on the speed of the object.

[0114] The adjustment of the FM signal is closely related to the speed of the target object. The faster the object is, the greater the impact of the FM signal on the carrier frequency may be, that is, the greater the frequency change amplitude of the signal may be.

[0115] The frequency modulation signal is not only related to the speed, but also to the distance between the target object and the target object. Usually, in practical applications, if the object is far away from the observer, it may be necessary to adjust the frequency according to the relative position of the target object (such as using the Doppler effect). In other words, the magnitude of the carrier frequency is related to the distance from the target object after the object moves. Usually, when the object approaches the observer, the frequency increases; when it moves away, the frequency decreases. For example, the Doppler effect can be formed by adding frequency modulation information to the speed in the range of 500Hz to 2000Hz to help people feel the movement trend of the object approaching or moving away and the speed of the movement.

[0116] Finally, the frequency characteristics of the final target auditory signal can be determined by combining the initial frequency and the adjustment result of the FM signal. This means that based on the speed of the object, the distance between the object and the target object, and the initially set frequency, a frequency characteristic that matches these factors will eventually be generated.

[0117] It should be noted that the frequency characteristics of the generated auditory signal can be dynamically adjusted according to the motion state of the target object (whether it is moving, how fast it is, and the distance between the object and the target). Specifically, first confirm whether the object is moving and set a base frequency. Then, according to the speed of the object and the distance from the target object, the base frequency is adjusted using the frequency modulation signal to generate the final auditory signal.

[0118] In this embodiment, this approach simulates the sound changes in reality, such as the Doppler effect (the change in frequency when an object moves), so that the target auditory signal more accurately reflects the dynamic state of the object.

[0119] In some embodiments, the timbre of the target auditory signal is determined based on the type of the target object and the distance from the target object, including: determining a basic timbre model corresponding to the target object based on the type of the target object; calculating the attenuation rate of the sound through an attenuation function according to the distance of the target object, adjusting the high-frequency and low-frequency components in the spectrum of the basic timbre model corresponding to the target object, and determining the timbre of the target auditory signal.

[0120] Timbre is the quality or characteristics of sound that enables the target object to distinguish different sound sources. Specifically, it is reflected by the spectral characteristics of the sound, such as the ratio of high-frequency components to low-frequency components.

[0121] Each object has a specific "timbre", which can be modeled as a basic timbre model. This model describes the spectral characteristics of the sound emitted by the object.

[0122] For example, if the target object is a car, its basic timbre model may contain stronger low-frequency components; if the target object is a bird, its basic timbre model may contain more high-frequency components.

[0123] The basic timbre model is a preliminary description of audio characteristics for different objects or sound sources. The basic timbre model mainly reflects the spectral characteristics of the sound, that is, the intensity distribution of different frequency components in the sound.

[0124] For example, the basic timbre model for a person's voice may show that the mid-high frequencies are more prominent and the low-frequency components are less, while the basic timbre model for a truck may include stronger low-frequency components.

[0125] Sound will attenuate during propagation, especially high-frequency components will attenuate faster than low-frequency components. This is because high-frequency sound has a shorter wavelength and is easily absorbed and scattered by the air.

[0126] The attenuation function is a mathematical tool used to simulate the attenuation effect of sound propagation. This function is usually related to distance. The farther the distance, the more obvious the sound attenuation.

[0127] For example, when the target object is closer, the components of each frequency in the sound will be clearer; when the target object is farther away, the high-frequency components of the sound will be attenuated more, making the sound heard more "deeper".

[0128] According to the distance of the object, we can calculate the attenuation rate of the sound through the attenuation function. This attenuation rate will affect the basic timbre model corresponding to the target object, especially the intensity of high-frequency and low-frequency components.

[0129] As the distance increases, the high-frequency components decay faster, so it is necessary to reduce the high-frequency components in the basic timbre model. That is, if the target object is far away, the high-frequency components can be reduced in the timbre to make the sound sound deeper.

[0130] Low-frequency components decay more slowly than high-frequency components, so as the distance of an object increases, low-frequency components are usually relatively more preserved. Even at a greater distance, low frequencies will still be clearer.

[0131] It is understandable that, through the above attenuation function and distance calculation, the timbre of the target object can be adjusted according to the distance. The timbre of the target object will change with the distance, especially the ratio of high frequency and low frequency components will be adjusted accordingly.

[0132] For example, when the target object is very close, the timbre may be brighter (high frequencies are stronger); and when the target object is far away, the timbre may become deeper, that is, the high frequency components are weakened and more low frequency components are retained.

[0133] In order to obtain more realistic spatial perception and distance perception effects, auditory cues such as level difference, time difference and timbre difference can be combined for comprehensive encoding. For example, a dynamic mapping algorithm is designed to convert distance data into corresponding auditory signal parameters. Then, by adjusting parameters such as volume, delay and spectral characteristics of the left and right channels, binaural auditory signals with a sense of space and depth are generated.

[0134] Since users have different auditory perception abilities and preferences, the encoding parameters can be adaptively adjusted and optimized in practical applications. This can be achieved by collecting user feedback data and dynamically adjusting the encoding strategy based on the feedback results to improve user experience and accuracy.

[0135] In some embodiments, Figure 2 As shown, based on the type of the object, the movement trend, and the distance and angle from the target object, the target object is determined from the objects, which mainly includes step 210, step 220, step 230 and step 240.

[0136] Step 210, determining objects within the target range based on the distance and angle between the object and the target object;

[0137] Step 220, when the number of objects within the target range is greater than the target number, judging whether the objects are still within the target range after the target time period based on the movement trend of the objects within the target range;

[0138] Step 230, based on the first number of objects still within the target range after the target time and the preset priority, a second number of objects are screened out from the objects within the target range whose motion trend is a stationary state;

[0139] Step 240: Determine a first number of objects whose motion trends are in a moving state and a second number of objects in a stationary state as target objects.

[0140] First, it is necessary to determine which objects are within a predetermined target range based on the distance and angle between the objects and the target object.

[0141] Assuming that there are multiple objects within the target range, it is necessary to determine whether these objects are within the target range by calculating the distance and angle between the object and the target object. For example, a virtual "circular or three-dimensional range" can be set, and when the distance and angle between the object and the target object meet the range requirements, it will be considered to be within the target range.

[0142] The size and shape of the target range may depend on the specific scenario. For example, different target objects have different sensitivities, so the size of the target range is related to the sensitive area of ​​the target object. Alternatively, the target range may also be related to the environment in which the target object is located. If it is in a busy and complex environment such as a street, the target range may be smaller.

[0143] If the number of objects is greater than the number of targets, it means that there are too many objects and the feedback information to the target object is too much, which is not conducive to the target object obtaining effective information. When there are multiple objects in the target range, it is necessary to further determine whether these objects remain in the target range after a predetermined target time.

[0144] Target duration refers to a set time window during which an object is checked to see if it is still within the target range. For example, you may need to determine if an object remains within the target range for the next few seconds to ensure that it is still a valid target.

[0145] Based on the objects that are still within the target range after the target duration, the next step is to further filter the target objects according to the movement trend of the objects. Within the target range, the objects in motion are first filtered out. The number of these objects is the first number.

[0146] The preset priority means that each object may be sorted according to the priority, for example, some objects have a higher priority and may be more relevant to the target. The priority can be defined based on different characteristics, such as the type of target object, importance, etc.

[0147] A second number of objects are selected from the stationary objects within the target range. The stationary objects may be selected based on whether the objects have hardly moved within a specific time.

[0148] Finally, the two types of objects screened out are combined to determine the final target object, namely the first number of moving objects and the second number of stationary objects.

[0149] These two types of objects are combined together and finally determined as the target object. In other words, the target object can be selected according to the motion state of the object. Whether it is in motion or static state, it may become the target.

[0150] In this embodiment, when there are many objects, more important target objects can be screened out according to whether the objects are moving and their priority levels to provide feedback to the target objects, which can effectively reduce the disturbance of excessive information to the target objects.

[0151] In some embodiments, the target auditory signal is broadcast to the target object through the hearing aid worn by the target object, including: when the number of target objects is greater than one, the corresponding target auditory signal is broadcast to the target object through the hearing aid worn by the target object at intervals according to the sound emission priority corresponding to each target object; the sound emission priority is determined based on the type of target object, movement trend, and distance and angle from the target object.

[0152] If there are multiple target objects in the environment, such as multiple potentially dangerous objects, these target objects may need to generate auditory signals associated with them for the target object (the person wearing the hearing aid) to identify.

[0153] In some cases, there is more than one target object. For example, if there are multiple vehicles and pedestrians on a city street, a person wearing a hearing aid may need to receive a sound signal containing multiple information.

[0154] The sound priority means that the sound signals of each target object are sorted according to their priority, and the signals with higher priority will be broadcast first. For example, dangerous objects or faster objects will be given higher priority.

[0155] Since there are multiple target objects' sound signals, the signal of each target object can be broadcasted in sequence (intervals) according to priority, rather than broadcasting all signals at the same time. This ensures that the wearer can hear each signal clearly and respond according to its importance.

[0156] The nature of the target object may affect its priority. For example, if the target object is a threat (such as a fast-moving car), its priority will be high, while if it is a stationary object or a distant object, its priority will be lower.

[0157] The movement trend of an object (whether it is approaching quickly, slowing down, stopping, etc.) will affect its priority. For example, a fast-approaching object may be given a higher priority so that the wearer can detect and react in time.

[0158] The distance and angle between the target object and the target object are also key factors in determining the priority of the sound. If the target object is close to the wearer or is moving in its direction, it will have a higher priority. If the target object is out of the wearer's line of sight, the priority can be lower.

[0159] Suppose a pedestrian wearing a hearing aid is walking on the street. If multiple objects (such as cars, pedestrians, roadblocks, etc.) are approaching him, the car can be given a higher priority based on its speed and distance from the pedestrian, and the pedestrian can be notified of the car's approach first.

[0160] If the other pedestrian is farther away and stationary, the car's information may be announced first, and the pedestrian's information may be announced later.

[0161] This intelligent scheduling ensures that the transmission of auditory signals is more efficient and safer, avoids interference from multiple signals, and helps the wearer get the most important information first.

[0162] In some embodiments, the detected target objects can also be classified into different objects according to distance, angle and speed, and prompt sounds are issued in sequence from far to near, from left to right, and moving first and then stationary, with the interval between each prompt sound being equal.

[0163] After a scan and prompt are completed, regardless of whether there is an object being measured or not, a prompt sound different from other sounds that represent the object being measured should be emitted to indicate that the cycle is completed and the equipment is in working condition, such as a 500Hz beat sound.

[0164] The object perception device based on binaural auditory signal expression provided by the present invention is described below. The object perception device based on binaural auditory signal expression described below and the object perception method based on binaural auditory signal expression described above can refer to each other.

[0165] like Figure 3 As shown, the object perception device based on binaural auditory signal expression of the embodiment of the present invention mainly includes an acquisition module 310, an analysis module 320, a first processing module 330, a second processing module 340 and a feedback module 350.

[0166] The acquisition module 310 is used to acquire the position information of objects around the target object through the radar sensing device carried by the target object; the position information of the object includes the distance and angle between the object and the target object;

[0167] The analysis module 320 is used to analyze the location information of the object and determine the type and movement trend of the object;

[0168] The first processing module 330 is used to determine the target object from the objects based on the type of the object, the movement trend, and the distance and angle from the target object;

[0169] The second processing module 340 is used to generate a target auditory signal for simulating the perception of the state information of the target object by the two ears of a person at the position of the target object based on the type, movement trend, distance and angle of the target object; the state information of the target object includes the type, distance, angle and movement trend;

[0170] The feedback module 350 is used to broadcast the target auditory signal to the target object through the hearing aid worn by the target object, so that the target object obtains the state information of the target object.

[0171] The object perception device based on binaural auditory signal expression provided by the embodiment of the present invention realizes the intuitive perception of the surrounding environment by the target object in scenes with limited vision or limited hearing by integrating technologies such as radar perception, object analysis, target recognition, spatial sound simulation, and hearing aid broadcast. The target object does not need to rely on visual information, but can also understand the dynamics of the surrounding environment in real time, thereby enhancing the auxiliary ability of wearable devices to users in complex scenes. Compared with traditional visual display or voice prompts, the feedback method of auditory signals is more adapted to the needs of users with limited vision, limited hearing, or in specific complex scenes, and can also effectively improve the user experience of the target object.

[0172] An embodiment of the present invention also provides a hearing aid, comprising: a microphone, an audio processing module, a communication module, a power module and a speaker. The hearing aid uses the above-mentioned object perception method based on binaural auditory signal expression to broadcast the target auditory signal to the target object, so that the target object obtains the status information of the target object.

[0173] Figure 4 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 4 As shown, the electronic device may include: a processor (processor) 410, a communication interface (CommunicationsInterface) 420, a memory (memory) 430 and a communication bus 440, wherein the processor 410, the communication interface 420, and the memory 430 communicate with each other through the communication bus 440. The processor 410 can call the logic instructions in the memory 430 to execute an object perception method based on binaural auditory signal expression, the method comprising: obtaining the position information of objects around the target object through a radar sensing device carried by the target object; the position information of the object includes the distance and angle of the object from the target object; analyzing the position information of the object to determine the type and movement trend of the object; determining the target object from the objects based on the type, movement trend, and distance and angle from the target object; generating a target auditory signal for simulating the perception of the state information of the target object by the binaural of a person at the position of the target object based on the type, movement trend, and distance and angle from the target object; the state information of the target object includes type, distance, angle, and movement trend; broadcasting the target auditory signal to the target object through a hearing aid worn by the target object, so that the target object obtains the state information of the target object.

[0174] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0175] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the object perception method based on binaural auditory signal expression provided by the above methods, the method including: obtaining the position information of objects around the target object through a radar sensing device carried by the target object; the position information of the object includes the distance and angle of the object from the target object; analyzing the position information of the object to determine the type and movement trend of the object; determining the target object from the objects based on the type, movement trend, and distance and angle of the object from the target object; generating a target auditory signal for simulating the perception of the state information of the target object by the binaural ears of a person at the target object position based on the type, movement trend, and distance and angle of the target object from the target object; the state information of the target object includes type, distance, angle, and movement trend; broadcasting the target auditory signal to the target object through a hearing aid worn by the target object, so that the target object obtains the state information of the target object.

[0176] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the object perception method based on binaural auditory signal expression provided by the above-mentioned methods, the method comprising: obtaining position information of objects around the target object through a radar sensing device carried by the target object; the position information of the object includes the distance and angle of the object from the target object; analyzing the position information of the object to determine the type and movement trend of the object; determining the target object from the objects based on the type, movement trend, and distance and angle of the object from the target object; generating a target auditory signal for simulating the perception of the state information of the target object by the binaural ears of a person at the target object position based on the type, movement trend, and distance and angle of the target object from the target object; the state information of the target object includes type, distance, angle, and movement trend; broadcasting the target auditory signal to the target object through a hearing aid worn by the target object, so that the target object obtains the state information of the target object.

[0177] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0178] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0179] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An object perception method based on binaural auditory signal expression, characterized in that: include: Acquiring location information of objects around the target object through a radar sensing device carried by the target object; The position information of the object includes the distance and angle between the object and the target object; Analyze the location information of the object to determine the type and movement trend of the object; Determine a target object from the objects based on the type of the object, the movement trend, and the distance and angle from the target object; Based on the type, movement trend, distance and angle of the target object, a target auditory signal is generated to simulate the perception of state information of the target object by the two ears of a person at the position of the target object; the state information of the target object includes the type, distance, angle and movement trend; The target auditory signal is broadcast to the target object through the hearing aid worn by the target object, so that the target object obtains the status information of the target object.

2. The object perception method based on binaural auditory signal expression according to claim 1, characterized in that: The generating, based on the type, movement trend, distance and angle of the target object, a target auditory signal for simulating the perception of the state information of the target object by the two ears of a person at the position of the target object comprises: Determine the timbre of the target auditory signal based on the type of the target object and the distance from the target object; determine the intensity, binaural intensity difference and binaural time difference of the target auditory signal based on the distance and angle between the target object and the binaural ears of the target object; determine the frequency characteristics of the target auditory signal based on the movement trend of the target object; The target auditory signal is generated based on the determined timbre, intensity, interaural intensity difference, interaural time difference and frequency characteristics.

3. The object perception method based on binaural auditory signal expression according to claim 2, characterized in that: The determining the frequency characteristics of the target auditory signal based on the movement trend of the target object includes: When the moving speed of the target object is less than a speed threshold, determining that the moving trend of the target object is to remain stationary; The frequency characteristic of the target auditory signal is determined to be a pulsed pure tone with a fixed frequency.

4. The object perception method based on binaural auditory signal expression according to claim 2, characterized in that: The determining the frequency characteristics of the target auditory signal based on the movement trend of the target object includes: When the moving speed of the target object is greater than or equal to a speed threshold, determining that the moving trend of the target object is a moving state; Determine the carrier frequency of the target auditory signal as the initial frequency, and determine the frequency modulation signal of the target auditory signal based on the motion state of the target object to adjust the initial frequency; wherein the magnitude of the carrier frequency adjusted by the frequency modulation signal is associated with the motion speed of the target object, and the magnitude of the adjusted carrier frequency is associated with the distance between the target object and the target object after the motion; A frequency characteristic of the target auditory signal is determined based on the frequency modulation signal and the initial frequency.

5. The object perception method based on binaural auditory signal expression according to claim 2, characterized in that: The determining the timbre of the target auditory signal based on the type of the target object and the distance from the target object includes: Based on the type of the target object, determining a basic timbre model corresponding to the target object; According to the distance of the target object, the attenuation rate of the sound is calculated by using an attenuation function, and the high-frequency components and low-frequency components in the spectrum of the basic timbre model corresponding to the target object are adjusted to determine the timbre of the target auditory signal.

6. The object perception method based on binaural auditory signal expression according to claim 1, characterized in that: The step of determining the target object from the objects based on the type of the object, the movement trend, and the distance and angle from the target object includes: Determining objects within a target range based on the distance and angle of the objects from the target object; When the number of objects within the target range is greater than the target number, judging whether the objects are still within the target range after the target time based on the movement trend of the objects within the target range; Based on the first number of objects still within the target range after the target time and the preset priority, a second number of objects are screened out from objects within the target range whose motion trend is a stationary state; The first number of objects whose movement trends are in a moving state and the second number of objects in a stationary state are determined as the target objects.

7. The object perception method based on binaural auditory signal expression according to claim 1, characterized in that: The step of broadcasting the target auditory signal to the target object through a hearing aid worn by the target object includes: When the number of target objects is greater than one, the corresponding target auditory signals are broadcast to the target objects at intervals through the hearing aids worn by the target objects according to the sound emission priorities corresponding to the target objects; the sound emission priorities are determined based on the type of the target objects, the movement trend, and the distance and angle from the target objects.

8. An object perception device based on binaural auditory signal expression, characterized in that: An acquisition module is used to acquire the position information of objects around the target object through a radar sensing device carried by the target object; the position information of the object includes the distance and angle between the object and the target object; An analysis module, used to analyze the location information of the object and determine the type and movement trend of the object; A first processing module, configured to determine a target object from the objects based on the type of the object, the movement trend, and the distance and angle from the target object; A second processing module is used to generate a target auditory signal for simulating the perception of state information of the target object by two ears of a person at the position of the target object based on the type, motion trend, distance and angle of the target object from the target object; the state information of the target object includes type, distance, angle and motion trend; A feedback module is used to broadcast the target auditory signal to the target object through the hearing aid worn by the target object, so that the target object obtains the status information of the target object.

9. A hearing aid, characterized in that: include: A microphone, an audio processing module, a communication module, a power module and a speaker, wherein the hearing aid uses the object perception method based on binaural auditory signal expression as described in any one of claims 1 to 7 to broadcast the target auditory signal to the target object, so that the target object obtains the status information of the target object.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that: When the processor executes the program, the object perception method based on binaural auditory signal expression as described in any one of claims 1 to 7 is implemented.

Images