Audio-visual integrated perception system

Abstract

The invention discloses an audio-visual integrated sensing system, and the system comprises a sensing module which responds to an excitation signal so as to emit a detection sound wave signal, and is used for receiving a reflection sound wave signal and a sound source sound wave signal at the same time; the signal processing module is connected with the sensing module and comprises a first filter circuit and a second filter circuit, the first filter circuit is used for extracting imaging sound wave signals in the reflected sound wave signals and the sound source sound wave signals, and the second filter circuit is used for extracting positioning sound wave signals in the reflected sound wave signals and the sound source sound wave signals; and the data processing module is used for acquiring environment structure information based on the imaging sound wave signal, acquiring sound source position information based on the positioning sound wave signal, fusing the environment structure information and the sound source position information and outputting a target three-dimensional image. According to the invention, a single sensing system is used for replacing a visual sensing system and an auditory sensing system which are separated from each other, and the system has a dual sensing function of generating an environment structure and a dynamic sound source in a three-dimensional space.

Description

Technical Field

[0001] This invention belongs to the field of sensing technology, specifically relating to an integrated audiovisual sensing system. Background Technology

[0002] With the rise of the concept of embodied intelligent robots, the demand for multimodal perception systems is increasing. Visual and auditory sensors, as the core entry points for environmental interaction, directly determine the robot's environmental adaptability and the naturalness of human-robot interaction. Current mainstream solutions primarily rely on CMOS sensor imaging, LiDAR to generate 3D point cloud information, and ToF ultrasonic sensors for ranging for visual perception, while auditory perception typically uses microphone array technology to obtain the location of sound sources. Simultaneous implementation of visual and auditory perception requires the independent deployment of two systems, which not only increases system complexity, cost, and size but also hinders the real-time collaboration and fusion of data from the two modes, limiting the perception dimensions and efficiency.

[0003] Therefore, in order to address the aforementioned technical problems, it is necessary to provide an integrated audiovisual perception system. Summary of the Invention

[0004] The purpose of this invention is to provide an integrated audiovisual sensing system that can simultaneously support imaging and sound source localization through a single ultrasonic sensor, thereby achieving miniaturization of the sensing system and efficient, collaborative three-dimensional spatial perception.

[0005] To achieve the above objectives, a specific embodiment of the present invention provides the following technical solution:

[0006] An integrated audiovisual perception system, comprising:

[0007] The sensing module responds to an excitation signal to emit a detection acoustic wave signal and is used to simultaneously receive reflected acoustic wave signals and sound source acoustic wave signals.

[0008] The signal processing module, connected to the sensing module, includes a first filtering circuit and a second filtering circuit. The first filtering circuit is used to extract the imaging sound wave signal from the reflected sound wave signal and the sound source sound wave signal, and the second filtering circuit is used to extract the positioning sound wave signal from the reflected sound wave signal and the sound source sound wave signal.

[0009] The data processing module is used to acquire environmental structure information based on the imaging acoustic signal, acquire sound source location information based on the positioning acoustic signal, and fuse the environmental structure information and sound source location information to output a target three-dimensional image.

[0010] In one or more embodiments of the present invention, the signal processing module further includes a first analog-to-digital conversion circuit and a second analog-to-digital conversion circuit. The first analog-to-digital conversion circuit is connected between the first filtering circuit and the data processing module and is used to convert the imaging acoustic signal into an imaging digital signal. The second analog-to-digital conversion circuit is connected between the second filtering circuit and the data processing module and is used to convert the positioning acoustic signal into a positioning digital signal.

[0011] In one or more embodiments of the present invention, the data processing module includes an imaging receiving unit, a positioning receiving unit, and a data processing unit;

[0012] The imaging receiving unit is connected to the first analog-to-digital conversion circuit to receive imaging digital signals;

[0013] The positioning receiving unit is connected to the second analog-to-digital converter circuit to receive positioning digital signals;

[0014] The data processing unit is connected to the imaging receiving unit and the positioning receiving unit, and is used to acquire environmental structure information based on the imaging digital signal, acquire sound source location information based on the positioning digital signal, reconstruct a three-dimensional point cloud image of the environmental structure information and the sound source location information in the same three-dimensional coordinate system, and output a target three-dimensional image.

[0015] In one or more embodiments of the present invention, the signal processing module further includes a signal amplification circuit for amplifying the reflected sound wave signal and the sound source sound wave signal.

[0016] In one or more embodiments of the present invention, the first filtering circuit is a bandpass filter circuit; and / or,

[0017] The second filtering circuit is a bandpass filter circuit.

[0018] In one or more embodiments of the present invention, the sensing module includes a sixth amplifier, a first switch, and an ultrasonic sensor;

[0019] The input terminal of the sixth amplifier receives the excitation signal, and the output terminal is connected to the first terminal of the first switch, which is used to amplify the excitation signal to drive the ultrasonic sensor to emit a detection sound wave signal.

[0020] The second end of the first switch is connected to the signal processing module, and the third end is connected to the ultrasonic sensor. The first switch is used to control the connection between the sixth amplifier and the ultrasonic sensor based on the control signal, or to control the connection between the ultrasonic sensor and the signal processing module.

[0021] In one or more embodiments of the present invention, the ultrasonic sensor includes at least one of PMUT, CMUT, and piezoelectric ceramic.

[0022] In one or more embodiments of the present invention, the sensing module includes at least three ultrasonic sensor array elements and / or multiple ultrasonic sensor array elements arranged in an array.

[0023] In one or more embodiments of the present invention, the audiovisual integrated sensing system includes at least three channels, each channel including an ultrasonic sensor array element, a sixth amplifier, a first switch, and a signal processing module.

[0024] In one or more embodiments of the present invention, the audiovisual integrated perception system further includes an excitation signal module for generating excitation signals.

[0025] Compared with existing technologies, the audiovisual integrated perception system of the present invention utilizes the same ultrasonic sensor to simultaneously achieve ultrasonic imaging in active mode and sound source localization in passive mode. It can effectively distinguish, process and fuse these two different acoustic signal modes, and ultimately provide three-dimensional environmental structure information and three-dimensional spatial dynamic sound source location information simultaneously and synchronously, realizing the fusion perception of "imaging + localization". It eliminates the space occupation caused by multiple sensors, reduces system complexity and cost, and realizes dual perception of environmental structure and dynamic sound source in three-dimensional space, thereby improving perception efficiency. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a schematic diagram of the integrated audiovisual perception system in Embodiment 1 of the present invention;

[0028] Figure 2 This is a circuit diagram of the signal amplification circuit of the audiovisual integrated sensing system in Embodiment 1 of the present invention;

[0029] Figure 3 This is a circuit diagram of the first filter circuit of the audiovisual integrated perception system in Embodiment 1 of the present invention;

[0030] Figure 4 This is a circuit diagram of the second filter circuit of the audiovisual integrated perception system in Embodiment 1 of the present invention;

[0031] Figure 5This is a schematic diagram illustrating the working principle of the integrated audiovisual perception system in Embodiment 1 of the present invention.

[0032] Figure 6 This is a schematic diagram of the ultrasonic sensor in Embodiment 1 of the present invention. Detailed Implementation

[0033] To enable those skilled in the art to better understand the technical solutions in this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this disclosure.

[0034] The description uses the phrases "in this embodiment," "in other embodiments," or "in some embodiments," each of which may refer to one or more of the same or different embodiments. Furthermore, the terms "comprising," "including," "having," etc., used in relation to embodiments of this application are synonymous.

[0035] The terms "coupled," "connected," or "linked" in this specification include both direct and indirect connections. An indirect connection is a connection made through an intermediate medium, such as an electrical conduction medium, which may have parasitic inductance or capacitance. Indirect connections may also include connections made through other active or passive devices to achieve the same or similar functional purpose, such as connections through switches, follower circuits, or other circuits or components. Furthermore, in this invention, terms such as "first" and "second" are primarily used to distinguish one technical feature from another, and do not necessarily require or imply any actual relationship, quantity, or order between these technical features.

[0036] As mentioned in the background section, existing sensing systems can only acquire three-dimensional spatial information of the target area through one or more methods such as cameras, ultrasonic transducers, and lidar to achieve environmental modeling. Then, by combining this with ultrasonic receiving base station array to receive the ultrasonic waves emitted by the target, the system can achieve precise positioning of the target in the space.

[0037] The main drawback of existing technologies lies in the fact that their multimodal sensing capabilities require the combination of multiple different types of sensors: using visual sensors for environmental perception and auditory sensors for sound source localization require separate sensors. This discrete architecture is not only complex, costly, and bulky, but also suffers from a lack of real-time data interaction mechanisms between systems, resulting in the inability to synchronously integrate environmental and sound source information, thus isolating the multidimensional sensing data.

[0038] This invention provides an integrated audiovisual perception system, comprising:

[0039] The sensing module responds to an excitation signal to emit a detection acoustic wave signal and is used to simultaneously receive reflected acoustic wave signals and sound source acoustic wave signals.

[0040] The signal processing module, connected to the sensing module, includes a first filtering circuit and a second filtering circuit. The first filtering circuit is used to extract the imaging sound wave signal from the reflected sound wave signal and the sound source sound wave signal, and the second filtering circuit is used to extract the positioning sound wave signal from the reflected sound wave signal and the sound source sound wave signal.

[0041] The data processing module is used to acquire environmental structure information based on imaging acoustic signals, acquire sound source location information based on positioning acoustic signals, and fuse the environmental structure information and sound source location information to output a target three-dimensional image.

[0042] In one embodiment, the signal processing module further includes a first analog-to-digital conversion circuit and a second analog-to-digital conversion circuit. The first analog-to-digital conversion circuit is connected between the first filtering circuit and the data processing module and is used to convert the imaging acoustic wave signal into an imaging digital signal. The second analog-to-digital conversion circuit is connected between the second filtering circuit and the data processing module and is used to convert the positioning acoustic wave signal into a positioning digital signal.

[0043] In one embodiment, the data processing module includes an imaging receiving unit, a positioning receiving unit, and a data processing unit;

[0044] The imaging receiving unit is connected to the first analog-to-digital converter circuit to receive imaging digital signals;

[0045] The positioning receiving unit is connected to the second analog-to-digital converter circuit to receive positioning digital signals;

[0046] The data processing unit is connected to the imaging receiving unit and the positioning receiving unit. It is used to acquire environmental structure information based on imaging digital signals, acquire sound source location information based on positioning digital signals, reconstruct three-dimensional point cloud images of the environmental structure information and sound source location information in the same three-dimensional coordinate system, and output the target three-dimensional image.

[0047] In one embodiment, the signal processing module further includes a signal amplification circuit for amplifying the reflected sound wave signal and the sound source sound wave signal.

[0048] In one embodiment, the first filtering circuit is a fourth-order Chebyshev active bandpass filter.

[0049] In one embodiment, the second filtering circuit is a fourth-order Chebyshev active bandpass filter.

[0050] In one embodiment, the sensing module includes a sixth amplifier, a first switch, and an ultrasonic sensor;

[0051] The input of the sixth amplifier receives the excitation signal, and its output is connected to the first terminal of the first switch. It is used to amplify the excitation signal to drive the ultrasonic sensor to emit a detection sound wave signal.

[0052] The second end of the first switch is connected to the signal processing module, and the third end is connected to the ultrasonic sensor. The first switch is used to control the conduction between the sixth amplifier and the ultrasonic sensor based on the control signal, or to control the conduction between the ultrasonic sensor and the signal processing module.

[0053] This invention replaces two separate visual and auditory sensing systems with a single sensing system, possessing dual sensing capabilities for generating environmental structures and dynamic sound sources in three-dimensional space. This improves the synchronization accuracy of imaging and positioning, and reduces cost, system complexity, and size. The sensing system of this invention detects the position and shape of obstacles in the surrounding environment by emitting ultrasonic waves, while simultaneously locating sound sources in the surrounding environment.

[0054] The present invention will be further described below with reference to specific embodiments.

[0055] Example 1:

[0056] like Figure 1 As shown, an integrated audiovisual perception system in this embodiment includes:

[0057] Excitation signal module 40 is used to generate excitation signals;

[0058] The sensing module 10 is used to emit a detection sound wave signal in response to an excitation signal, and is also used to simultaneously receive the reflected sound wave signal S1 and the sound source sound wave signal S2.

[0059] The signal processing module 20 is connected to the sensing module 10 and includes a first filtering circuit 22 and a second filtering circuit 23. The first filtering circuit 22 is used to extract the imaging sound wave signal S11 from the reflected sound wave signal S1 and the sound source sound wave signal S2, and the second filtering circuit 23 is used to extract the positioning sound wave signal S21 from the reflected sound wave signal S1 and the sound source sound wave signal S2.

[0060] The data processing module 30 is used to acquire environmental structure information based on the imaging acoustic signal S11, acquire sound source location information based on the positioning acoustic signal S21, and fuse the environmental structure information and sound source location information to output a target three-dimensional image.

[0061] It is understood that the sound wave signal is used to detect the position and shape of obstacles in the surrounding environment, and a reflected sound wave signal S1 is generated after contact with the obstacle. The sound source in the surrounding environment will generate a sound source sound wave signal S2. Therefore, the sensing module 10 in this embodiment can simultaneously actively receive the reflected sound wave signal S1 and passively receive the sound source sound wave signal S2, so that the perception system realizes audiovisual integration, simultaneously supports ultrasonic imaging and sound source localization, can eliminate the space occupation caused by multiple sensors, reduce system complexity and cost, realize dual perception of environmental structure and dynamic sound source in three-dimensional space, and improve perception efficiency.

[0062] like Figure 1 As shown, the sensing module 10 includes a sixth amplifier 12, a first switch 13, and an ultrasonic sensor 11. The input terminal of the sixth amplifier 12 is connected to the excitation signal module 40 to receive the excitation signal, and the output terminal is connected to the first terminal of the first switch 13 to amplify the excitation signal to a suitable voltage to drive the ultrasonic sensor 11 to emit a detection sound wave signal.

[0063] The second end of the first switch 13 is connected to the signal processing module 20, and the third end is connected to the ultrasonic sensor 11. The first switch 13 is used to control the connection between the sixth amplifier 12 and the ultrasonic sensor 11 based on the control signal, or to control the connection between the ultrasonic sensor 11 and the signal processing module 20.

[0064] Understandably, in the first state, the first switch 13 controls the connection between the sixth amplifier 12 and the ultrasonic sensor 11, and the ultrasonic sensor 11 responds to the excitation signal to emit a detection sound wave signal. In the second state, the first switch 13 controls the connection between the ultrasonic sensor 11 and the signal processing module 20, and the signal processing module 20 receives the reflected sound wave signal S1 and the sound source sound wave signal S2. The first switch 13 periodically switches between the first state and the second state.

[0065] In this embodiment, by setting the first switch 13, the signal processing module 20 is prevented from being interfered with by the excitation signal when receiving the reflected sound wave signal S1 and the sound source sound wave signal S2, thereby improving the accuracy of signal processing.

[0066] The signal processing module 20 also includes a signal amplification circuit 21, which is used to amplify the reflected sound wave signal S1 and the sound source sound wave signal S2.

[0067] like Figure 2 As shown, the signal amplification circuit 21 includes a first sub-filter unit 211, a second resistor R2, a third resistor R3, a first amplifier AD1, a first voltage regulator unit 212, and a second voltage regulator unit 213.

[0068] In this embodiment, the first sub-filter unit 211 includes a first capacitor C1 and a first resistor R1, the first voltage regulator unit 212 includes a fourth capacitor C4 and a fifth capacitor C5, and the second voltage regulator unit 213 includes a second capacitor C2 and a third capacitor C3.

[0069] In this embodiment, the first sub-filter unit 211 is a low-pass filter used to retain low-frequency effective signals and filter out high-frequency noise. The second resistor R2 is used to increase the stability of the first sub-filter unit 211.

[0070] Specifically, the first terminal of the first resistor R1 is used to receive the reflected sound wave signal S1 and the sound source sound wave signal S2. The second terminal is connected to the first terminal of the first capacitor C1, the first terminal of the second resistor R2, and the non-inverting input terminal of the first amplifier AD1. The second terminal of the first capacitor C1 and the second terminal of the second resistor R2 are connected to the reference potential. In this embodiment, the reference potential is the ground potential.

[0071] The inverting input of the first amplifier AD1 is connected to the reference potential, and its output is used to output the amplified reflected sound wave signal S1 and the sound source sound wave signal S2. The third resistor R3 is connected to the first gain setting terminal of the first amplifier AD1 (i.e., Figure 2 Pin 8) shown is connected to the second gain setting terminal (i.e. Figure 2 Between pin 1 shown, the bias terminal of the first amplifier AD1 (i.e., Figure 2 Pin 5 shown is connected to the reference potential.

[0072] The second power supply terminal of the first amplifier AD1 is connected to the negative power supply and the second voltage regulator unit 213. The first power supply terminal of the first amplifier AD1 is connected to the positive power supply and the first voltage regulator unit 212. That is, the first terminals of the fourth capacitor C4 and the fifth capacitor C5 are connected to the reference potential, and the second terminals of the fourth capacitor C4 and the fifth capacitor C5 are connected to the first power supply terminal of the first amplifier AD1. The first terminals of the second capacitor C2 and the third capacitor C3 are connected to the reference potential, and the second terminals of the second capacitor C2 and the third capacitor C3 are connected to the second power supply terminal of the first amplifier AD1.

[0073] In this embodiment, the first amplifier AD1 is an AD620. It is understood that the operating principle of the signal amplification circuit 21 is well known in the prior art, and therefore will not be described in detail here. Any known or unknown first amplifier AD can be used without restriction.

[0074] like Figure 3As shown, the first filter circuit 22 includes a first voltage divider unit 221, a second sub-filter unit 222, a second amplifier AD2, a seventh resistor R7, an eighth resistor R8, a second voltage divider unit 225, a third sub-filter unit 226, a third amplifier AD, a twelfth resistor R12, a thirteenth resistor R13, a third voltage regulator unit 223, and a fourth voltage regulator unit 224.

[0075] In this embodiment, the first voltage divider unit 221 includes a fourth resistor R4 and a fifth resistor R5. The second sub-filter unit 222 includes a sixth capacitor C, a sixth resistor R6, and a seventh capacitor C7. The third voltage regulator unit 223 includes an eighth capacitor C8 and a ninth capacitor C9, and the fourth voltage regulator unit includes a tenth capacitor C10 and an eleventh capacitor C11.

[0076] The first terminal of the fourth resistor R4 serves as the input terminal of the first voltage divider unit 221. The second terminal of the fourth resistor R4 is connected to the first terminal of the fifth resistor R5 and serves as the output terminal of the second voltage divider unit 225. The second terminal of the fifth resistor R5 is connected to the reference potential. The first terminal of the sixth capacitor C6 is connected to the first terminal of the seventh capacitor C7 and serves as the first terminal of the second sub-filter unit 222. The first terminal of the sixth resistor R6 is connected to the second terminal of the sixth capacitor C6 and serves as the third terminal of the second sub-filter unit 222. The second terminal of the sixth resistor R6 is connected to the second terminal of the seventh capacitor C7 and serves as the second terminal of the second sub-filter unit 222.

[0077] The input terminal of the first voltage divider unit 221 receives the reflected sound wave signal S1 and the sound source sound wave signal S2. The output terminal is connected to the first terminal of the second sub-filter unit 222. The second terminal of the second sub-filter unit 222 is connected to the inverting input terminal of the second amplifier AD2. The third terminal is connected to the output terminal of the second amplifier AD2 and the second terminal of the eighth resistor R8. The non-inverting input terminal of the second amplifier AD2 is connected to the first terminal of the seventh resistor R7 and the first terminal of the eighth resistor R8. The second terminal of the seventh resistor R7 is connected to the reference potential.

[0078] The first power supply terminal of the second amplifier AD2 is connected to the positive power supply and the third voltage regulator unit 223, and the second power supply terminal is connected to the negative power supply and the fourth voltage regulator unit 224. Specifically, the first terminals of the eighth capacitor C and the ninth capacitor C are connected to the reference potential, and the second terminals of the eighth capacitor C8 and the ninth capacitor C9 are connected to the first power supply terminal of the second amplifier AD2. The first terminals of the tenth capacitor C10 and the eleventh capacitor C11 are connected to the reference potential, and the second terminals of the tenth capacitor C10 and the eleventh capacitor C11 are connected to the second power supply terminal of the second amplifier AD2.

[0079] The second voltage divider unit 225 includes a ninth resistor R9 and a tenth resistor R10. The first end of the ninth resistor R9 is connected to the output terminal of the second amplifier AD2 and serves as the input terminal of the second voltage divider unit 225. The second end of the ninth resistor R9 is connected to the first end of the tenth resistor R10 and serves as the output terminal of the second voltage divider unit 225. The third sub-filter unit 226 includes a twelfth capacitor C12, a thirteenth capacitor C13, and an eleventh resistor R11. The first end of the twelfth capacitor C12 is connected to the first end of the thirteenth capacitor C13 and forms the first terminal of the third sub-filter unit 226. The second end of the thirteenth capacitor C13 is connected to the second end of the eleventh resistor R11 and forms the second terminal of the third sub-filter unit 226. The second end of the twelfth capacitor C12 is connected to the first end of the eleventh resistor R11 and forms the third terminal of the third sub-filter unit 226.

[0080] The input terminal of the second voltage divider unit 225 is connected to the output terminal of the second amplifier AD2, and the output terminal is connected to the first terminal of the third sub-filter unit 226. The second terminal of the third sub-filter unit 226 is connected to the inverting input terminal of the third amplifier AD3, and the third terminal of the third sub-filter unit 226 is connected to the output terminal of the third amplifier AD3. The twelfth resistor R12 is connected between the non-inverting input terminal and the output terminal of the third amplifier AD3. The output terminal of the third amplifier AD3 is used to output the imaging acoustic signal S11. The thirteenth resistor R13 is connected between the non-inverting input terminal and the reference potential of the third amplifier AD3.

[0081] like Figure 4 As shown, the second filter circuit 23 includes a third voltage divider unit 231, a fourth sub-filter unit 232, a fourth amplifier AD, a fourth voltage divider unit 235, a fifth sub-filter unit 236, a fifth amplifier AD, a fifth voltage regulator unit 233, and a sixth voltage regulator unit 234.

[0082] The third voltage divider unit 231 in this embodiment includes a fourteenth resistor R14 and a fifteenth resistor R15. The first end of the fourteenth resistor R14 serves as the input end of the third voltage divider unit 231, and the second end of the fourteenth resistor R14 is connected to the first end of the fifteenth resistor R15 and serves as the output end of the third voltage divider unit 231.

[0083] In this embodiment, the fourth sub-filter unit 232 includes a fourteenth capacitor C14, a fifteenth capacitor C15, and a sixteenth resistor R16. The first terminal of the fourteenth capacitor C14 and the first terminal of the fifteenth capacitor C15 are connected and serve as the first terminal of the fourth sub-filter unit 232. The first terminal of the sixteenth resistor R16 and the second terminal of the fourteenth capacitor C14 serve as the third terminal of the second sub-filter unit 222. The second terminal of the sixteenth resistor R16 and the second terminal of the fifteenth capacitor C15 are connected and serve as the second terminal of the second sub-filter unit 222.

[0084] The input terminal of the third voltage divider unit 231 receives the reflected sound wave signal S1 and the sound source sound wave signal S2. The output terminal is connected to the first terminal of the fourth sub-filter unit 232. The second terminal of the fourth sub-filter unit 232 is connected to the inverting input terminal of the fourth amplifier AD4. The third terminal of the fourth sub-filter unit 232 is connected to the output terminal of the fourth amplifier AD4. The non-inverting input terminal of the fourth amplifier AD4 is connected to the reference potential.

[0085] The fifth voltage regulator unit 233 includes the sixteenth capacitor C16 and the seventeenth capacitor C17, and the sixth voltage regulator unit 234 includes the eighteenth capacitor C18 and the nineteenth capacitor C19.

[0086] The first power supply terminal of the fourth amplifier AD4 is connected to the positive power supply and the fifth voltage regulator unit 233, and the second power supply terminal is connected to the negative power supply and the sixth voltage regulator unit 234. Specifically, the first terminals of the eighteenth capacitor C18 and the nineteenth capacitor C19 are connected to the reference potential, and the second terminals of the eighteenth capacitor C18 and the nineteenth capacitor C19 are connected to the second power supply terminal of the fourth amplifier AD4. The first terminals of the sixteenth capacitor C16 and the seventeenth capacitor C17 are connected to the reference potential, and the second terminals of the sixteenth capacitor C16 and the seventeenth capacitor C17 are connected to the first power supply terminal of the fourth amplifier AD4.

[0087] In this embodiment, the fourth voltage divider unit 235 includes a seventeenth resistor R17 and an eighteenth resistor R18. The first end of the seventeenth resistor R17 is connected to the output of the fourth amplifier AD4 and serves as the input of the fourth voltage divider unit 235. The second end of the seventeenth resistor R17 is connected to the first end of the eighteenth resistor R18 and serves as the output of the fourth voltage divider unit 235. The second end of the eighteenth resistor R18 is connected to the reference potential.

[0088] In this embodiment, the fifth sub-filter unit 236 includes a twentieth capacitor C20, a twenty-first capacitor C21, and a nineteenth resistor R19. The first end of the twentieth capacitor C20 and the first end of the twenty-first capacitor C21 are connected and serve as the first end of the fifth sub-filter unit 236. The second end of the twentieth capacitor C20 is connected to the first end of the nineteenth resistor R19 and serves as the second end of the fifth sub-filter unit 236. The second end of the twenty-first capacitor C21 is connected to the second end of the nineteenth resistor R19 and serves as the third end of the fifth sub-filter unit 236.

[0089] The output of the fourth amplifier AD4 is connected to the input of the fourth voltage divider unit 235. The output of the fourth voltage divider unit 235 is connected to the first end of the fifth sub-filter unit 236. The second end of the fifth sub-filter unit 236 is connected to the output of the fifth amplifier AD5. The third end of the fifth sub-filter unit 236 is connected to the inverting input of the fifth amplifier AD5. The non-inverting input of the fifth amplifier AD5 is connected to the reference potential. The output is used to output the positioning acoustic signal S21.

[0090] The sensing system of this invention detects the position and shape of obstacles in the surrounding environment by emitting ultrasonic waves, and simultaneously locates sound sources in the surrounding environment. It is understood that the first filtering circuit 22 is a bandpass filter, which can extract the ultrasonic frequencies used for imaging from the amplified reflected sound wave signal S1 and the sound source sound wave signal S2. The second filtering circuit 23 is also a bandpass filter, which can extract the ultrasonic frequencies used for positioning from the amplified reflected sound wave signal S1 and the sound source sound wave signal S2. Therefore, this invention can replace two separate visual and auditory sensing systems with a single sensing system, possessing dual sensing functions of generating environmental structures and dynamic sound sources in three-dimensional space, improving the synchronization accuracy of imaging and positioning, and reducing cost, system complexity, and size.

[0091] like Figure 1 As shown, the signal processing module 20 also includes a first analog-to-digital converter circuit 24 and a second analog-to-digital converter circuit 25. The first analog-to-digital converter circuit 24 is connected between the first filter circuit 22 and the data processing module 30 and is used to convert the imaging acoustic signal S11 into an imaging digital signal. The second analog-to-digital converter circuit 25 is connected between the second filter circuit 23 and the data processing module 30 and is used to convert the positioning acoustic signal S21 into a positioning digital signal.

[0092] Understandably, the data processing module 30 is used to obtain environmental structure information based on the imaging acoustic signal S11 (i.e., imaging digital signal) after analog-to-digital conversion, to obtain sound source location information based on the positioning acoustic signal S21 (i.e., positioning digital signal) after analog-to-digital conversion, and to fuse the environmental structure information and sound source location information to output a target three-dimensional image.

[0093] like Figure 1 As shown, the data processing module 30 includes an imaging receiving unit 31, a positioning receiving unit 32, a data processing unit 33, and a data transmission unit 34;

[0094] The imaging receiving unit 31 is connected to the first analog-to-digital converter circuit 24 to receive imaging digital signals;

[0095] The positioning receiving unit 32 is connected to the second analog-to-digital converter circuit 25 to receive positioning digital signals;

[0096] The data processing unit 33 is connected to the imaging receiving unit 31 and the positioning receiving unit 32. It is used to acquire environmental structure information based on imaging digital signals, acquire sound source location information based on positioning digital signals, reconstruct three-dimensional point cloud images of environmental structure information and sound source location information in the same three-dimensional coordinate system, and output the target three-dimensional image.

[0097] Combination Figure 5 As shown, in this embodiment, the audiovisual integrated perception system ST responds to the excitation signal by emitting a detection sound wave signal. Upon contact with obstacles A1 and A2, the detection sound wave signal generates a corresponding reflected sound wave signal S1. Additionally, sound source B1 generates a sound source sound wave signal S2. The audiovisual integrated perception system ST simultaneously receives both the reflected sound wave signal S1 and the sound source sound wave signal S2, and performs three-dimensional point cloud image reconstruction in the same three-dimensional coordinate system through the data processing unit 33, outputting a target three-dimensional image. It can be understood that the target three-dimensional image includes the image and position information of obstacles A1 and A2, as well as the position information of sound source B1.

[0098] It is understood that the imaging and positioning principles of the data processing unit 33 are well known in the prior art, and therefore will not be described in detail here. Any known or unknown data processing unit 33 may be used here without restriction.

[0099] Furthermore, the ultrasonic sensor 11 in this embodiment includes at least one of PMUT, CMUT, and piezoelectric ceramic. The working medium of the ultrasonic sensor 11 includes air, other gases, liquids, and solids.

[0100] The ultrasonic sensor 11 includes at least three ultrasonic sensor elements or multiple ultrasonic sensor elements arranged in an array.

[0101] The ultrasonic sensor array element 101, the sixth amplifier 12, the first switch 13, and the signal processing module 20 constitute a channel 100 for signal processing. The audiovisual integrated sensing system in this embodiment includes at least three channels, with each channel 100 containing one ultrasonic sensor array element; that is, the ultrasonic sensor includes at least three ultrasonic sensor array elements 101. It is understood that a greater number of channels 100 can be selected as needed.

[0102] like Figure 6 As shown, the ultrasonic sensor 11 in this embodiment includes multiple ultrasonic sensor array elements arranged in an array.

[0103] It should be noted that, regardless of how many ultrasonic sensor array elements are used in the audiovisual integrated perception system, each ultrasonic sensor array element is used to simultaneously receive the reflected sound wave signal S1 and the sound source sound wave signal S2. That is, the ultrasonic sensor in this embodiment is used to simultaneously receive the reflected sound wave signal S1 and the sound source sound wave signal S2.

[0104] Furthermore, the integrated audiovisual perception system also includes an execution module 50, which is connected to the data transmission unit 34. The execution module 50 is used to perform display operations based on the target 3D image, or to issue obstacle avoidance, navigation, and other commands based on the position information in the target 3D image. In this embodiment, the execution module 50 is a computer (PC) used to display the target 3D image.

[0105] Furthermore, in this embodiment, the data processing module 30 is integrated on the electronic device 300, which includes an FPGA, a microcontroller, or an ASIC chip.

[0106] The beneficial effects of this invention include:

[0107] This invention utilizes the same ultrasonic sensor to simultaneously achieve ultrasonic imaging in active mode and sound source localization in passive mode, and can effectively distinguish, process and fuse these two different acoustic signal modes. Ultimately, it can simultaneously and synchronously provide three-dimensional environmental structure information and three-dimensional spatial dynamic sound source location information, realizing the fusion perception of "imaging + localization".

[0108] This invention replaces two separate visual and auditory sensing systems with a single sensing system, possessing dual sensing capabilities for generating environmental structures and dynamic sound sources in three-dimensional space. This improves the synchronization accuracy of imaging and positioning, and reduces cost, system complexity, and size.

[0109] The sensing system of the present invention detects the position and shape of obstacles in the surrounding environment by emitting ultrasonic waves, and at the same time locates the sound source in the surrounding environment.

[0110] This invention can eliminate the space occupation caused by multiple sensors, reduce system complexity and cost, realize dual perception of generating environmental structure and dynamic sound source in three-dimensional space, and improve perception efficiency.

[0111] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0112] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0113] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0114] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0115] It will be apparent to those skilled in the art that this disclosure is not limited to the details of the exemplary embodiments described above, and that this disclosure can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of this disclosure is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within this disclosure. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0116] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. An integrated audiovisual perception system, characterized in that, include: The sensing module responds to an excitation signal to emit a detection acoustic wave signal and is used to simultaneously receive reflected acoustic wave signals and sound source acoustic wave signals. The signal processing module, connected to the sensing module, includes a first filtering circuit and a second filtering circuit. The first filtering circuit is used to extract the imaging sound wave signal from the reflected sound wave signal and the sound source sound wave signal, and the second filtering circuit is used to extract the positioning sound wave signal from the reflected sound wave signal and the sound source sound wave signal. The data processing module is used to acquire environmental structure information based on the imaging acoustic signal, acquire sound source location information based on the positioning acoustic signal, and fuse the environmental structure information and sound source location information to output a target three-dimensional image.

2. The audiovisual integrated perception system according to claim 1, characterized in that, The signal processing module further includes a first analog-to-digital conversion circuit and a second analog-to-digital conversion circuit. The first analog-to-digital conversion circuit is connected between the first filtering circuit and the data processing module and is used to convert the imaging acoustic signal into an imaging digital signal. The second analog-to-digital conversion circuit is connected between the second filtering circuit and the data processing module and is used to convert the positioning acoustic signal into a positioning digital signal.

3. The audiovisual integrated perception system according to claim 2, characterized in that, The data processing module includes an imaging receiving unit, a positioning receiving unit, and a data processing unit; The imaging receiving unit is connected to the first analog-to-digital conversion circuit to receive imaging digital signals; The positioning receiving unit is connected to the second analog-to-digital converter circuit to receive positioning digital signals; The data processing unit is connected to the imaging receiving unit and the positioning receiving unit, and is used to acquire environmental structure information based on the imaging digital signal, acquire sound source location information based on the positioning digital signal, reconstruct a three-dimensional point cloud image of the environmental structure information and the sound source location information in the same three-dimensional coordinate system, and output a target three-dimensional image.

4. The audiovisual integrated perception system according to claim 1, characterized in that, The signal processing module also includes a signal amplification circuit for amplifying the reflected sound wave signal and the sound source sound wave signal.

5. The audiovisual integrated perception system according to claim 1, characterized in that, The first filtering circuit is a bandpass filter circuit; and / or, The second filtering circuit is a bandpass filter circuit.

6. The audiovisual integrated perception system according to claim 1, characterized in that, The sensing module includes a sixth amplifier, a first switch, and an ultrasonic sensor; The input terminal of the sixth amplifier receives the excitation signal, and the output terminal is connected to the first terminal of the first switch, which is used to amplify the excitation signal to drive the ultrasonic sensor to emit a detection sound wave signal. The second end of the first switch is connected to the signal processing module, and the third end is connected to the ultrasonic sensor. The first switch is used to control the connection between the sixth amplifier and the ultrasonic sensor based on the control signal, or to control the connection between the ultrasonic sensor and the signal processing module.

7. The audiovisual integrated perception system according to claim 6, characterized in that, The ultrasonic sensor includes at least one of PMUT, CMUT, and piezoelectric ceramic.

8. The audiovisual integrated perception system according to claim 6, characterized in that, The ultrasonic sensor includes at least three ultrasonic sensor elements and / or multiple ultrasonic sensor elements arranged in an array.

9. The audiovisual integrated perception system according to claim 8, characterized in that, The audiovisual integrated perception system includes at least three channels, each channel comprising an ultrasonic sensor array, a sixth amplifier, a first switch, and a signal processing module.

10. The audiovisual integrated perception system according to claim 1, characterized in that, The audiovisual integrated perception system also includes an excitation signal module for generating excitation signals.

Images