Control method and device of acoustic interaction device, vehicle and storage medium

Abstract

The embodiment of the application provides a kind of acoustic interaction device control method, device, vehicle and storage medium, the method comprises: the original sound field signal is collected by microphone array in acoustic interaction device, wherein, original sound field signal includes: at least one sound source signal;Original sound field signal is extracted using target feature extraction network, and target feature tensor is generated, wherein, target feature tensor is used to characterize the spatial feature and semantic feature of at least one sound source signal;Based on target feature tensor, the sound source analysis result of original sound field signal is determined, wherein, sound source analysis result contains the sound source direction corresponding to at least one sound source signal, and the target sound event type corresponding to at least one sound source signal;Acoustic interaction device is controlled to execute the response operation corresponding to sound source analysis result.The application solves the technical problem that the response accuracy of acoustic interaction device and user acoustic interaction is low in the related art.

Description

Technical Field

[0001] This application relates to the field of electrical technology, and more specifically, to a control method, apparatus, vehicle, and storage medium for an acoustic interactive device. Background Technology

[0002] With the popularization of smart homes, acoustic interaction devices have become important terminals in the home. The auditory interaction capabilities of acoustic interaction devices mainly rely on microphone arrays to achieve sound source localization. Related technologies usually adopt a serial mode of "localization first, interaction later", which often responds to every sound. It is easy to misinterpret environmental interference such as car noise and the sound of home appliances as interaction signals, resulting in invalid head-turning responses. In multi-sound source mixed scenarios, it is difficult to distinguish the user's voice from passive noise, and it tends to mechanically capture noise for response. The anti-interference ability is weak, which restricts the interactive intelligence and environmental perception capabilities of acoustic interaction devices, resulting in low response accuracy of acoustic interaction devices and users.

[0003] There is currently no good solution to the above problems. Summary of the Invention

[0004] This application provides a control method, apparatus, vehicle, and storage medium for an acoustic interaction device, to at least solve the technical problem of low response accuracy of acoustic interaction devices when interacting with users in related technologies.

[0005] According to one aspect of the embodiments of this application, a control method for an acoustic interaction device is provided, comprising: acquiring an original sound field signal through a microphone array in the acoustic interaction device, wherein the original sound field signal includes at least one sound source signal; extracting features from the original sound field signal using a target feature extraction network to generate a target feature tensor, wherein the target feature tensor is used to characterize the spatial and semantic features of at least one sound source signal; determining a sound source analysis result of the original sound field signal based on the target feature tensor, wherein the sound source analysis result includes the sound source location corresponding to at least one sound source signal and a target sound event type corresponding to at least one sound source signal; and controlling the acoustic interaction device to perform a response operation corresponding to the sound source analysis result.

[0006] In this embodiment of the application, a target feature extraction network is used to extract features from the original sound field signal to generate a target feature tensor, including: preprocessing the original sound field signal to obtain an acoustic feature tensor; and performing multi-dimensional joint feature extraction on the acoustic feature tensor based on the target feature extraction network to obtain the target feature tensor.

[0007] In this embodiment of the application, multi-dimensional joint feature extraction is performed on the acoustic feature tensor based on the target feature extraction network to obtain the target feature tensor. This includes: feature encoding of the acoustic feature tensor based on the feature encoder in the target feature extraction network to obtain a compressed feature tensor; and multi-dimensional attention feature decoding of the compressed feature tensor based on the feature decoder in the target feature extraction network to obtain the target feature tensor. The multi-dimensional attention includes channel attention, spectral attention, and temporal attention.

[0008] In this embodiment, the feature decoder includes a local perceptual unit layer, a multidimensional attention decoding layer, and an inverted residual feedforward network layer. Based on the feature decoder in the target feature extraction network, multidimensional attention feature decoding is performed on the compressed feature tensor to obtain the target feature tensor, including: based on the local perceptual unit layer, performing local time-frequency correlation perception on the compressed feature tensor to obtain a locally enhanced feature tensor; based on the multidimensional attention decoding layer, performing multidimensional self-attention feature decoding on the locally enhanced feature tensor to obtain an attention feature tensor; and based on the inverted residual feedforward network layer, performing feature enhancement processing on the attention feature tensor to obtain the target feature tensor.

[0009] In this embodiment, the multidimensional attention decoding layer includes a channel attention module, a spectrum attention module, and a temporal attention module. Based on the multidimensional attention decoding layer, multidimensional self-attention feature decoding is performed on the local enhancement feature tensor to obtain an attention feature tensor, including: based on the channel attention module, performing channel-dimensional self-attention processing on the local enhancement feature tensor to obtain a channel feature tensor; based on the spectrum attention module, performing frequency-dimensional self-attention processing on the channel feature tensor to obtain a frequency feature tensor; and based on the temporal attention module, performing time-dimensional self-attention processing on the frequency feature tensor to obtain an attention feature tensor.

[0010] In this embodiment of the application, based on the inverted residual feedforward network layer, the attention feature tensor is subjected to feature enhancement processing to obtain the target feature tensor, including: performing feature channel expansion processing on the attention feature tensor to obtain an expanded feature tensor; performing local feature enhancement processing on the expanded feature tensor to obtain an enhanced feature tensor; and performing residual fusion processing on the enhanced feature tensor to obtain the target feature tensor.

[0011] In this embodiment, the feature encoder includes a convolutional block layer, a normalization layer, an activation function layer, a time-frequency domain pooling layer, and a regularization layer. Based on the feature encoder in the target feature extraction network, feature encoding is performed on the acoustic feature tensor to obtain a compressed feature tensor, including: based on the convolutional block layer, local feature extraction is performed on the acoustic feature tensor to obtain an intermediate feature tensor; based on the normalization layer, the intermediate feature tensor is normalized to obtain a normalized feature tensor; based on the activation function layer, the normalized feature tensor is transformed to obtain an activated feature tensor; based on the time-frequency domain pooling layer, the activated feature tensor is compressed in the time-frequency domain to obtain a dimensionality-reduced feature tensor; and based on the regularization layer, the dimensionality-reduced feature tensor is regularized to obtain a compressed feature tensor.

[0012] In this embodiment, the original sound field signal is preprocessed to obtain an acoustic feature tensor, including: performing time-frequency domain transformation on the original sound field signal to obtain complex frequency domain signals for each microphone channel; mapping the complex frequency domain signals using a Mel filter bank to obtain Mel spectral features; performing a correlation phase transformation on the complex frequency domain signals to obtain correlation phase transformation features; and fusing the Mel spectral features and correlation phase transformation features to obtain an acoustic feature tensor. Preferably, fusing the Mel spectral features and correlation phase transformation features to obtain an acoustic feature tensor includes: concatenating the Mel spectral features and correlation phase transformation features along the feature channel dimension to obtain the acoustic feature tensor.

[0013] In this embodiment of the application, the sound source analysis result of the original sound field signal is determined based on the target feature tensor, including: parsing the target feature tensor to obtain the sound source location corresponding to at least one sound source signal and the confidence level of the sound event type corresponding to at least one sound source signal; determining the target sound event type corresponding to at least one sound source signal based on the confidence level of the sound event type corresponding to at least one sound source signal; and integrating the sound source location and the corresponding target sound event type corresponding to at least one sound source signal to obtain the sound source analysis result.

[0014] In this embodiment of the application, parsing the target feature tensor to obtain the sound source location corresponding to at least one sound source signal and the confidence level of the sound event type corresponding to at least one sound source signal includes: parsing the target feature tensor to obtain the vector direction and vector length of the target feature tensor; determining the sound source location corresponding to at least one sound source signal based on the vector direction of the target feature tensor; and determining the confidence level of the sound event type corresponding to at least one sound source signal based on the vector length of the target feature tensor.

[0015] In this embodiment, controlling the acoustic interaction device to perform a response operation corresponding to the sound source analysis result includes: when at least one sound source signal is multiple sound source signals, obtaining the response priority of the target sound event type corresponding to the multiple sound source signals; determining the response operation corresponding to the multiple sound source signals based on the target sound event type corresponding to the multiple sound source signals; and controlling the acoustic interaction device to sequentially execute the response operation corresponding to the multiple sound source signals based on the response priority. Preferably, the target sound event type includes: security event type, user voice interaction event type, and environmental interference event type, wherein the response priority corresponding to the security event type is greater than the response priority corresponding to the user voice interaction event type, and the response priority corresponding to the user voice interaction event type is greater than the response priority corresponding to the environmental interference event type.

[0016] In this embodiment of the application, the method further includes: when the target sound event type is a security event type, guiding the acoustic interaction device to monitor the corresponding security event based on the location of the corresponding sound source, and performing an alarm operation when the danger level of the security event is greater than a preset danger level; when the target sound event type is a user voice interaction event type, controlling the microphone array to turn to the location of the corresponding sound source to enhance user voice interaction; and when the target sound event type is an environmental interference event type, suppressing the sound source signal at the location of the corresponding sound source.

[0017] According to another aspect of the embodiments of this application, a control device for an acoustic interaction device is also provided, comprising: a acquisition module, configured to acquire an original sound field signal through a microphone array in the acoustic interaction device, wherein the original sound field signal includes at least one sound source signal; an extraction module, configured to extract features from the original sound field signal using a target feature extraction network to generate a target feature tensor, wherein the target feature tensor is used to characterize the spatial and semantic features of at least one sound source signal; a determination module, configured to determine the sound source analysis result of the original sound field signal based on the target feature tensor, wherein the sound source analysis result includes the sound source orientation corresponding to at least one sound source signal and the target sound event type corresponding to at least one sound source signal; and an execution module, configured to control the acoustic interaction device to perform a response operation corresponding to the sound source analysis result.

[0018] According to another aspect of the embodiments of this application, a vehicle is also provided, including: a memory storing an executable program; and a processor for running the program, wherein the program executes the methods in various embodiments of this application when it runs.

[0019] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein, when the executable program is running, it controls the device where the computer-readable storage medium is located to perform the methods of various embodiments of this application.

[0020] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the methods of various embodiments of this application.

[0021] According to another aspect of the embodiments of this application, a computer program product is also provided, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the methods in various embodiments of this application.

[0022] According to another aspect of the embodiments of this application, a computer program is also provided, which, when executed by a processor, implements the methods of the various embodiments of this application.

[0023] In this embodiment, firstly, the original sound field signal is acquired using a microphone array in the acoustic interaction device; then, a target feature extraction network is used to extract features from the original sound field signal, generating a target feature tensor; next, based on the target feature tensor, the sound source analysis result of the original sound field signal is determined; finally, the acoustic interaction device is controlled to execute a response operation corresponding to the sound source analysis result. This application acquires the original sound field signal using a microphone array and generates a target feature tensor containing spatial and semantic features using a target feature extraction network. This spatiotemporal-frequency joint extraction allows the target feature extraction network to simultaneously capture the directional information and semantic type of the sound source signal in the deep feature space, avoiding error accumulation and delay caused by step-by-step processing. Subsequently, the sound source directional and target acoustic event type are calculated based on the target feature tensor, realizing the binding recognition of directional and semantic type, enabling the acoustic interaction device to distinguish different sound source signals according to the sound source directional and sound event type. Finally, differentiated response operations are executed based on the sound source analysis results. This proactive interaction strategy based on the sound source location and sound event type effectively filters out invalid signals, improves the anti-interference capability and interaction targeting of acoustic interaction devices in complex sound fields, thereby improving the response accuracy and naturalness of acoustic interaction, and thus solving the technical problem of low response accuracy of acoustic interaction devices and users in related technologies. Attached Figure Description

[0024] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0025] Figure 1 This is a flowchart of a control method for an acoustic interactive device according to an embodiment of the present invention;

[0026] Figure 2 This is a schematic diagram of the network structure of a target feature extraction network according to an embodiment of the present invention;

[0027] Figure 3 This is a schematic diagram of the control process of an acoustic interaction device according to an embodiment of the present invention;

[0028] Figure 4 This is a schematic diagram of the control process of an optional acoustic interaction device according to an embodiment of the present invention;

[0029] Figure 5 This is a schematic diagram of the control process of another optional acoustic interaction device according to an embodiment of the present invention;

[0030] Figure 6 This is a schematic diagram of a control device for an acoustic interactive device according to an embodiment of the present invention. Detailed Implementation

[0031] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0032] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0033] According to an embodiment of this application, an embodiment of a control method for an acoustic interactive device is provided. The steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowcharts, in some cases the steps shown or described may be performed in a different order than that shown here.

[0034] This embodiment provides a control method for an acoustic interactive device. Figure 1 This is a flowchart of a control method for an acoustic interactive device according to an embodiment of this application, such as... Figure 1 As shown, the process includes the following steps:

[0035] Step S102: Acquire the original sound field signal through the microphone array in the acoustic interaction device.

[0036] The original sound field signal includes at least one sound source signal.

[0037] The aforementioned acoustic interaction device can refer to an intelligent hardware terminal that possesses sound perception capabilities and can interact with users via voice. In specific embodiments of this application, it can refer to companion-type physical devices, such as companion home service robots, smart speakers, etc. This acoustic interaction device integrates a microphone array, processor, and actuators, such as servo motors, cameras, and speakers, and can collect ambient sound and perform actions such as turning its head, engaging in dialogue, issuing alarms, or suppressing noise based on the analysis results.

[0038] The aforementioned microphone array can refer to an acoustic sensor set consisting of multiple microphones installed on an acoustic interaction device. Its function is to collect the original sound field signal in the environment and to provide basic data for subsequent spatial orientation estimation by measuring the time difference or phase difference of the signals received by different microphones.

[0039] The aforementioned raw sound field signal can refer to a continuous audio stream directly acquired by a microphone array without in-depth processing. In practice, time-domain audio signals can be acquired at a sampling frequency. This raw sound field signal can contain mixed information of different sounds in the environment, including target sound sources, background noise, and interference sounds.

[0040] The aforementioned at least one sound source signal can refer to a sound source with a specific spatial location and physical properties that exists within the original sound field signal. It can be a single sound source signal or multiple sound source signals existing simultaneously, such as the presence of a user's voice and television background noise. In the joint perception task, it is necessary to be able to distinguish and identify these different sound source signals.

[0041] As an optional implementation, a microphone array consisting of multiple microphones is deployed in the acoustic interaction device. The number of microphones can be four to six to balance computational resources and positioning accuracy. These microphones are evenly distributed on the device surface, forming a specific geometric structure, such as a circular or linear array. When at least one sound source is present in the ambient sound field, sound waves arrive at each microphone with different propagation paths and arrival times. Each microphone in the array independently acquires a time-domain analog signal, which is then converted into a digital audio stream through analog-to-digital conversion. Due to the different physical locations of the sound source and each microphone, the same sound source signal exhibits significant time and amplitude differences in each channel. The acquired raw sound field signal is a collection of signals from multiple channels, which may include target sound sources, such as user voices and hazard sounds, as well as ambient noise, reverberation, and other interfering sound sources. This process preserves the original spatiotemporal information of the sound waves, providing a high-fidelity data foundation for subsequent feature extraction.

[0042] In a real home environment, multiple sound sources may exist simultaneously. For example, a user might be speaking on one side while a television is playing on the other, or a user's voice might be accompanied by the sound of breaking glass, indicating a safety incident. The signal collected by the microphone array is a spatial mixture of these sound source signals. In practice, front-end signal processing modules, such as automatic gain control and echo cancellation, can perform preliminary cleaning of the original sound field signal, removing device noise or far-end echoes while retaining aliasing information from external sound sources. Acoustic interactive devices possess multi-source target perception capabilities. The collected mixed signal can be segmented into short time frames and entered into a time-frequency domain transformation stage. This allows subsequent feature extraction networks to simultaneously analyze sound source features from different directions and categories, providing raw data input containing multi-source information for joint perception.

[0043] With the above setup, by employing a microphone array to acquire raw sound field signals containing signals from at least one sound source, and by preserving the time delay and phase information of sound reaching different microphones through multi-channel acquisition, the acoustic interaction device can handle scenarios with multiple sound sources coexisting in complex home environments. It can focus on the main interactive object and also capture accompanying environmental noise or sudden safety events. This physical-level preservation of multi-channel information provides data support for subsequent deep learning networks to extract spatiotemporal joint features, enabling accurate differentiation of sounds from different directions and categories in noisy environments. This provides a reliable basis for subsequent priority response and intelligent interaction decisions, improving the robustness and practicality of the acoustic interaction device in real-world scenarios.

[0044] Step S104: Use the target feature extraction network to extract features from the original sound field signal and generate the target feature tensor.

[0045] The target feature tensor is used to characterize the spatial and semantic features of at least one sound source signal.

[0046] The aforementioned target feature extraction network can refer to the neural network used in this application to extract high-dimensional features from the original sound field signal. This target feature extraction network can employ an encoder-decoder architecture, specifically including: a feature encoder for compressing features; and a feature decoder that preserves the index relationship of microphone channel pairs and sequentially fuses features according to spatial phase, spectral texture, and temporal evolution to address near-field interference issues.

[0047] The aforementioned target feature tensor refers to the deep feature data representation output by the target feature extraction network after encoding and decoding. The target feature tensor can characterize the spatial and semantic features of the sound source signal and can serve as the basic data carrier for subsequent location calculation and event type identification.

[0048] As an optional implementation, the acquired multi-channel audio data can be preprocessed to extract two complementary feature streams: a semantic feature stream and a spatial feature stream. Specifically, the time-domain signal can be converted to a complex frequency-domain signal using a short-time Fourier transform. Then, a log-Mel spectrogram is obtained using a Mel filter bank mapping as the semantic feature stream. This log-Mel spectrogram can characterize the frequency texture and semantic category clues of the sound, such as the spectral distribution characteristics of user voice, music, or noise. The generalized cross-correlation phase transform map between each pair of microphone channels can be calculated as the spatial feature stream. This generalized cross-correlation phase transform map preserves the index relationship of the microphone channel pairs, characterizing the time delay and phase difference of sound reaching different microphones, i.e., spatial location clues. Subsequently, the semantic and spatial feature streams are concatenated or fused along the channel dimension to form an acoustic feature tensor containing time frames, frequency indices, and multi-channel information. This process, through structured feature construction, explicitly preserves the semantic content and spatial phase information of the sound, providing input data rich in spatiotemporal semantic information for subsequent deep networks. Next, the acoustic feature tensor can be input into the target feature extraction network, which can include a feature encoder and a feature decoder. The feature encoder performs local feature extraction and dimensionality reduction compression on the input tensor through multi-layer convolutional operations, capturing local time-frequency correlations to generate a compressed feature tensor. The feature decoder performs multi-dimensional attention decoding on the compressed features, reconstructing the feature dimensions through serial or parallel processing of channel attention, spectral attention, and temporal attention. During this process, the target feature extraction network automatically learns and strengthens spatial phase patterns related to sound source localization and semantic spectral patterns related to sound event recognition. Finally, the target feature extraction network can output a high-dimensional target feature tensor, which is a deep representation abstracted by a deep neural network, implicitly containing a highly coupled representation of the spatial orientation information and semantic category information of the sound source signal, providing a unified data carrier for subsequent analysis.

[0049] As an alternative implementation, the target feature extraction network can employ a dual-stream parallel feature extraction architecture to extract features from the original sound field signal and generate a target feature tensor. Specifically, the original sound field signal can be divided into two independent processing streams. The spatial processing stream can utilize a generalized cross-correlation phase transform to calculate the cross-correlation function of channel pairs between microphone arrays, preserving phase information to construct a spatial phase feature map. This spatial phase feature map retains the channel pair index to reflect spatial geometric relationships. The semantic processing stream can utilize a short-time Fourier transform to obtain spectral amplitude, which is then mapped using a Mel filter bank and logarithmically compressed to generate a log-Mel spectrogram, representing the spectral texture of the sound. Subsequently, a spatiotemporal alignment module is introduced to align the spatial phase feature map and the Mel spectrogram in terms of time frames and frequency indices. Next, a cross-modal fusion layer can be used to weight the fused multi-channel data using a channel attention mechanism to enhance the response of important spatial phase and spectral features and suppress noise interference. Finally, the weighted feature sequence is input into a lightweight convolutional neural network for dimensionality reduction and nonlinear transformation, outputting a high-dimensional dense target feature tensor. This target feature tensor integrates spatial phase cues and spectral semantic cues through the channel dimension, achieving deep fusion of spatial and semantic features and providing a unified representation for subsequent joint perception.

[0050] Through the above setup, a target feature tensor representing spatial and semantic features is generated using a target feature extraction network, achieving joint perception of sound source localization and sound event detection. By fusing semantic and spatial information, the content and location information of the sound can be utilized simultaneously. The multi-dimensional joint extraction capability of deep neural networks enables automatic learning and enhancement of important features in complex sound fields, suppressing background noise interference and improving the ability to distinguish multi-source scenes. The generated target feature tensor maps spatial location and semantic category to a unified vector space, allowing subsequent parsing steps to simultaneously acquire sound source location and event type. This improves the interaction accuracy and anti-interference capability of acoustic interaction devices in real-world home environments, providing a high-quality feature foundation for subsequent intelligent response decisions.

[0051] Step S106: Based on the target feature tensor, determine the sound source analysis results of the original sound field signal.

[0052] The sound source analysis results include the sound source location corresponding to at least one sound source signal, and the target sound event type corresponding to at least one sound source signal.

[0053] The aforementioned sound source orientation can refer to the spatial coordinates of the sound source signal, which can be expressed as azimuth and elevation angles. In this application, the sound source orientation can be obtained by analyzing the direction of the multidimensional vector of the target feature tensor.

[0054] The aforementioned target sound event types can refer to the category labels obtained after semantic classification of the identified sound source signals. Target sound event types can include: safety event types, such as the sound of breaking glass, screams, and falling heavy objects, which have high response priority; user voice interaction event types, such as user speech and wake words, used to trigger dialogue interaction; and environmental interference event types, such as television sounds, fan sounds, and robot vacuum cleaner noise, which are identified as invalid interaction signals or noise that needs to be suppressed. The target sound event type can be determined by analyzing the length of the vectors in the target feature tensor and combining it with classification logic.

[0055] As an optional implementation, the target feature tensor can be analyzed to determine the sound source analysis results of the original sound field signal. The analysis process calculates the vector magnitude of the target feature tensor, which represents the confidence level or activity intensity of the sound event. If the confidence level exceeds a preset threshold, the event is determined to exist. Furthermore, the sound source azimuth can be calculated using the vector direction of the target feature tensor, and the azimuth and elevation angles can be solved using the arctangent function. This processing method couples direction and magnitude into the same vector, eliminating the need for additional classification or regression heads. A single forward propagation can simultaneously obtain the spatial location of the sound source and the validity judgment of the event type, achieving a unified mathematical space for localization and recognition.

[0056] As an alternative implementation, after parsing the target feature tensor, the confidence level of each preset category of sound events and the corresponding sound source location are obtained. Each preset category is traversed, and categories with confidence levels higher than a threshold are identified as target sound event types. For example, if the confidence level of the user voice category is 0.8 and the confidence level of the television sound category is 0.3, and the corresponding threshold is set to 0.5, then the user voice is identified as the target sound event type. Subsequently, the selected valid target sound event types are integrated with their corresponding sound source locations. This process can involve multi-source processing; that is, if multiple valid sound sources of different categories are detected, such as simultaneous user voice and glass breaking sound, their respective locations and types are determined. Finally, the sound source analysis results are constructed into a structured dataset containing multiple entries, each entry representing a confirmed sound source instance, providing explicit data input for subsequent prioritization and response operations.

[0057] Through the above settings, the sound source analysis results are determined based on the target feature tensor, achieving synchronous output of sound source localization and sound event detection. By unifying the analysis of vector magnitude and direction, localization and recognition can be completed in parallel during a single neural network forward propagation, improving real-time performance and meeting the low-latency interaction requirements of the companion agent. This joint perception mechanism enables the differentiation of sound sources from different directions and categories, improving the accuracy and anti-interference capability of the interaction. Finally, the output sound source analysis results contain clear location and event type, providing a structured basis for subsequent priority-based intelligent responses. This allows the acoustic interaction device to adopt differentiated control strategies based on event type, realizing a shift from passive listening to active intelligent interaction.

[0058] Step S108: Control the acoustic interaction device to perform a response operation corresponding to the sound source analysis results.

[0059] The aforementioned response operations refer to the specific control actions performed by the acoustic interaction device based on the sound source analysis results, namely the sound source location and the type of the target sound event. Depending on the priority and category of the event type, the response operations may include: for security events, executing alarm operations when the danger level exceeds a preset threshold, such as turning towards the sound source, starting camera recording, or sending an alarm notification; for user voice interaction events, controlling the microphone array or the device body to turn towards the corresponding sound source location to enhance sound pickup and thus enter voice wake-up or voice dialogue mode; for environmental interference events, applying nulls at the corresponding sound source location to suppress noise signals, or ignoring the signal without performing any interactive actions, thereby avoiding invalid wake-up.

[0060] As an optional implementation, all identified target sound event types in the current sound source analysis results can be acquired and sorted according to a preset priority mapping relationship. The priority can be set as follows: safety event types, such as the sound of breaking glass or screaming, are of higher priority; user voice interaction event types are secondary; and environmental interference event types are of lower priority. When multiple sound source signals are detected, the execution order is determined according to priority. For example, if a user's voice and a TV background noise are detected simultaneously, the user's voice event is processed first, controlling the device to turn towards the user's voice and initiate interaction; for the TV background noise, due to its lower priority, no turning or wake-up operation is performed, and it is treated as an interference signal. If a safety event and a user's voice are detected, the alarm or monitoring strategy for the safety event is executed first, temporarily suspending or interrupting the user's voice interaction process. Through this priority-based queue management mechanism, it is ensured that the intelligent agent can prioritize responding to high-value or high-risk events in complex multi-sound source environments, avoiding action conflicts and resource waste, and achieving orderly and reasonable multi-task processing.

[0061] For sound sources identified as user voice interaction events, the mechanical components of the acoustic interaction device, such as the head servo and the pan-tilt unit, can be controlled to turn towards the azimuth angle corresponding to the sound source. This improves the microphone array's pickup gain for user voices in that direction and can activate beamforming technology to enhance the signal-to-noise ratio, subsequently entering voice wake-up or dialogue mode. For sound sources identified as security events, the acoustic interaction device can be controlled to immediately turn towards the sound source, activate the camera for visual monitoring or recording, send an alarm notification to the user terminal via the communication module, and trigger a local audible and visual alarm. For sound sources identified as environmental interference events, no mechanical turning or hardware activation is performed. At the signal processing level, nulls are set in the beamformer corresponding to the direction of the interfering sound source to actively suppress noise signals from that direction from entering the subsequent voice recognition link, preventing false wake-up triggers or interference with normal interaction. This differentiated control allows the acoustic interaction device to flexibly allocate hardware resources or adjust signal processing parameters according to the semantic intent of the sound, achieving accurate intelligent responses.

[0062] Through the above settings, the acoustic interaction device is controlled to perform response operations corresponding to the sound source analysis results, realizing closed-loop control from auditory perception to physical interaction, thus improving the practicality and intelligence level of the companion intelligent agent. The event-category-based differentiated control strategy enables appropriate actions to be taken according to the specific meaning of the sound, such as directional sound pickup to enhance interaction quality, noise suppression to improve recognition accuracy, and proactive monitoring of dangerous events to ensure home safety, thereby enhancing the environmental adaptability and naturalness of the acoustic interaction device in smart home scenarios.

[0063] In this embodiment, firstly, the original sound field signal is acquired using a microphone array in the acoustic interaction device; then, a target feature extraction network is used to extract features from the original sound field signal, generating a target feature tensor; next, based on the target feature tensor, the sound source analysis result of the original sound field signal is determined; finally, the acoustic interaction device is controlled to execute a response operation corresponding to the sound source analysis result. This application acquires the original sound field signal using a microphone array and generates a target feature tensor containing spatial and semantic features using a target feature extraction network. This spatiotemporal-frequency joint extraction allows the target feature extraction network to simultaneously capture the directional information and semantic type of the sound source signal in the deep feature space, avoiding error accumulation and delay caused by step-by-step processing. Subsequently, the sound source directional and target acoustic event type are calculated based on the target feature tensor, realizing the binding recognition of directional and semantic type, enabling the acoustic interaction device to distinguish different sound source signals according to the sound source directional and sound event type. Finally, differentiated response operations are executed based on the sound source analysis results. This proactive interaction strategy based on the sound source location and sound event type effectively filters out invalid signals, improves the anti-interference capability and interaction targeting of acoustic interaction devices in complex sound fields, thereby improving the response accuracy and naturalness of acoustic interaction, and thus solving the technical problem of low response accuracy of acoustic interaction devices and users in related technologies.

[0064] In this embodiment of the application, a target feature extraction network is used to extract features from the original sound field signal to generate a target feature tensor, including: preprocessing the original sound field signal to obtain an acoustic feature tensor; and performing multi-dimensional joint feature extraction on the acoustic feature tensor based on the target feature extraction network to obtain the target feature tensor.

[0065] The aforementioned acoustic feature tensor refers to a multidimensional data tensor formed by preprocessing the raw time-domain sound field signal acquired by the microphone array. This preprocessing can include short-time Fourier transform, semantic feature extraction, spatial feature extraction, and feature fusion, resulting in a tensor containing information about time, frequency, and channel dimensions. The acoustic feature tensor can serve as a standardized data format for neural network input. It transforms physical sound field signals into a machine-understandable tensor form, providing fundamental data support for subsequent encoding using target feature extraction networks, multidimensional attention decoding, and sound source localization and event detection.

[0066] As an optional implementation, the original sound field signal can be preprocessed to generate an acoustic feature tensor. Specifically, a short-time Fourier transform can be performed on the multi-channel audio to obtain a complex frequency domain signal. A log-Mel spectrogram can be generated using Mel filter bank mapping to characterize the spectral texture and semantic cues of the sound. Furthermore, the generalized cross-correlation phase transform between microphone channel pairs can be calculated to characterize spatial cues such as arrival time difference and phase difference. Subsequently, the log-Mel spectrogram features and the generalized cross-correlation phase transform features can be concatenated or fused along the channel dimension, preserving the channel pair index relationship of the generalized cross-correlation phase transform features, forming an acoustic feature tensor containing time frames, frequency indices, and multi-channel information. Next, this acoustic feature tensor is input into a target feature extraction network for multi-dimensional joint feature extraction. The target feature extraction network can perform local feature extraction and compression on the input tensor using a feature encoder to obtain a compressed feature tensor. Then, a feature decoder performs multi-dimensional attention decoding, sequentially utilizing channel attention, spectral attention, and temporal attention modules to progressively enhance and fuse the compressed features in terms of spatial phase, spectral texture, and temporal evolution features. Finally, the target feature extraction network outputs a deeply coupled target feature tensor, which simultaneously contains the spatial orientation information of the sound source and the semantic category information of the sound event, providing a data foundation for subsequent joint analysis.

[0067] Through the above setup, an acoustic feature tensor incorporating spatiotemporal cues is generated via preprocessing. Then, a target feature extraction network is used for multi-dimensional joint feature extraction, achieving feature-level deep fusion of sound source localization and sound event detection. The generated target feature tensor maps spatial location and semantic category to a unified high-dimensional space, enabling subsequent parsing steps to synchronously and efficiently acquire sound source location and event type. This provides high-quality, highly consistent feature support for intelligent interaction decisions in acoustic interactive devices.

[0068] In this embodiment of the application, multi-dimensional joint feature extraction is performed on the acoustic feature tensor based on the target feature extraction network to obtain the target feature tensor. This includes: feature encoding of the acoustic feature tensor based on the feature encoder in the target feature extraction network to obtain a compressed feature tensor; and multi-dimensional attention feature decoding of the compressed feature tensor based on the feature decoder in the target feature extraction network to obtain the target feature tensor. The multi-dimensional attention includes channel attention, spectral attention, and temporal attention.

[0069] The aforementioned feature encoder can refer to the first half of the target feature extraction network. Its main function is to reduce the dimensionality and perform preliminary feature compression on the input acoustic feature tensor in order to extract important semantic and spatial information and reduce the amount of computation.

[0070] The compressed feature tensor mentioned above refers to the output of the feature encoder, which is a high-dimensional data representation after dimensionality reduction and feature compression. The compressed feature tensor retains important semantic and spatial information from the original acoustic features, removes redundant data, and makes subsequent network processing more efficient.

[0071] The aforementioned feature decoder can refer to the latter part of the target feature extraction network. Its main function is to upsample, enhance, and fuse multidimensional attention in the compressed feature tensor to recover or generate the target feature tensor for localization and recognition.

[0072] As an optional implementation, the acoustic feature tensor can be dimensionality reduced and its features compressed using a feature encoder. The acoustic feature tensor is sequentially passed through convolutional block layers to extract local time-frequency correlations, resulting in an intermediate feature tensor. Subsequently, it undergoes normalization and nonlinear transformation through normalization and activation function layers to obtain a compressed feature tensor. Then, a feature decoder can perform multi-dimensional attention feature decoding on the compressed feature tensor. The decoding process can sequentially include three steps: a channel attention module performs channel-dimensional self-attention processing on the compressed features to capture the phase correlation between microphone channels, thereby locking the sound source's directional range; a spectral attention module performs frequency-dimensional self-attention processing on the channel features to distinguish the spectral texture differences of different sound events; and a temporal attention module performs temporal-dimensional self-attention processing on the frequency features to confirm the temporal stability of the sound events. Through these three sequential processing dimensions, the network gradually refines the feature representation, ultimately outputting a target feature tensor containing spatial orientation and semantic category information.

[0073] Through the above setup, efficient and accurate feature decoupling and reconstruction are achieved via compression by the feature encoder and multidimensional serial attention processing by the feature decoder. The introduction of the feature encoder effectively reduces the data dimensionality, decreases the computational load of subsequent networks, and preserves important feature information. Multidimensional attention feature decoding of the compressed feature tensor enhances robustness in the face of strong near-field interference and multi-source mixed scenes, ensuring that the generated target feature tensor can accurately represent the true location and event type of the sound source, providing a reliable guarantee for subsequent high-precision joint perception.

[0074] In this embodiment, the feature decoder includes a local perceptual unit layer, a multidimensional attention decoding layer, and an inverted residual feedforward network layer. Based on the feature decoder in the target feature extraction network, multidimensional attention feature decoding is performed on the compressed feature tensor to obtain the target feature tensor, including: based on the local perceptual unit layer, performing local time-frequency correlation perception on the compressed feature tensor to obtain a locally enhanced feature tensor; based on the multidimensional attention decoding layer, performing multidimensional self-attention feature decoding on the locally enhanced feature tensor to obtain an attention feature tensor; and based on the inverted residual feedforward network layer, performing feature enhancement processing on the attention feature tensor to obtain the target feature tensor.

[0075] The aforementioned local perceptual unit layer can refer to the first processing module in the feature decoder, which aims to preserve the spatial location information of the input features and enhance local time-frequency correlation. It captures the time-frequency patterns of the feature tensor within a local window through local convolution and utilizes residual connections to preserve the location information of the original features, preventing the loss of location information in deep networks, and outputting a locally enhanced feature tensor.

[0076] The aforementioned multidimensional attention decoding layer can refer to the processing layer in the feature decoder that performs self-attention feature decoding of the local enhanced feature tensor in three dimensions: channel, spectrum, and time, in a serial manner. It can progressively constrain candidate orientation, distinguish event categories, and confirm event stability according to the serial order of channel, spectrum, and time, thereby adapting to complex scenarios where acoustic interaction devices experience numerous near-field interferences and weak far-field target sound.

[0077] The aforementioned inverted residual feedforward network layer can refer to a processing module in the feature decoder designed to enhance features and project channels onto the attention feature tensor after multi-dimensional attention processing. Through an inverted residual structure that first expands, then processes locally, and finally projects, the expressive power and nonlinear transformation capability of the features are enhanced. Residual connections ensure gradient flow, guaranteeing that the output target feature tensor contains rich semantic and spatial information while possessing a stable numerical distribution.

[0078] As an optional implementation, local perception, multidimensional attention, and feedforward enhancement can be performed sequentially in the feature decoder. Specifically, the local perception unit layer performs local time-frequency correlation perception on the compressed feature tensor, using deep convolution combined with residual connections to extract detailed features of the local time-frequency structure, generating a locally enhanced feature tensor while preserving positional information. The multidimensional attention decoding layer performs multidimensional self-attention decoding on the locally enhanced feature tensor, sequentially capturing the phase correlation between microphone channels to constrain orientation through a channel attention module, distinguishing event spectral texture through a spectral attention module, and confirming the stability of temporal evolution through a temporal attention module, generating attention feature tensors through serial processing. Finally, the inverted residual feedforward network layer enhances the attention feature tensor by first expanding and activating channels through convolution, then extracting local features through deep convolution, and finally projecting back to the original dimension through convolution and superimposing the original input for residual fusion, outputting the target feature tensor, ensuring the complete transmission of deep semantics and shallow features.

[0079] Through the above configuration, this hierarchical decoding structure improves the accuracy and robustness of feature extraction. The local perception unit enhances its sensitivity to local time-frequency details through deep convolution, compensating for subtle features overlooked by global attention. The serial processing mechanism of multi-dimensional attention aligns with the processing logic of acoustic signals. It first uses spatial phase to quickly lock the sound source region to suppress interference, and then combines spectral and temporal information to accurately identify events, effectively solving the problem of strong near-field noise interfering with far-field target sound. The inverted residual feedforward network further enhances the nonlinear feature representation capability through channel expansion and deep convolution, while residual connections alleviate the gradient vanishing problem in deep network training, ensuring efficient information flow. The overall structure achieves a step-by-step improvement from local details to global semantics, and from spatial constraints to temporal stability. The generated target feature tensor has higher discriminative power, providing a solid foundation for high-precision joint perception.

[0080] In this embodiment, the multidimensional attention decoding layer includes a channel attention module, a spectrum attention module, and a temporal attention module. Based on the multidimensional attention decoding layer, multidimensional self-attention feature decoding is performed on the local enhancement feature tensor to obtain an attention feature tensor, including: based on the channel attention module, performing channel-dimensional self-attention processing on the local enhancement feature tensor to obtain a channel feature tensor; based on the spectrum attention module, performing frequency-dimensional self-attention processing on the channel feature tensor to obtain a frequency feature tensor; and based on the temporal attention module, performing time-dimensional self-attention processing on the frequency feature tensor to obtain an attention feature tensor.

[0081] The aforementioned channel attention module can refer to the first processing unit in the multidimensional attention decoding layer, specifically designed to capture the correlation of feature tensors along the channel dimension, primarily serving as a preliminary constraint on spatial orientation. It utilizes spatial phase cues to pinpoint the directional range of candidate sound sources as early as possible. By focusing on channels with strong phase correlation, it suppresses interference from irrelevant channels, thereby constraining spatial location before semantic classification.

[0082] The aforementioned spectral attention module can refer to a processing unit in the multidimensional attention decoding layer used to capture the correlation of feature tensors in the frequency dimension, primarily serving to distinguish between different types of sound events. It can utilize spectral texture cues to differentiate between various sound events, such as user voices, knocking sounds, and cracking sounds, based on their differences in spectral energy distribution. By capturing harmonic structures and spectral envelope features, it enhances the ability to identify specific event categories.

[0083] The aforementioned temporal attention module refers to the processing unit in the multidimensional attention decoding layer used to capture the correlation of feature tensors in the temporal dimension, primarily serving to confirm event stability and suppress noise. It can utilize temporal evolution clues to determine whether a sound is a short-term burst or continuous. By capturing long-distance temporal dependencies, it distinguishes stable user voice interaction signals from instantaneous environmental noise, such as fan noise or electrical noise, thereby suppressing misjudgments caused by instantaneous high-energy noise and improving robustness in multi-source mixed scenarios.

[0084] As an optional implementation, sequential self-attention processing across three dimensions—channel, spectrum, and time—can be performed in the multi-dimensional attention decoding layer. Specifically, the channel attention module performs channel-dimensional self-attention calculation on the local enhancement feature tensor, treating the number of channels as the sequence length. It calculates the phase correlation weights between channels through query, key, and value matrix mapping, thereby capturing the time and phase difference information between different channels of the microphone array, generating a channel feature tensor, and initially pinpointing the sound source's directional range. The spectrum attention module performs frequency-dimensional self-attention processing on the channel feature tensor, treating the frequency index as the sequence length, analyzing the energy distribution and harmonic structure between frequency bands, distinguishing the spectral texture features of different events such as user voice and noise, and generating a frequency feature tensor. The time attention module performs time-dimensional self-attention processing on the frequency feature tensor, treating time frames as the sequence length, modeling the temporal evolution of sound, judging the persistence and stability of events, suppressing transient noise interference, and generating an attention feature tensor containing spatial, semantic, and temporal information.

[0085] Through the above setup, this serial multidimensional attention mechanism enhances perception capabilities in complex sound fields. Channel attention prioritizes spatial phase information, effectively constraining candidate sound source directions before semantic classification, addressing the problem of strong near-field interference masking far-field target sounds, and improving localization robustness. Spectral attention focuses on frequency domain features, accurately distinguishing spectral differences between different sound events, enhancing event recognition accuracy. Temporal attention focuses on temporal dynamics, effectively filtering short-term burst noise by capturing the evolution of sound, confirming it as a real event. The serial processing of the three dimensions—channel, spectrum, and time—simulates the cognitive process of first determining location, then timbre, and finally persistence, giving feature extraction clear physical meaning and logical hierarchy. The generated attention feature tensor possesses both high spatial resolution and high semantic discriminative power, providing superior feature support for subsequent high-precision joint perception.

[0086] In this embodiment of the application, based on the inverted residual feedforward network layer, the attention feature tensor is subjected to feature enhancement processing to obtain the target feature tensor, including: performing feature channel expansion processing on the attention feature tensor to obtain an expanded feature tensor; performing local feature enhancement processing on the expanded feature tensor to obtain an enhanced feature tensor; and performing residual fusion processing on the enhanced feature tensor to obtain the target feature tensor.

[0087] The aforementioned extended feature tensor refers to the intermediate feature data obtained by expanding the channel dimension of the attention feature tensor in the inverted residual feedforward network layer. By increasing the channel dimension, i.e., increasing the width of the feature, more capacity is provided for the network to learn and represent more complex local feature patterns. This expansion operation is an important feature of the inverted residual structure, aiming to perform feature mapping in a high-dimensional space when it is difficult to fully represent information in a low-dimensional space.

[0088] The aforementioned enhanced feature tensor can refer to the feature data obtained by extracting local spatial features from the expanded feature tensor and reactivating it in the inverted residual feedforward network layer. Deep convolution is used to extract more accurate local time-frequency patterns, such as specific sound source textures or transient features, in the expanded high-dimensional channel space, thereby enhancing the features. This step aims to extract more discriminative local information from the expanded features.

[0089] As an alternative implementation, the attention feature tensor can be nonlinearly enhanced using an inverted residual feedforward network. Specifically, the attention feature tensor can undergo feature channel expansion processing, using convolution to expand the number of feature channels to a multiple of the original dimension, and introducing a nonlinear transformation through a Gaussian error linear unit activation function to generate an expanded feature tensor, thereby increasing the model's expressive power. Next, local feature enhancement processing is performed on the expanded feature tensor, using depthwise convolution to extract local time-frequency correlations in the expanded channel dimension, capturing subtle acoustic patterns, and then processing again through an activation function to obtain the enhanced feature tensor. Finally, residual fusion processing is performed on the enhanced feature tensor, projecting the number of channels back to the original dimension through convolution, and then adding the projected features element-wise to the original attention feature tensor. This residual connection mechanism ensures that the enhanced features extracted from deep layers can be effectively combined with shallow information, ultimately outputting a target feature tensor containing rich local details and global semantics.

[0090] The above settings enhance the nonlinearity of feature representation and the ability to capture local details. Channel expansion increases the dimensionality of the feature space, enabling the learning of more complex nonlinear mapping relationships and improving the resolution of subtle acoustic features. Deep convolution performs local perception in the expanded dimension, effectively extracting key patterns in the local time-frequency structure and compensating for local details ignored by global attention. The residual fusion mechanism solves the gradient vanishing problem in deep networks, ensuring efficient information transmission and allowing the target feature tensor to retain the global semantics of the original attention features while incorporating enhanced local details. This design improves robustness in complex sound fields, ensuring that the generated feature tensor accurately represents the sound source location and event type, providing a feature foundation for subsequent high-precision joint perception and intelligent interactive decision-making.

[0091] In this embodiment, the feature encoder includes a convolutional block layer, a normalization layer, an activation function layer, a time-frequency domain pooling layer, and a regularization layer. Based on the feature encoder in the target feature extraction network, feature encoding is performed on the acoustic feature tensor to obtain a compressed feature tensor, including: based on the convolutional block layer, local feature extraction is performed on the acoustic feature tensor to obtain an intermediate feature tensor; based on the normalization layer, the intermediate feature tensor is normalized to obtain a normalized feature tensor; based on the activation function layer, the normalized feature tensor is transformed to obtain an activated feature tensor; based on the time-frequency domain pooling layer, the activated feature tensor is compressed in the time-frequency domain to obtain a dimensionality-reduced feature tensor; and based on the regularization layer, the dimensionality-reduced feature tensor is regularized to obtain a compressed feature tensor.

[0092] The aforementioned convolutional block layer can refer to a component in a feature encoder used to extract local spatiotemporal features from the original acoustic feature tensor. It can be composed of multiple stacked convolutional operations, scanning the input data in the time and frequency dimensions through a sliding window to capture local acoustic patterns, such as specific spectral textures or phase patterns.

[0093] The aforementioned normalization layer refers to a component applied after the output of convolutional block layers to stabilize the neural network training process and accelerate convergence. Standardizing intermediate feature tensors, such as subtracting the mean and dividing by the standard deviation, addresses internal covariate shift issues, making the data distribution more stable and thus improving the efficiency and stability of model training.

[0094] The activation function layer mentioned above can refer to a component that introduces nonlinear transformation capabilities, enabling the neural network to fit complex nonlinear acoustic feature mapping relationships. Performing element-wise nonlinear transformations on the normalized feature tensor, such as setting negative values ​​to zero while keeping positive values ​​unchanged, increases the model's expressive power, allowing it to distinguish more complex sound features.

[0095] The aforementioned time-frequency domain pooling layer refers to a component used for downsampling in both time and frequency dimensions, aiming to reduce data volume and expand the receptive field. Time-frequency domain compression is performed on the activation feature tensor. For example, the number of time frames is compressed from 250 frames to 125 frames, and the Mel frequency dimension is compressed from 64 dimensions to 8 dimensions. This reduces the computational burden on subsequent networks and also extracts more robust global statistical features.

[0096] The aforementioned regularization layer can refer to a component used to prevent model overfitting and improve generalization ability. During training, a portion of neurons or outputs are randomly discarded, forcing the network to learn more robust feature representations and avoiding excessive reliance on specific feature channels.

[0097] As an alternative implementation, the acoustic feature tensor can be processed stepwise by a feature encoder to generate a compressed feature tensor. Specifically, convolutional block layers can use convolutional kernels to extract local features from the acoustic feature tensor, capturing local correlations in the time-frequency domain and generating intermediate feature tensors. Subsequently, a normalization layer, such as batch normalization, standardizes the intermediate feature tensors, eliminating distribution bias and accelerating convergence to obtain a normalized feature tensor. Next, an activation function layer can perform a nonlinear transformation on the normalized feature tensor, introducing nonlinear expressive power to obtain an activation feature tensor. Then, a time-frequency pooling layer downsamples the features in the time and frequency dimensions, reducing the spatial resolution and data volume of the features to obtain a dimensionality-reduced feature tensor. Finally, a regularization layer can regularize the dimensionality-reduced feature tensor to suppress overfitting, ensure the model's generalization ability, and ultimately output a low-dimensional compressed feature tensor rich in important information, providing efficient input for subsequent decoding stages.

[0098] Through the above configuration, this hierarchical coding structure improves the efficiency and robustness of feature extraction. Convolutional block layers effectively capture the local time-frequency patterns of the acoustic signal, while normalization and activation layers ensure the network's stability and non-linear expressive power. Time-frequency domain pooling layers reduce computational complexity and memory usage through dimensionality reduction, while preserving key acoustic feature information, allowing the network to focus on global semantics rather than local noise. Regularization layers further enhance the model's generalization ability, preventing overfitting on the training data. Overall, this encoder compresses high-dimensional, redundant raw acoustic features into low-dimensional, compact feature representations, preserving key information related to localization and recognition, while reducing the computational burden of subsequent multi-dimensional attention decoding layers. This achieves efficient and robust feature preprocessing, providing a data foundation for joint perception.

[0099] In this embodiment, the original sound field signal is preprocessed to obtain an acoustic feature tensor, including: performing time-frequency domain transformation on the original sound field signal to obtain complex frequency domain signals for each microphone channel; mapping the complex frequency domain signals using a Mel filter bank to obtain Mel spectral features; performing a correlation phase transformation on the complex frequency domain signals to obtain correlation phase transformation features; and fusing the Mel spectral features and correlation phase transformation features to obtain an acoustic feature tensor. Preferably, fusing the Mel spectral features and correlation phase transformation features to obtain an acoustic feature tensor includes: concatenating the Mel spectral features and correlation phase transformation features along the feature channel dimension to obtain the acoustic feature tensor.

[0100] The aforementioned complex frequency domain signal can refer to the frequency domain representation obtained by transforming the original audio signal in the time domain through time-frequency transformation, such as short-time Fourier transform. This preserves the complete spectral information of the sound, including energy distribution (amplitude) and time delay clues (phase) of the signal reaching different microphones, providing basic data for subsequent extraction of semantic and spatial features.

[0101] The aforementioned Mel filter bank can refer to a set of linear bandpass filters used to simulate auditory characteristics. It maps the linear frequency axis of a complex frequency domain signal onto the Mel frequency axis. This process simulates the nonlinear perception of frequency by hearing, helping to extract acoustic features that better conform to the laws of human auditory perception, serving as a tool for constructing Mel spectroscopic features.

[0102] The aforementioned Mel spectrogram features refer to the logarithmic Mel spectrogram obtained by mapping the amplitude spectrum and aggregating the energy of a complex frequency domain signal using a Mel filter bank. Mel spectrogram features can characterize the semantic features and spectral texture of sound, reflecting the frequency component distribution of sound, and can be used to distinguish different types of sound events, such as user voices, music, and noise.

[0103] The aforementioned related phase transform features refer to feature maps calculated based on the generalized cross-correlation phase transform algorithm, used to characterize the time delay information between microphone channel pairs. These features can characterize the spatial characteristics and phase cues of sound, using the phase difference between signals received by different microphones to estimate the sound source's location, thus serving as spatial cues for sound source localization.

[0104] The aforementioned feature channel dimension can refer to the dimension in a multidimensional tensor used to distinguish different feature types or different signal sources. Integrating semantic and spatial information into the same tensor forms a unified input representation, enabling subsequent networks to simultaneously utilize this semantic and spatial information for joint perception.

[0105] As an optional implementation, the raw sound field signal acquired by the microphone array can be transformed in the time-frequency domain. A short-time Fourier transform converts the time-domain signal into a complex frequency-domain signal for each channel, preserving amplitude and phase information. Next, a Mel filter bank is used to map the amplitude spectrum of the complex frequency-domain signal, calculate the logarithmic energy, and generate Mel spectroscopic features characterizing the semantics and spectral texture of the sound. Furthermore, a generalized cross-correlation phase transform can be performed on the complex frequency-domain signal to calculate the cross-correlation function between different microphone channel pairs, preserving phase information to eliminate amplitude effects and generating a correlation phase transform feature characterizing the arrival time difference and phase difference of the sound source. This correlation phase transform feature preserves the index relationship of the microphone channel pairs. Finally, the Mel spectroscopic features and the correlation phase transform features can be concatenated and fused along the feature channel dimension to form a joint acoustic feature tensor containing spectral semantic information and spatial phase information.

[0106] By fusing Mel-spectral signatures and related phase transform features, complementary and enhanced semantic and spatial localization information of sound is achieved. Mel-spectral signatures effectively characterize the content and category of sound, while related phase transform features are specifically designed for improving sound source location estimation and exhibit good robustness to reverberation and white noise. Concatenation along the channel dimension allows the network to learn the characteristics of both Mel-spectral signatures and related phase transform features separately, enabling flexible utilization in subsequent multi-dimensional attention processing. This dual-stream feature construction method avoids the spatial information loss problem caused by using only time-frequency maps and overcomes the insufficient semantic information when using only phase information. By preserving the channel pair structure of related phase transform features, spatial constraints can be introduced in the early stages, improving localization accuracy and event recognition accuracy in multi-source scenarios, providing a high-quality feature foundation for intelligent interaction in acoustic interactive devices.

[0107] In this embodiment of the application, the sound source analysis result of the original sound field signal is determined based on the target feature tensor, including: parsing the target feature tensor to obtain the sound source location corresponding to at least one sound source signal and the confidence level of the sound event type corresponding to at least one sound source signal; determining the target sound event type corresponding to at least one sound source signal based on the confidence level of the sound event type corresponding to at least one sound source signal; and integrating the sound source location and the corresponding target sound event type corresponding to at least one sound source signal to obtain the sound source analysis result.

[0108] The confidence level of the aforementioned sound event type can refer to a numerical index used to determine whether a preset type of sound event truly exists within a specific time segment, and the magnitude of its existence. In this application, the confidence level of the sound event type is obtained by parsing the specific vector mathematical attributes output by the network. By using the vector magnitude as the confidence level, sound source localization and sound event detection can be completed in parallel during the same forward propagation process, avoiding the delay and error accumulation caused by localization followed by classification, and improving the real-time performance and robustness of the system.

[0109] As an optional implementation, the target feature tensor can be simultaneously parsed to extract the sound source location and event confidence. Specifically, the target feature tensor can be parsed into a three-dimensional vector for each preset type of sound event. Next, the magnitude of this three-dimensional vector is calculated, and this magnitude is used as a confidence index for the existence of the sound event; if the magnitude exceeds a preset threshold, the event is determined to exist, and the event with higher confidence is selected as the target sound event type. The sound source location is then calculated based on the direction of the three-dimensional vector, and the azimuth and elevation angles are calculated using the arctangent function, directly mapping them to spatial coordinates. Finally, the calculated sound source location is associated and integrated with the determined target sound event type to form a sound source analysis result containing a binary pair of location and category. This process achieves the unification of localization and recognition in mathematical space, eliminating the need for additional post-processing matching steps and directly outputting the sound source analysis result.

[0110] Through the above settings, this joint analytical mechanism based on vector magnitude and direction achieves efficient synchronization and inherent unity between sound source localization and event detection. By uniformly using three-dimensional vectors to simultaneously represent orientation and confidence, it avoids feature fragmentation and delay issues caused by the separation of localization and recognition modules, thus improving the system's real-time performance. Using the magnitude of the target feature tensor to reflect the intensity of event activity allows for flexible threshold setting to filter false detections, improving the reliability of recognition; using the direction of the target feature tensor to provide spatial information eliminates the need for complex angle search algorithms, reducing computational complexity. This design enables acoustic interaction devices to simultaneously know what sounds are present and where they originate, providing accurate, consistent, and immediate perceptual basis for subsequent intelligent interaction decisions based on event priority, enhancing the response speed and naturalness of acoustic interaction devices in complex environments.

[0111] In this embodiment of the application, parsing the target feature tensor to obtain the sound source location corresponding to at least one sound source signal and the confidence level of the sound event type corresponding to at least one sound source signal includes: parsing the target feature tensor to obtain the vector direction and vector length of the target feature tensor; determining the sound source location corresponding to at least one sound source signal based on the vector direction of the target feature tensor; and determining the confidence level of the sound event type corresponding to at least one sound source signal based on the vector length of the target feature tensor.

[0112] The aforementioned vector direction can refer to the direction of the three-dimensional vector output by the target feature extraction network in space, corresponding to the spatial orientation of the sound source relative to the microphone array. This directional information is used to determine the origin of the sound and drives the hardware actions of the acoustic interaction device, such as controlling the robot's head or body to turn in that direction for subsequent beamforming sound pickup or camera alignment.

[0113] The aforementioned vector length can refer to the magnitude of the three-dimensional vector output by the target feature extraction network. This vector magnitude quantifies the intensity or probability of a specific sound event existing in the sound field. It is used to determine what sound was heard and whether the sound is worthy of attention. By comparing the vector length with a preset threshold, it is determined whether to trigger the corresponding interaction strategy, such as wake-up, dialogue, alarm, or suppression.

[0114] As an optional implementation, geometric analysis can be performed on the target feature tensor to extract the sound source location and event confidence. Specifically, the three-dimensional vector corresponding to each type of sound event in the target feature tensor can be regarded as a direction vector in space. To determine the sound source location, the azimuth angle is obtained by calculating the projection angle onto the horizontal plane using the direction information of this vector, and the pitch angle is obtained by calculating the angle with the horizontal plane, thereby accurately calculating the specific location of the sound source in three-dimensional space. To determine the confidence of the sound event type, the magnitude of the three-dimensional vector is calculated, and the calculated value is used as the probability or intensity index of the event's existence. If the magnitude exceeds a preset threshold, the sound event is determined to exist, and the numerical value reflects the system's degree of confidence in the event category. Through this analytical method, the output three-dimensional vector simultaneously carries spatial location information and semantic confidence information, achieving simultaneous acquisition of localization and recognition results.

[0115] Through the above settings, this joint analysis method based on vector magnitude and direction simplifies the post-processing workflow and improves the system's real-time performance. It eliminates the need for independent angle searches or classification judgments, instead integrating the positioning direction and detection magnitude into a single vector through a unified mathematical representation, avoiding computational delays caused by multi-step processing. Based on vector direction mapping of spatial orientation, no additional coordinate transformations or matching algorithms are required; and based on vector magnitude quantification of event activity strength, an intuitive confidence index is provided, facilitating subsequent strategy judgment. This design reduces algorithm complexity and ensures inherent consistency between positioning accuracy and recognition accuracy, enabling acoustic interaction devices to respond instantly to changes in sound source location and content attributes, improving the smoothness and response speed of interaction in companionship scenarios.

[0116] In this embodiment, controlling the acoustic interaction device to perform a response operation corresponding to the sound source analysis result includes: when at least one sound source signal is multiple sound source signals, obtaining the response priority of the target sound event type corresponding to the multiple sound source signals; determining the response operation corresponding to the multiple sound source signals based on the target sound event type corresponding to the multiple sound source signals; and controlling the acoustic interaction device to sequentially execute the response operation corresponding to the multiple sound source signals based on the response priority. Preferably, the target sound event type includes: security event type, user voice interaction event type, and environmental interference event type, wherein the response priority corresponding to the security event type is greater than the response priority corresponding to the user voice interaction event type, and the response priority corresponding to the user voice interaction event type is greater than the response priority corresponding to the environmental interference event type.

[0117] The aforementioned response priority can refer to a pre-defined hierarchy of urgency and execution order based on the identified sound event categories. This ensures that acoustic interaction devices can rationally allocate limited hardware resources, such as cameras, speakers, steering servos, and computing resources, in multi-sound-source mixed scenarios, prioritizing important tasks and avoiding ineffective interactions or resource waste caused by responding to every sound source.

[0118] The aforementioned security event types can refer to sound events that may threaten the user's personal safety or property security, requiring the acoustic interaction device to immediately take alert, alarm, or monitoring measures. Examples include the sound of breaking glass, screams, the sound of heavy objects falling, and alarm sounds.

[0119] The aforementioned user voice interaction event types can refer to sound events initiated by the user to communicate with or input commands to a smart interactive device. Examples include wake words, everyday conversational voices, and command voices.

[0120] The aforementioned types of environmental interference events can refer to sound events that are not intentionally emitted by the user, exist as background noise, and do not require active interaction or response from the intelligent interactive device. Examples include background noise from a television, fan noise, air conditioner noise, the sound of a robot vacuum cleaner operating, and traffic noise outside the window.

[0121] As an optional implementation, when multiple sound source signals are detected simultaneously, the target sound event type corresponding to each signal can be obtained, and response priorities can be assigned according to preset rules. The priority ranking rule can be set as follows: security event types are higher, user voice interaction event types are next, and environmental interference event types are lower. Subsequently, specific response operations are determined based on the event type. For example, a security event triggers an alarm, redirects the sound source, and starts camera recording; a user voice event triggers redirection of the sound source, beamforming pickup, and voice interaction; and an environmental interference event performs signal suppression or ignores the signal. Finally, these operations can be executed sequentially according to the response priority. For example, if user voice and television sound are detected simultaneously, redirection and pickup of the user voice direction are prioritized, while nulling is applied to the television sound direction to suppress interference; if a security event is detected, other operations are immediately interrupted, and alarm and patrol actions are prioritized to ensure that the security response is not interfered with by other low-priority events.

[0122] Through the above settings, this priority-based hierarchical response mechanism enhances the intelligent decision-making capabilities and safety of the companion agent in complex, multi-sound-source environments. By clearly distinguishing the weights of three types of events—safety, interaction, and interference—it avoids blindly responding to every sound, effectively filters background noise, and reduces unnecessary hardware wear and energy consumption. The high-priority safety event handling mechanism ensures rapid response and proactive security protection when danger occurs; the priority processing of user voice interactions guarantees the timeliness and fluency of user communication; and the suppression of environmental interference improves noise immunity. This closed-loop control strategy enables the agent not only to hear sounds but also to understand their semantics and importance, thereby achieving a more natural, efficient, and safe intelligent interaction experience.

[0123] In this embodiment of the application, the method further includes: when the target sound event type is a security event type, guiding the acoustic interaction device to monitor the corresponding security event based on the location of the corresponding sound source, and performing an alarm operation when the danger level of the security event is greater than a preset danger level; when the target sound event type is a user voice interaction event type, controlling the microphone array to turn to the location of the corresponding sound source to enhance user voice interaction; and when the target sound event type is an environmental interference event type, suppressing the sound source signal at the location of the corresponding sound source.

[0124] The aforementioned security incidents can refer to sound events that indicate an abnormality, danger, or intrusion in the home environment, requiring timely intervention from acoustic interaction devices for monitoring or alarm.

[0125] The aforementioned preset risk level can refer to a numerical indicator used to quantify the severity of a security incident, thereby determining whether to trigger a formal alarm.

[0126] The alarm operation mentioned above can refer to a series of proactive defense and notification actions performed by the acoustic interaction device when the confidence level of a security event exceeds a preset danger level threshold.

[0127] As an optional implementation, differentiated control can be performed based on the sound source analysis results. If a security event is identified, the acoustic interaction device can control the head to turn towards the sound source, activate the camera for visual monitoring, and combine this with audio confidence levels to determine the degree of danger of the security event; if the degree of danger of the security event is greater than a preset degree, an alarm is triggered and the user is notified. If a user voice interaction event is identified, the microphone array is controlled to turn towards the sound source, utilizing the turning to enhance signal reception, and then beamforming is performed in that direction to improve the sound pickup effect, initiating a voice wake-up or dialogue process. If an environmental interference event is identified, a null can be set in that direction in the beamforming algorithm to actively suppress noise signals in that direction, preventing false wake-ups and thus keeping the system silent or only listening to other valid sound sources.

[0128] Through the above settings, the event-type-based closed-loop control strategy achieves accurate linkage between perception and action. The proactive monitoring and tiered alarm mechanism for security events fills the security gaps in the companion intelligent agent, improving home safety. Automatic redirection and beamforming in user voice interaction work together to improve far-field sound pickup quality and increase speech recognition rate. Proactive suppression of environmental interference avoids ineffective interactions and resource waste. This enables the acoustic interaction device to adapt to different scenarios, providing a natural and smooth interactive experience while ensuring safety, thus enhancing the system's practicality and user satisfaction.

[0129] Figure 2 This is a schematic diagram of the network structure of a target feature extraction network according to an embodiment of the present invention, such as... Figure 2 As shown, the target feature extraction network takes an acoustic feature tensor as input and outputs a target feature tensor. The network includes a feature encoder and a feature decoder. The feature encoder consists of a series of interconnected convolutional block layers, a normalization layer, an activation function layer, a time-frequency pooling layer, and a regularization layer. The feature decoder consists of a series of interconnected local perceptual unit layers, a multidimensional attention decoding layer, and an inverted residual feedforward network layer.

[0130] Figure 3 This is a schematic diagram of the control process of an acoustic interaction device according to an embodiment of the present invention, such as... Figure 3 As shown, firstly, the raw sound field signal is acquired through a microphone array in the acoustic interaction device. Next, the raw sound field signal is preprocessed to obtain an acoustic feature tensor. Based on the feature encoder in the target feature extraction network, the acoustic feature tensor is feature-encoded to obtain a compressed feature tensor. Based on the feature decoder in the target feature extraction network, the compressed feature tensor is subjected to multi-dimensional attention feature decoding to obtain the target feature tensor, where multi-dimensional attention includes channel attention, spectral attention, and temporal attention. Then, based on the target feature tensor, the sound source analysis result of the raw sound field signal is determined. Finally, the acoustic interaction device is controlled to execute the response operation corresponding to the sound source analysis result.

[0131] Figure 4 This is a schematic diagram of the control process of an optional acoustic interaction device according to an embodiment of the present invention, such as... Figure 4 As shown, firstly, the raw sound field signal is acquired through a microphone array in the acoustic interaction device. Next, the raw sound field signal is preprocessed to obtain an acoustic feature tensor. Based on the feature encoder in the target feature extraction network, the acoustic feature tensor is feature-encoded to obtain a compressed feature tensor. Based on the feature decoder in the target feature extraction network, the compressed feature tensor is subjected to multi-dimensional attention feature decoding to obtain the target feature tensor, where multi-dimensional attention includes channel attention, spectral attention, and temporal attention. Next, the target feature tensor is parsed to obtain its vector direction and vector length. Based on the vector direction of the target feature tensor, the sound source location corresponding to at least one sound source signal is determined. Based on the vector length of the target feature tensor, the target sound event type corresponding to at least one sound source signal is determined. Finally, the acoustic interaction device is controlled to execute a response operation corresponding to the sound source analysis results.

[0132] Figure 5 This is a schematic diagram of the control process of another optional acoustic interaction device according to an embodiment of the present invention, such as... Figure 5As shown, firstly, the raw sound field signal is acquired through a microphone array in the acoustic interaction device. Next, the raw sound field signal is preprocessed to obtain an acoustic feature tensor. Based on the feature encoder in the target feature extraction network, the acoustic feature tensor is feature-encoded to obtain a compressed feature tensor. Based on the feature decoder in the target feature extraction network, the compressed feature tensor is subjected to multi-dimensional attention feature decoding to obtain a target feature tensor, where multi-dimensional attention includes channel attention, spectral attention, and temporal attention. Next, the target feature tensor is parsed to obtain its vector direction and length. Based on the vector direction of the target feature tensor, the sound source location corresponding to at least one sound source signal is determined. Based on the vector length of the target feature tensor, the target sound event type corresponding to at least one sound source signal is determined. Finally, when at least one sound source signal is multiple sound source signals, the response priority of the target sound event types corresponding to the multiple sound source signals is obtained. Based on the target sound event types corresponding to the multiple sound source signals, the response operation corresponding to the multiple sound source signals is determined. Based on the response priority, the acoustic interaction device is controlled to sequentially execute the response operations corresponding to the multiple sound source signals.

[0133] The technical solution proposed in this application is described below with reference to an optional embodiment. This application proposes a method for joint sound source localization and sound event perception for companion intelligent agents. It relates to the fields of human-computer interaction and machine hearing, primarily concerning companion entities such as companion home service robots and smart speakers. This method utilizes deep neural networks to simultaneously perform interactive joint perception in complex sound field environments, enabling simultaneous sound source localization and sound event detection. This application addresses the problem of blind interaction and weak anti-interference capabilities caused by distinguishing valuable interactive sound sources from environmental noise. The proposed spatiotemporal-frequency joint perception method allows the intelligent agent to calculate the location and type of sound after hearing it, thereby achieving event-based intelligent selective interaction.

[0134] This application provides a sound source localization and sound event joint perception method for companion agents, which overcomes the limitations of related technologies that only process localization and recognition in separate steps, and also breaks down the limitation of only localization without recognition. It relies on deep learning methods to extract multi-dimensional features of spatiotemporal sound signals, integrating localization and recognition processing into a consistent mathematical vector space, and simultaneously calculating the sound source location and sound type identification. The specific technical solution is as follows: First, environmental sound field data is acquired and features are constructed using a microphone array on the companion agent. Semantic and spatial dual-stream features are employed. Log-Mel spectrograms are used to express the spectral texture and semantic category clues of the sound, while generalized cross-correlation phase transform maps are used to express the arrival time difference and phase difference clues between microphone channel pairs. This application retains the channel pair index relationship of the generalized cross-correlation phase transform map during feature construction and fuses them step-by-step in the subsequent network according to the order of spatial phase, spectral texture, and temporal evolution. Spatial clues first participate in candidate direction constraints, spectral clues then participate in event category differentiation, and temporal clues finally participate in stability confirmation, thereby forming a joint feature tensor suitable for the interaction scenario of the companion agent. Regarding the design of the perception network, this application designs a spatial, frequency, and temporal sequential three-dimensional joint feature extraction network. This structure first utilizes a spatial channel phase attention module to process the minute phase delays between different channels of the microphone array, locking in the candidate directional range as early as possible; then, a frequency texture attention module distinguishes the differences in spectral energy distribution between different events such as user voices, knocking sounds, and breaking sounds; finally, a temporal evolution attention module determines whether the sound is a short-term burst or continuous, suppressing misjudgments caused by transient noise. Using this sequential order, in actual operation, the accompanying agent often encounters situations where near-field interference sounds are louder than the target user's voice. If semantic classification is performed first and spatial phase information is introduced later, the network is easily swayed by high-energy interference. This application advances the spatial phase constraint, allowing semantic judgment to be completed within the candidate direction range, thereby improving stability in multi-source mixed scenarios. Furthermore, regarding the design of the output representation, for a defined type of sound, such as user voices, knocking sounds, and alarm sounds, this application outputs a three-dimensional spatial vector. The vector direction represents the azimuth angle of the sound, and the vector length is uniformly defined as the event activity confidence or activity intensity. This confidence level is used to determine whether the sound type truly exists and is correlated with subsequent interaction strategies, alarm levels, and device action control. Through this unified definition, the detection and localization results are mathematically consistent, avoiding the disconnect between locating the sound but not knowing what it is, or knowing the event category but being unable to drive the device to change direction. Finally, during the execution of the interaction strategy, the agent executes different strategies based on the output of this synchronization result. A priority mapping relationship can also be established based on event categories, mapping dangerous events, user voice interaction events, and ordinary interference events to different response levels, ensuring that dangerous events trigger alarms first, user voice events trigger dialogue, and ordinary interference events enter a suppression or ignore process.It can automatically exclude sound source signals from the environment, such as television sounds and fan noises, and output wake-up and dialogue logic based on the location of such user voices, or output alarm and patrol logic based on the location of dangerous events, demonstrating proactive interaction with semantic understanding. For safety events such as broken glass and screams, two alarm conditions can be set: single-window high-confidence triggering and multi-window continuous triggering. When the activity of an event in a single analysis window exceeds a higher alarm threshold, or when the activity of an event in multiple consecutive analysis windows exceeds a lower continuous threshold, a safety alarm response can be triggered.

[0135] The specific execution process of the method provided in this application embodiment is as follows: Environmental sound field acquisition and joint feature preprocessing are performed to convert the physical acoustic information into tensors with time, location, and frequency band meanings, which are then input into the model. Array acquisition is performed by equipping the companion agent with a microphone array, the number of which is denoted as M (e.g., M=6), to acquire sound information of ongoing activities in the field, with the sampling frequency set to... (e.g., 16kHz) Acquire continuous audio streams Short-time Fourier transform processing is performed, and the continuous audio stream is segmented by windowing, such as with a window length of 0.04s and a frame shift of 0.02s, to transform the time-domain audio stream from the time domain to the complex frequency band. Semantic feature extraction is performed, mapping the spectral amplitude to a Mel scale. The logarithm of the filters is calculated for F Mel filters (e.g., F=64), resulting in a log-Mel spectrogram, which serves as the semantic description of the sound, i.e., the semantic feature stream, representing the pitch characteristics of the sound. Spatial feature extraction is then performed. To further obtain the location and time of the sound source, the generalized cross-correlation phase transform between each channel of each microphone is calculated. Next, feature fusion is performed. The log-Mel spectrogram and the correlation phase transform feature map are aligned according to time frame and frequency index, preserving the index relationship of each microphone channel in the correlation phase transform feature. These are combined into a fused input tensor, i.e., the acoustic feature tensor. The shape of the input tensor can be represented as... T represents the number of time frames, and F represents the Mel frequency dimension. M is the number of microphones; if M=4, then =7, if M=6, then =16. To stably form a two-dimensional orientation estimate, the number of microphones M is usually not less than 3; under the common array size and computational resource conditions of companion agents, the number of microphones M can be 3 to 8, with 4 to 6 generally being better. When the number of microphones M increases, the number of microphone channel pairs increases by M(M + 1) / 2, which provides more spatial phase difference and mutual information, which is beneficial to improving positioning accuracy and robustness. This also increases the number of relevant phase transformation feature channels and the computational load of the subsequent neural network. Therefore, for robots with limited resources, 4 or 6 microphones can be used preferentially. Next, the network fuses this input tensor hierarchically in space according to spatial phase, spectral texture, and temporal variation.

[0136] Next, multi-dimensional attention joint feature extraction is performed. The input tensor is fed into the neural network, and deeply coupled features are obtained through the encoder and decoder. First, shallow feature encoding is performed, with the input tensor passing through multiple stacked convolutional blocks. A set of feasible options is: the speech sampling rate is set to 16kHz, the analysis window to 40ms, the analysis frame length to 20ms, the length T of the 5s audio is 250, and the number of Mel filters F is 64. If a 4-microphone array is used, the number of relevant phase transform feature channel pairs is 6, and the number of input feature channels is... The number is 7; if a 6-microphone array is used, the number of correlated phase transform feature channel pairs is 15, and the number of input feature channels is 7. The value is 16. The convolutional block can be set to 3 layers, each using a 3×3 convolutional kernel. The number of feature channels in the convolutional output is set to 64, 128, and 128 respectively, and activated by batch normalization and a linear rectified function. The pooling kernel can be set to 1×2, 1×2, and 2×2 respectively, reducing the filter scale from F=64 to F'=8, the time dimension from T=250 to T'=125, and finally the number of feature channels C becomes 128. The number of stacked layers N of the subsequent 3D attention module can be 3 to 6, preferably 4; these parameters can be selected within the range of 3 to 6 microphones based on the number of microphones and computing power.

[0137] Next, the 3D attention module performs deep feature decoding, and the encoded features are... The input is fed into N stacked 3D attention modules. The operation within each 3D attention module is that the local perceptual unit embeds the input feature tensor. By using local perceptual units, the feature tensor is perceived for local time-frequency correlation and its positional information is preserved. This operation employs depthwise convolution and residual connections, and can be expressed by the following formula:

[0138] ;

[0139] in, This indicates a depthwise convolution operation with a kernel size of 3×3.

[0140] A multi-dimensional, multi-head self-attention mechanism is employed. Specifically, attention is calculated sequentially across the three dimensions of channel, spectrum, and time for each feature. The multi-head self-attention mechanism is used for each dimension. Assuming the input is a feature sequence X, the computation process of the multi-head self-attention mechanism is as follows: a linear mapping is performed between the query, key, and value, linearly mapping the input to different subspaces. For the h-th attention head, there is a linear mapping matrix. Calculate the query, key, and value matrix:

[0141] ;

[0142] To perform scaled dot product attention, calculate the dot product of the key transpose and the query, scale and normalize to generate the attention matrix, then multiply it by the value matrix. The calculation formula can be expressed as follows:

[0143] ;

[0144] The above formula represents the inner product similarity of features, adaptively capturing long-distance correlations, where, The feature dimensions of the key matrix and query matrix are represented.

[0145] Next, the outputs of each of the H heads are stitched together based on multi-head splicing and linear projection, and then the output projection matrix is ​​used to perform the splicing. Fusion Information:

[0146] ;

[0147] in, .

[0148] Next, serial processing of 3D attention is performed. This application adopts a serial processing order of channel, spectrum, and time, and achieves feature extraction of different dimensions through tensor dimension permutation: channel attention processing, the input tensor is reshaped into a sequence with channel dimension C as the sequence length, the phase correlation matrix between channels is calculated, and the output is denoted as... Spectral attention processing will The dimension permutation, with frequency dimension F as the sequence length, captures the harmonic structure, and the output is denoted as... Time attention will The dimension permutation, with time dimension T as the sequence length, captures the temporal evolution pattern, and the output is denoted as... .

[0149] Next, the output of the attention layer of the inverted residual feedforward network is normalized and then fed into the inverted residual feedforward network. This inverted residual feedforward network introduces depthwise convolution to enhance local feature extraction capabilities. The calculation steps are as follows: First, the feature channels are expanded by a 1×1 convolution, for example, by a factor of 4, and then activated by a Gaussian error linear unit. Second, a 3×3 depthwise convolution is performed on the expanded dimension to extract spatial features. Finally, the number of channels is projected back to the original dimension by a 1×1 convolution, and residual connections are superimposed. After N layers of superposition processing, the network outputs a deeply coupled target feature tensor.

[0150] Next, the vector magnitude of the output target feature tensor can be used to determine the confidence level of the event. The vector direction of the target feature tensor is used to calculate the direction of the coupled sound source, and the coupled Cartesian coordinates of the multi-source activity output by the network are analyzed. Synchronous parsing logic is designed to perform event detection, calculate the magnitude of the vector as the confidence level of the sound event type, and perform sound source localization, using the direction of this vector as the sound source orientation, without the need for additional matching steps.

[0151] Finally, interaction strategy control is implemented based on the recognition results. Before executing the interaction strategy, the priority of the recognized sound event categories is set. The priority can be set as follows: safety events have higher priority, human voice interaction events have lower priority, and interfering environmental events have lower priority. If safety events such as the sound of breaking glass or screams and user voice events are perceived simultaneously, the alarm strategy is executed with priority for safety events. If user voices and interfering events such as television sounds or fan sounds are perceived simultaneously, the interaction direction is preserved based on user voices, while these interfering directions are suppressed. When only interfering environmental events are detected, and no user voices or safety events are perceived, wake-up interaction is not executed, and the interfering sound source signals are suppressed. Based on the recognition results, wake-up interaction, beamforming suppression, or safety alarms are performed respectively. For interactive event responses, such as user voices, if the decoding result is user voices, the direction of the sound source pointed to by the decoded vector is obtained. The accompanying intelligent agent controls the corresponding head servo head to point to the direction of the sound source, and then beamforming is performed to pick up the sound source, entering the voice wake-up and voice dialogue process. Interference events, such as television sound, are suppressed. If the analysis result is television sound or fan sound, it is considered a noise event, and the specific location of the sound source has been calculated, but it is not a valid target. In the signal processing link, a null is added to the location of the sound source to actively suppress the signal in that direction to an imperceptible level, thus suppressing wake-up triggers.

[0152] When triggering safety event alarms, such as shattering sounds, and the analysis result is broken glass or a scream, two methods are used for judgment: single-window high-confidence triggering and multi-window continuous triggering. The first path is single-window high-confidence triggering, where if the activity level of this type of event in any analysis window is greater than the first alarm threshold, the safety event is considered to have high confidence, and a high-level interruption is immediately issued. The second path is multi-window continuous triggering, where if the activity level of this type of event in multiple consecutive analysis windows is greater than the second alarm threshold, the safety event is considered to have a slightly lower single-window confidence, but it persists over time, and an alarm is still triggered. The second alarm threshold is lower than the first alarm threshold to accommodate both short, high-confidence hazard sounds and continuous, medium-confidence hazard sounds. The analysis window in the alarm judgment is an event judgment time window composed of multiple consecutive short-time Fourier transform frames. Taking a window length of 0.04s and a frame shift of 0.02s as an example, if an analysis window takes 50 consecutive short-time Fourier transform frames, the actual audio duration covered by the analysis window is approximately 0.04s + (50-1) × 0.02s = 1.02s; if an analysis window takes 100 consecutive short-time Fourier transform frames, the actual audio duration covered is approximately 0.04s + (100-1) × 0.02s = 2.02s. Three consecutive analysis windows can be arranged in a non-overlapping manner or in a partially overlapping sliding arrangement; in the non-overlapping case, three consecutive analysis windows of approximately 1.02s correspond to an observation interval of approximately 3.06s. After triggering, the agent directly turns towards the sound source, turns on the camera, records video, and provides an alarm message to notify the user.

[0153] The beneficial effects of this application include: achieving a leap in auditory perception, realizing consistent auditory perception for both localization and recognition, endowing the companion agent with the ability to recognize surrounding sounds, avoiding in-place reactions to noise, and improving the accuracy and usability of interaction; and realizing the principle of auditory discrimination in multi-sound source problems corresponding to target constraints. Class information and location information are contained in the same output vector, binding events and locations together after output, allowing the agent to distinguish different sound sources from different angles, enabling accurate real-time noise-resistant interaction; improving the real-time performance of the system, with localization and detection completed in parallel during the forward propagation of the same neural network, avoiding the real-time latency caused by the gradual intermediate process compared to the traditional two-step synthesis system of localization followed by recognition, and better meeting the real-time requirements of companion scenarios. A semantic and spatial dual-stream feature construction method is proposed for companion agent scenarios. This application retains the microphone channel pair structure of generalized cross-correlation phase transformation and participates in candidate orientation constraints in the early stages of the network, then gradually fuses it with Mel spectrogram features. This paper proposes a sequential three-dimensional attention processing order encompassing space, frequency, and time. This order first uses spatial phase information to limit the directional range, then uses spectral texture to distinguish event types, and finally uses temporal evolution to confirm event stability, thus addressing the problem that strong near-field interference can easily overwhelm far-field target sound. Based on the known output representation of multi-source active coupling Cartesian sound sources, it adds confidence interpretation, event priority mapping, and action control rules for companion agent scenarios, enabling the output results to directly drive subsequent behaviors such as turning, camera access, alarms, and dialogue. A closed-loop process is established between sound source localization, sound event recognition, and agent interaction control. In addition to outputting angle and category, it adopts different response strategies based on different scenarios such as user voice interaction, dangerous events, and ordinary ambient sounds, thus distinguishing itself from acoustic perception models that only provide offline recognition results.

[0154] According to another aspect of the embodiments of this application, a control device for an acoustic interaction device is also provided. This device can execute the control method of the acoustic interaction device in the above embodiments. The specific implementation method and preferred application scenarios are the same as those in the above embodiments, and will not be repeated here.

[0155] Figure 6 This is a schematic diagram of a control device for an acoustic interactive device according to an embodiment of this application, such as... Figure 6 As shown, the device includes the following: acquisition module 602, extraction module 604, determination module 606, and execution module 608.

[0156] The acoustic interaction device includes a data acquisition module 602 for acquiring raw sound field signals via a microphone array, the raw sound field signals including at least one sound source signal; an extraction module 604 for extracting features from the raw sound field signals using a target feature extraction network to generate a target feature tensor, the target feature tensor representing the spatial and semantic features of at least one sound source signal; a determination module 606 for determining the sound source analysis results of the raw sound field signals based on the target feature tensor, the sound source analysis results including the sound source location corresponding to at least one sound source signal and the target sound event type corresponding to at least one sound source signal; and an execution module 608 for controlling the acoustic interaction device to perform response operations corresponding to the sound source analysis results.

[0157] The extraction module is also used to preprocess the original sound field signal to obtain the acoustic feature tensor; based on the target feature extraction network, multi-dimensional joint feature extraction is performed on the acoustic feature tensor to obtain the target feature tensor.

[0158] The extraction module is further used to encode the acoustic feature tensor based on the feature encoder in the target feature extraction network to obtain a compressed feature tensor; and to perform multi-dimensional attention feature decoding on the compressed feature tensor based on the feature decoder in the target feature extraction network to obtain a target feature tensor, wherein the multi-dimensional attention includes channel attention, spectral attention and temporal attention.

[0159] The feature decoder includes a local perceptual unit layer, a multidimensional attention decoding layer, and an inverted residual feedforward network layer. The extraction module is further used to perform local time-frequency correlation perception on the compressed feature tensor based on the local perceptual unit layer to obtain a locally enhanced feature tensor; to perform multidimensional self-attention feature decoding on the locally enhanced feature tensor based on the multidimensional attention decoding layer to obtain an attention feature tensor; and to perform feature enhancement processing on the attention feature tensor based on the inverted residual feedforward network layer to obtain the target feature tensor.

[0160] The multidimensional attention decoding layer includes a channel attention module, a spectrum attention module, and a temporal attention module. The extraction module is also used to perform channel-dimensional self-attention processing on the local enhancement feature tensor based on the channel attention module to obtain a channel feature tensor; to perform frequency-dimensional self-attention processing on the channel feature tensor based on the spectrum attention module to obtain a frequency feature tensor; and to perform time-dimensional self-attention processing on the frequency feature tensor based on the temporal attention module to obtain an attention feature tensor.

[0161] The extraction module is further used to perform feature channel expansion processing on the attention feature tensor to obtain an expanded feature tensor; to perform local feature enhancement processing on the expanded feature tensor to obtain an enhanced feature tensor; and to perform residual fusion processing on the enhanced feature tensor to obtain the target feature tensor.

[0162] The feature encoder includes convolutional block layers, normalization layers, activation function layers, time-frequency domain pooling layers, and regularization layers. The extraction module further performs local feature extraction on the acoustic feature tensor based on the convolutional block layers to obtain intermediate feature tensors; normalizes the intermediate feature tensors based on the normalization layers to obtain normalized feature tensors; transforms the normalized feature tensors based on the activation function layers to obtain activated feature tensors; compresses the activated feature tensors in the time-frequency domain based on the time-frequency domain pooling layers to obtain dimensionality-reduced feature tensors; and performs feature regularization on the dimensionality-reduced feature tensors based on the regularization layers to obtain compressed feature tensors.

[0163] The extraction module is further configured to perform time-frequency domain transformation on the original sound field signal to obtain complex frequency domain signals for each microphone channel; map the complex frequency domain signals using a Mel filter bank to obtain Mel spectral features; perform correlation phase transformation on the complex frequency domain signals to obtain correlation phase transformation features; and fuse the Mel spectral features and correlation phase transformation features to obtain an acoustic feature tensor. Preferably, fusing the Mel spectral features and correlation phase transformation features to obtain the acoustic feature tensor includes: concatenating the Mel spectral features and correlation phase transformation features along the feature channel dimension to obtain the acoustic feature tensor.

[0164] The determination module is further used to parse the target feature tensor to obtain the sound source location corresponding to at least one sound source signal and the confidence level of the sound event type corresponding to at least one sound source signal; based on the confidence level of the sound event type corresponding to at least one sound source signal, the target sound event type corresponding to at least one sound source signal is determined; and the sound source location and the corresponding target sound event type corresponding to at least one sound source signal are integrated to obtain the sound source analysis result.

[0165] The determination module is also used to parse the target feature tensor to obtain the vector direction and vector length of the target feature tensor; based on the vector direction of the target feature tensor, determine the sound source location corresponding to at least one sound source signal; and based on the vector length of the target feature tensor, determine the confidence level of the sound event type corresponding to at least one sound source signal.

[0166] The execution module is further configured to, when at least one sound source signal is multiple sound source signals, acquire the response priority of the target sound event type corresponding to the multiple sound source signals; determine the response operation corresponding to the multiple sound source signals based on the target sound event type corresponding to the multiple sound source signals; and control the acoustic interaction device to sequentially execute the response operation corresponding to the multiple sound source signals based on the response priority. Preferably, the target sound event type includes: security event type, user voice interaction event type, and environmental interference event type, wherein the response priority corresponding to the security event type is greater than the response priority corresponding to the user voice interaction event type, and the response priority corresponding to the user voice interaction event type is greater than the response priority corresponding to the environmental interference event type.

[0167] The execution module is further configured to, when the target sound event type is a security event, guide the acoustic interaction device to monitor the corresponding security event based on the location of the corresponding sound source, and execute an alarm operation when the danger level of the security event is greater than a preset danger level; when the target sound event type is a user voice interaction event, control the microphone array to turn to the location of the corresponding sound source to enhance user voice interaction; and when the target sound event type is an environmental interference event, suppress the sound source signal at the location of the corresponding sound source.

[0168] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0169] According to another aspect of the embodiments of this application, a vehicle is also provided, including: a memory storing an executable program; and a processor for running the program, wherein the program executes the methods in various embodiments of this application when it runs.

[0170] The aforementioned memory can refer to devices inside a computer used to store data and programs, including RAM, hard disks, etc. RAM can be used to temporarily store running programs and data, while hard disks can be used to store programs and data long-term. Memory enables the computer to read and write data and execute programs. The aforementioned processor is responsible for executing instructions in computer programs and performing data processing. It can also be responsible for controlling and executing various operations, including arithmetic operations, logical operations, and data transmission.

[0171] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein, when the executable program is running, it controls the device where the computer-readable storage medium is located to perform the methods of various embodiments of this application.

[0172] The aforementioned computer storage media can refer to the media used in computer memory to store certain discontinuous physical quantities. Computer storage media mainly include semiconductors, magnetic cores, magnetic drums, magnetic tapes, laser discs, etc. Computer-readable storage media include stored programs, which can be a set of instructions that a computer can recognize and execute, running on an electronic computer to meet certain information needs.

[0173] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the methods of various embodiments of this application.

[0174] The aforementioned computer program products can refer to software programs that have been written, tested, and released, and can run on computers or other devices. Computer program products can include application programs, operating systems, utility software, etc., used to achieve specific functions or solve specific problems.

[0175] According to another aspect of the embodiments of this application, a computer program product is also provided, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the methods in various embodiments of this application.

[0176] The aforementioned non-volatile computer-readable storage medium can refer to a medium for storing data. Non-volatile computer-readable storage media can retain data without loss when power is off and can be used to store long-term data, such as operating systems, applications, and user files. Non-volatile storage media can include hard disk drives, solid-state drives, optical disks, and flash memory storage devices, etc.

[0177] According to another aspect of the embodiments of this application, a computer program is also provided, which, when executed by a processor, implements the methods of the various embodiments of this application.

[0178] The aforementioned computer program can refer to a set of instructions used to tell the computer to perform specific tasks or operations. Computer programs can be written by programmers using specific programming languages ​​and can include algorithms, data structures, logic, and control flow. Computer programs can be used for a variety of purposes, including application software, operating systems, etc.

[0179] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0180] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0181] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0182] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0183] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0184] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A control method for an acoustic interactive device, characterized in that, include: The original sound field signal is acquired by a microphone array in an acoustic interaction device, wherein the original sound field signal includes at least one sound source signal; The original sound field signal is subjected to feature extraction using a target feature extraction network to generate a target feature tensor, wherein the target feature tensor is used to characterize the spatial and semantic features of the at least one sound source signal; Based on the target feature tensor, the sound source analysis result of the original sound field signal is determined, wherein the sound source analysis result includes the sound source location corresponding to the at least one sound source signal, and the target sound event type corresponding to the at least one sound source signal; Control the acoustic interaction device to perform a response operation corresponding to the sound source analysis result.

2. The control method for the acoustic interactive device according to claim 1, characterized in that, The original sound field signal is used to extract features using a target feature extraction network to generate a target feature tensor, including: The original sound field signal is preprocessed to obtain the acoustic feature tensor; Based on the target feature extraction network, multidimensional joint feature extraction is performed on the acoustic feature tensor to obtain the target feature tensor.

3. The control method for the acoustic interactive device according to claim 2, characterized in that, Based on the target feature extraction network, multi-dimensional joint feature extraction is performed on the acoustic feature tensor to obtain the target feature tensor, including: Based on the feature encoder in the target feature extraction network, the acoustic feature tensor is feature-encoded to obtain a compressed feature tensor. Based on the feature decoder in the target feature extraction network, the compressed feature tensor is subjected to multidimensional attention feature decoding to obtain the target feature tensor, wherein the multidimensional attention includes channel attention, spectral attention and temporal attention.

4. The control method for the acoustic interactive device according to claim 3, characterized in that, The feature decoder includes a local perceptual unit layer, a multidimensional attention decoding layer, and an inverted residual feedforward network layer; Based on the feature decoder in the target feature extraction network, multidimensional attention feature decoding is performed on the compressed feature tensor to obtain the target feature tensor, including: Based on the local sensing unit layer, local time-frequency correlation sensing is performed on the compressed feature tensor to obtain a locally enhanced feature tensor. Based on the multidimensional attention decoding layer, multidimensional self-attention feature decoding is performed on the local enhanced feature tensor to obtain the attention feature tensor; Based on the inverted residual feedforward network layer, feature enhancement processing is performed on the attention feature tensor to obtain the target feature tensor.

5. The control method for the acoustic interactive device according to claim 4, characterized in that, The multidimensional attention decoding layer includes a channel attention module, a spectrum attention module, and a temporal attention module; Based on the multidimensional attention decoding layer, multidimensional self-attention feature decoding is performed on the local enhanced feature tensor to obtain an attention feature tensor, including: Based on the channel attention module, the local enhanced feature tensor is subjected to channel-dimensional self-attention processing to obtain the channel feature tensor. Based on the aforementioned spectrum attention module, the channel feature tensor is subjected to self-attention processing in the frequency dimension to obtain the frequency feature tensor. Based on the time attention module, the frequency feature tensor is subjected to time-dimensional self-attention processing to obtain the attention feature tensor.

6. The control method for the acoustic interactive device according to claim 4, characterized in that, Based on the inverted residual feedforward network layer, feature enhancement processing is performed on the attention feature tensor to obtain the target feature tensor, including: The attention feature tensor is subjected to feature channel expansion processing to obtain an expanded feature tensor; The extended feature tensor is subjected to local feature enhancement processing to obtain the enhanced feature tensor; The enhanced feature tensor is subjected to residual fusion processing to obtain the target feature tensor.

7. The control method for the acoustic interactive device according to claim 3, characterized in that, The feature encoder includes a convolutional block layer, a normalization layer, an activation function layer, a time-frequency pooling layer, and a regularization layer; Based on the feature encoder in the target feature extraction network, the acoustic feature tensor is feature-encoded to obtain a compressed feature tensor, including: Based on the convolutional block layer, local feature extraction is performed on the acoustic feature tensor to obtain the intermediate feature tensor; Based on the normalization layer, the intermediate feature tensor is normalized to obtain the normalized feature tensor. Based on the activation function layer, the normalized feature tensor is transformed to obtain the activation feature tensor; Based on the time-frequency domain pooling layer, the activation feature tensor is compressed in the time-frequency domain to obtain a dimension-reduced feature tensor. Based on the regularization layer, the reduced-dimensional feature tensor is subjected to feature regularization to obtain the compressed feature tensor.

8. The control method for the acoustic interactive device according to claim 2, characterized in that, The original sound field signal is preprocessed to obtain the acoustic feature tensor, including: The original sound field signal is subjected to time-frequency domain transformation to obtain the complex frequency domain signal of each microphone channel; The complex frequency domain signal is mapped using a Mel filter bank to obtain Mel spectral characteristics; Perform a correlation phase transform on the complex frequency domain signal to obtain the correlation phase transform characteristics; The Mel spectrum features and the related phase transform features are fused to obtain the acoustic feature tensor; Preferably, the acoustic feature tensor is obtained by fusing the Mel spectrum features and the related phase transform features, including: The Mel spectrum features and the related phase transform features are concatenated along the feature channel dimension to obtain the acoustic feature tensor.

9. The control method for the acoustic interactive device according to any one of claims 1 to 8, characterized in that, Based on the target feature tensor, the sound source analysis results of the original sound field signal are determined, including: The target feature tensor is analyzed to obtain the sound source location corresponding to the at least one sound source signal, and the confidence level of the sound event type corresponding to the at least one sound source signal; Based on the confidence level of the sound event type corresponding to the at least one sound source signal, the target sound event type corresponding to the at least one sound source signal is determined; The sound source location and the corresponding target sound event type corresponding to the at least one sound source signal are integrated to obtain the sound source analysis result.

10. The control method for the acoustic interactive device according to claim 9, characterized in that, The target feature tensor is parsed to obtain the sound source location corresponding to the at least one sound source signal, and the confidence level of the sound event type corresponding to the at least one sound source signal, including: The target feature tensor is analyzed to obtain its vector direction and vector length. Based on the vector direction of the target feature tensor, the sound source orientation corresponding to the at least one sound source signal is determined; Based on the vector length of the target feature tensor, the confidence level of the sound event type corresponding to the at least one sound source signal is determined.

11. The control method for the acoustic interactive device according to any one of claims 1 to 8, characterized in that, Controlling the acoustic interaction device to perform a response operation corresponding to the sound source analysis result includes: When the at least one sound source signal is a plurality of sound source signals, the response priority of the target sound event type corresponding to the plurality of sound source signals is obtained; Based on the target sound event type corresponding to the multiple sound source signals, determine the response operation corresponding to the multiple sound source signals; Based on the response priority, the acoustic interaction device is controlled to sequentially execute response operations corresponding to the plurality of sound source signals; Preferably, the target sound event types include: security event types, user voice interaction event types, and environmental interference event types, wherein the response priority of the security event type is greater than the response priority of the user voice interaction event type, and the response priority of the user voice interaction event type is greater than the response priority of the environmental interference event type.

12. The control method for the acoustic interactive device according to claim 11, characterized in that, The method further includes: When the target sound event type is the security event type, the acoustic interaction device is guided to monitor the corresponding security event based on the location of the corresponding sound source, and an alarm operation is performed when the danger level of the security event is greater than the preset danger level. When the target sound event type is the user voice interaction event type, the microphone array is controlled to turn to the corresponding sound source direction to enhance user voice interaction; When the target sound event type is the environmental interference event type, the sound source signal at the corresponding sound source location is suppressed.

13. A control device for an acoustic interactive device, characterized in that, include: The acquisition module is used to acquire raw sound field signals through a microphone array in an acoustic interaction device, wherein the raw sound field signals include at least one sound source signal; An extraction module is used to extract features from the original sound field signal using a target feature extraction network to generate a target feature tensor, wherein the target feature tensor is used to characterize the spatial and semantic features of the at least one sound source signal; The determination module is used to determine the sound source analysis result of the original sound field signal based on the target feature tensor, wherein the sound source analysis result includes the sound source location corresponding to the at least one sound source signal and the target sound event type corresponding to the at least one sound source signal; The execution module is used to control the acoustic interaction device to perform response operations corresponding to the sound source analysis results.

14. A vehicle, characterized in that, include: Memory, which stores executable programs; A processor for running the program, wherein the program, when running, executes the control method of the acoustic interactive device according to any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored executable program, wherein, when the executable program is executed, it controls the device on which the storage medium is located to perform the control method of the acoustic interactive device according to any one of claims 1 to 12.

Images