Intelligent sound box active interaction method and system based on complex acoustic environment

Abstract

The application discloses an intelligent sound box active interaction method and system based on a complex acoustic environment, and relates to the technical field of voice control.A kind of intelligent sound box active interaction system based on complex acoustic environment, comprising: acoustic environment construction module and sound box active interaction module.The application continuously collects environmental sound signals in complex acoustic environment, dynamically divides local sound field in target space, and constructs time-space continuous sound field unit with time continuity and space continuity, to ensure the accuracy and stability of sound field modeling under complex environment, and realize differentiated acoustic compensation for different propagation paths.

Description

Technical Field

[0001] This invention relates to the field of voice control technology, and in particular to a method and system for active interaction of smart speakers based on complex acoustic environments. Background Technology

[0002] With the development of intelligent voice interaction technology, smart speakers have been widely used in homes, offices, and intelligent control scenarios. Existing smart speakers typically achieve device interaction by collecting user voice and combining noise reduction, echo cancellation, and voice recognition technologies. However, in complex indoor environments, due to factors such as background noise, reverberation, multipath propagation, and dynamic changes in the sound field, existing technologies using a unified acoustic model for voice processing struggle to accurately adapt to local sound field changes. This can easily lead to a decrease in voice compensation effectiveness, thereby affecting the accuracy of voice recognition and the stability of interaction.

[0003] Therefore, this invention proposes an active interaction method for smart speakers based on complex acoustic environments, which accurately adapts to the dynamic changes of the sound field in complex acoustic environments, reduces the impact of multipath propagation and background noise on speech recognition, and improves the accuracy and stability of smart speaker voice interaction. Summary of the Invention

[0004] The present invention aims to provide a method and system for active interaction of smart speakers based on complex acoustic environments, which can accurately adapt to the dynamic changes of sound field in complex acoustic environments, reduce the impact of multipath propagation and background noise on speech recognition, and improve the accuracy and stability of smart speaker voice interaction.

[0005] A method for active interaction of a smart speaker in a complex acoustic environment includes the following steps: The system continuously collects ambient sound signals from various sound sampling points within the target acoustic environment using a smart speaker; it then combines all ambient sound signals to perform local sound field segmentation of the target acoustic environment, dividing it into several initial local sound field units; and finally, it performs temporal sound field state analysis on the initial local sound field units to obtain the corresponding temporal labels of the initial local sound field units. The target speech signal sequence is acquired in real time; the signal direction features of the target speech signal sequence are extracted; the corresponding initial local sound field unit is matched according to the signal direction features to obtain the target recognition sound field unit; and the sound field time sequence label corresponding to the target recognition sound field unit is used as the target sound field time sequence label. A local sound field model is constructed based on the target speech signal sequence collected within the target recognition sound field unit; acoustic compensation processing is performed on the target speech signal sequence using the local sound field model to obtain a target compensated speech sequence; speech recognition is performed based on the target compensated speech sequence to obtain the target speech text; corresponding device interaction commands are generated based on the target speech text to control the corresponding target device to perform the corresponding interactive operation.

[0006] As a preferred embodiment of the present invention, the specific steps for dividing the target acoustic environment into local sound fields by combining all ambient sound signals include: The ambient sound signal corresponding to each sound sampling point is extracted based on a preset time window; feature extraction is performed based on the ambient sound signal to obtain the ambient acoustic feature vector; the ambient acoustic feature vector includes at least sound energy distribution features, spectrum distribution features, spatial direction features and acoustic reverberation features; the spatial location of each sound sampling point is obtained; By combining the spatial location of each sampling point with the environmental acoustic feature vector, the spatial correlation between any two sound sampling points is calculated; based on the spatial correlation and spatial direction features, multiple sound sampling points are spatially clustered to obtain several initial local sound field units.

[0007] As a preferred embodiment of the present invention, the specific steps for performing time-series sound field state analysis on the initial local sound field unit include: The initial local sound field units corresponding to each preset time window are traversed according to the preset time order. For any two adjacent preset time windows, the temporal correlation degree between the corresponding initial local sound field units is calculated. The sound field mapping relationship between the corresponding initial local sound field units is established based on the temporal correlation degree. Based on the sound field mapping relationship, the corresponding quantitative relationship between each initial local sound field unit is statistically analyzed: when one initial local sound field unit is mapped to multiple initial local sound field units, the split mapping relationship is satisfied; when multiple initial local sound field units are mapped to one initial local sound field unit, the fusion mapping relationship is satisfied; when one initial local sound field unit is mapped to only one initial local sound field unit, the continuous mapping relationship is satisfied. Based on the split mapping relationship, fusion mapping relationship and continuous mapping relationship, the corresponding sound field temporal label is matched for the initial local sound field unit.

[0008] As a preferred embodiment of the present invention, the specific steps for constructing a local sound field model based on the target speech signal sequence collected within the target recognition sound field unit include: The target speech signal sequence is segmented according to a preset recognition window to obtain several target speech signals; the target temporal correlation between different target speech signals is extracted; based on the target temporal correlation and the signal direction features corresponding to the target speech signals, the propagation source and propagation direction of the target speech signals are identified; the target speech propagation path and the reflection propagation path associated with the target speech propagation path are established based on the propagation source and propagation direction. Propagation state analysis is performed on the target speech propagation path and reflection propagation path to obtain the background propagation path and dynamic propagation path; the features of the target speech propagation path, reflection propagation path, background propagation path and dynamic propagation path are fused to generate a local sound field model.

[0009] As a preferred embodiment of the present invention, the specific steps of performing acoustic compensation processing on the target speech signal sequence using a local sound field model to obtain a target compensated speech sequence and performing speech recognition based on the target compensated speech sequence to obtain target speech text include: In the local sound field model, the propagation relationships between different propagation paths are identified; feature components are extracted from the target speech signal to obtain the propagation components corresponding to each propagation path; propagation compensation relationships are established based on the propagation relationships corresponding to the propagation paths; each propagation component is independently compensated based on each propagation compensation relationship to obtain each propagation compensation signal; each propagation compensation signal is reconstructed to obtain the target compensated speech. Speech sorting is performed on all target compensated speech to obtain multiple target speech segments; the target speech segments are then semantically reorganized to generate target speech text.

[0010] As a preferred technical solution of the present invention, semantic recognition is performed on the target speech text to obtain several independent semantic texts; semantic compensation is performed on the independent semantic texts to obtain corresponding device operation texts; and device operation texts are matched with device instructions to obtain corresponding device interaction instructions.

[0011] An active interaction system for smart speakers based on complex acoustic environments, comprising: The acoustic environment construction module includes a sound field division unit. The sound field division unit is used to continuously collect ambient sound signals from each sound sampling point in the target acoustic environment using a smart speaker. It combines all ambient sound signals to perform local sound field division on the target acoustic environment, dividing the target acoustic environment into several initial local sound field units. It performs temporal sound field state analysis on the initial local sound field units to obtain the corresponding sound field temporal labels. The speaker's active interaction module includes a signal extraction unit, an interaction compensation unit, and an instruction execution unit. The signal extraction unit acquires the target speech signal sequence in real time; extracts the signal direction features of the target speech signal sequence; matches the corresponding initial local sound field unit based on the signal direction features to obtain the target recognition sound field unit; and uses the sound field timing label corresponding to the target recognition sound field unit as the target sound field timing label. The interaction compensation unit constructs a local sound field model based on the target speech signal sequence acquired within the target recognition sound field unit; performs acoustic compensation processing on the target speech signal sequence using the local sound field model to obtain the target compensated speech sequence; and performs speech recognition based on the target compensated speech sequence to obtain the target speech text. The instruction execution unit generates corresponding device interaction instructions based on the target speech text to control the corresponding target device to perform the appropriate interactive operation.

[0012] The present invention has the following advantages: 1. This invention continuously collects ambient sound signals from complex acoustic environments, dynamically divides the target space into local sound fields, and constructs spatiotemporally continuous sound field units with temporal and spatial continuity. This accurately describes the dynamic evolution of the sound field in complex indoor environments, ensuring the accuracy and stability of sound field modeling in complex environments. Propagation compensation relationships are established for the target propagation path, reflection propagation path, background propagation path, and dynamic propagation path, respectively, to achieve differentiated acoustic compensation for different propagation paths.

[0013] 2. This invention determines the propagation priority based on the propagation relationship between different propagation paths and generates speech recognition constraint information to guide the target compensated speech to perform constraint recognition processing. This enables the speech recognition process to actively constrain itself by combining the propagation characteristics of complex sound fields, rather than relying solely on a unified recognition model. This effectively reduces misidentification, missed identification, and semantic breaks in complex acoustic environments, ensuring the continuity and accuracy of target speech text generation and the overall reliability of the smart speaker's active interaction. Attached Figure Description

[0014] Figure 1 This is a schematic diagram of the structure of an intelligent speaker active interaction system based on a complex acoustic environment, as used in an embodiment of the present invention. Detailed Implementation

[0015] To enable those skilled in the art to better understand the technical solutions of this invention, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of this invention.

[0016] Example 1: A method for active interaction of a smart speaker in a complex acoustic environment, comprising the following steps: The system continuously collects ambient sound signals from various sound sampling points within the target acoustic environment using a smart speaker; it then combines all ambient sound signals to perform local sound field segmentation of the target acoustic environment, dividing it into several initial local sound field units; and finally, it performs temporal sound field state analysis on the initial local sound field units to obtain the corresponding temporal labels of the initial local sound field units. The specific steps for dividing the target acoustic environment into local sound fields by combining all ambient sound signals include: The ambient sound signal corresponding to each sound sampling point is extracted based on a preset time window; feature extraction is performed based on the ambient sound signal to obtain the ambient acoustic feature vector; the ambient acoustic feature vector includes at least sound energy distribution features, spectrum distribution features, spatial direction features and acoustic reverberation features; the spatial location of each sound sampling point is obtained; The smart speaker uses a microphone array to continuously collect ambient sound signals from the target acoustic environment and processes them in segments according to a preset time window. The preset time window can be set from 100 milliseconds to 500 milliseconds according to the actual application scenario, so that the sound field characteristics within each time window remain basically consistent. The ambient sound signal corresponding to each sound sampling point is digitally filtered and amplitude normalized to eliminate the impact of gain differences in the acquisition devices on subsequent analysis. Then, multi-dimensional acoustic features of the ambient sound signal are extracted. Among these, the sound energy distribution feature describes the sound energy magnitude per unit time at each sampling point, which can be obtained by calculating the mean square value of the signal within the time window; the spectral distribution feature describes the energy distribution of different frequency components, which can be obtained by converting the time-domain signal to the frequency-domain signal using a fast Fourier transform and then statistically analyzing the energy ratio of each frequency band; the spatial direction feature reflects the main propagation direction of the sound, which can be calculated by combining the arrival time difference and phase difference between microphone arrays; and the acoustic reverberation feature describes the attenuation characteristics of sound after multiple reflections indoors, which can be calculated based on the sound energy attenuation curve and the duration of the reverberation tail. Finally, the above features are combined in a unified order to form the corresponding ambient acoustic feature vector, providing a unified data representation for subsequent local sound field segmentation.

[0017] Establish the correspondence between each sound sampling point and its spatial coordinates in the smart speaker; where a sound sampling point refers to a fixed sampling position corresponding to one or more microphones in the smart speaker used to collect ambient sound signals, and each sound sampling point has unique spatial coordinate information; using the installation position of the smart speaker itself as the spatial coordinate reference origin, and based on the pre-calibrated microphone array geometry inside the smart speaker, obtain the three-dimensional spatial coordinates corresponding to each sound sampling point; if multiple smart speakers are deployed in the target acoustic environment, obtain the installation position and installation posture of each smart speaker in the target acoustic environment, and establish a unified environmental coordinate system; based on the installation position, installation posture of each smart speaker, and the relationship between each sound sampling point and its corresponding microphone array, obtain the spatial coordinates of each smart speaker. The local coordinates of the speaker are used to calculate the spatial position of each sound sampling point in a unified environmental coordinate system through coordinate transformation. Specifically, the coordinate transformation includes: constructing a corresponding rotation transformation matrix based on the installation posture to rotate the local coordinates; then, combining the translation vector corresponding to the installation position, performing spatial translation calculation on the rotated coordinates to obtain the three-dimensional spatial coordinates of each sound sampling point in the unified environmental coordinate system; establishing a one-to-one correspondence between the three-dimensional spatial coordinates of each sound sampling point and the corresponding environmental sound signal to generate spatial sampling data containing the sound sampling point number, spatial position, and environmental sound signal, providing a unified spatial position basis for subsequent calculation of the spatial correlation between sound sampling points and the execution of local sound field division.

[0018] By combining the spatial location of each sampling point with the environmental acoustic feature vector, the spatial correlation between any two sound sampling points is calculated; based on the spatial correlation and spatial direction features, multiple sound sampling points are spatially clustered to obtain several initial local sound field units.

[0019] The three-dimensional spatial coordinates and environmental acoustic feature vectors corresponding to each sound sampling point are used as input data. Spatial correlation is used to characterize the probability that the environmental sounds collected by two sound sampling points belong to the same local sound field. It reflects not only the spatial proximity between sound sampling points but also the consistency of corresponding environmental sound features. The spatial distance between any two sound sampling points is calculated, and a spatial distance similarity is generated based on this distance; the closer the spatial distance, the higher the spatial distance similarity. Simultaneously, the corresponding environmental acoustic feature vectors are normalized to ensure that each feature has a uniform data dimension. Then, the sound energy distribution characteristics are calculated separately. The similarity between spectral distribution features, spatial orientation features, and acoustic reverberation features is calculated and weighted according to preset weights to obtain the comprehensive acoustic similarity between corresponding sound sampling points. Finally, the spatial distance similarity and comprehensive acoustic similarity are fused to obtain the spatial correlation between any two sound sampling points, and a spatial correlation matrix containing the pairwise correlations of all sound sampling points is constructed. The spatial correlation matrix is ​​a data matrix used to record the spatial correlation between any two sound sampling points, and its matrix elements correspond to the spatial correlation between two sound sampling points, providing a correlation basis for subsequent spatial clustering.

[0020] Using the spatial correlation matrix as the clustering input data, each sound sampling point is taken as an initial clustering object. Subsequently, a local sound field division is performed using a spatial clustering method based on neighborhood expansion. Specifically, this includes: sequentially selecting an un-clustered sound sampling point as a clustering seed, searching for neighboring sound sampling points with a spatial correlation degree greater than a preset correlation threshold, and determining whether the spatial orientation features of the corresponding sound sampling points remain continuous. The continuity of spatial orientation features means that the changes in the main sound propagation direction corresponding to multiple sound sampling points remain within a preset angle range, so as to ensure that the sounds in the same cluster mainly come from consistent or continuously changing sound source directions. For sound sampling points that simultaneously meet the spatial association condition and the spatial orientation continuity condition, they are merged into the same cluster, and the neighborhood search is repeated with the newly added sound sampling points as new expansion seeds until no new sound sampling points that meet the conditions are found, at which point the current cluster expansion stops. The above clustering process is repeated for the remaining sound sampling points that have not participated in the clustering until all sound sampling points have been clustered. Each clustering result is determined as an initial local sound field unit, where the initial local sound field unit refers to a set of sound sampling points with similar spatial locations, similar environmental acoustic features, and continuous sound propagation directions within the same preset time window. It is used to represent a local continuous sound field region in the target acoustic environment and serves as the basic object for subsequent sound field temporal state analysis and sound field temporal label generation.

[0021] Assume the target acoustic environment is a living room, with a smart speaker equipped with multiple microphones as different sound sampling points. The following sounds are present in the living room: a user on the left sofa is saying "turn on the air conditioner"; a television on the right side of the living room is continuously playing news; and the air conditioner, located above the room, is continuously emitting a steady stream of airflow noise.

[0022] Within a preset time window, different environmental sound signals are collected at each sound sampling point, and the corresponding environmental acoustic feature vectors are extracted, including sound energy distribution features, spectral distribution features, spatial direction features, and acoustic reverberation features. Simultaneously, the spatial position of each sound sampling point is obtained based on the position of the microphone array.

[0023] The system calculates the spatial correlation between any two sound sampling points. For example, multiple sound sampling points near the left side of the living room are not only spatially close, but also collect sounds mainly from the user's speech on the left. Their sound energy changes, spectrum distribution, and sound propagation direction are relatively consistent. Therefore, these sound sampling points have a high degree of spatial correlation and are clustered to form the initial local sound field unit A. Multiple sound sampling points near the direction of the TV receive mainly the sound of the TV playback, and their environmental acoustic characteristics are similar, thus forming the initial local sound field unit B. Multiple sound sampling points near the air conditioner vent continuously receive wind noise signals, and their propagation direction is stable and their reverberation characteristics are consistent, thus forming the initial local sound field unit C. The environmental sound that was originally mixed throughout the living room is divided into multiple independent local sound field units: local sound field unit A mainly corresponds to the user's speech area, local sound field unit B mainly corresponds to the TV sound area, and local sound field unit C mainly corresponds to the air conditioner noise area.

[0024] When the system receives user voice messages, it can first determine that the voice message belongs to local sound field unit A and will not be affected by TV sound or air conditioner noise. It can also combine the sound field time sequence label corresponding to local sound field unit A to construct a local sound field model that is more in line with the current environment, and perform targeted acoustic compensation for user voice messages, thereby improving the accuracy of target speech extraction and speech recognition in complex acoustic environments.

[0025] The specific steps for performing temporal sound field state analysis on the initial local sound field unit include: The initial local sound field units corresponding to each preset time window are traversed according to the preset time order. For any two adjacent preset time windows, the temporal correlation degree between the corresponding initial local sound field units is calculated. The sound field mapping relationship between the corresponding initial local sound field units is established based on the temporal correlation degree. The initial local sound field units corresponding to each preset time window are read in chronological order, and the environmental acoustic feature vector, spatial location range, and spatial direction features corresponding to each initial local sound field unit are extracted. Among them, the temporal correlation degree is a correlation index used to describe whether two initial local sound field units in two adjacent preset time windows belong to the same continuous sound field. It comprehensively reflects the continuity of the sound field in time and space. For each pair of initial local sound field units in any two adjacent preset time windows, the feature similarity between environmental acoustic feature vectors, the degree of overlap between spatial positions, and the degree of directional consistency between spatial directional features are calculated, and the above results are normalized respectively. The normalized indicators are weighted according to preset weights to obtain the temporal correlation degree between the corresponding initial local sound field units. When the temporal correlation degree is greater than the preset temporal correlation threshold, it is considered that the two initial local sound field units have the same or similar acoustic properties in a continuous time, and a sound field mapping relationship is established between them; otherwise, no correspondence is established. The sound field mapping relationship refers to the one-to-one correspondence between initial local sound field units in two different time windows, which is used to represent the inheritance relationship of local sound field changes over time, providing a basis for subsequent analysis of the sound field evolution process.

[0026] Based on the sound field mapping relationship, the corresponding quantitative relationship between each initial local sound field unit is statistically analyzed: when one initial local sound field unit is mapped to multiple initial local sound field units, the split mapping relationship is satisfied; when multiple initial local sound field units are mapped to one initial local sound field unit, the fusion mapping relationship is satisfied; when one initial local sound field unit is mapped to only one initial local sound field unit, the continuous mapping relationship is satisfied. All established sound field mapping relationships are traversed and counted, and the number of mappings corresponding to each initial local sound field unit is counted. The number of mappings refers to the number of initial local sound field units that establish mapping relationships with each initial local sound field unit within the adjacent time window.

[0027] Based on the statistical results, the corresponding quantitative relationships are analyzed as follows: If the same initial local sound field unit corresponds to multiple subsequent initial local sound field units, it indicates that the original local sound field has been affected by factors such as the addition of new sound sources, changes in obstacles, or changes in propagation direction, and has been divided into multiple new local sound field regions. In this case, the splitting mapping relationship is satisfied. If multiple initial local sound field units correspond to the same subsequent initial local sound field unit, it indicates that multiple originally independent local sound fields have merged into a unified local sound field due to the disappearance of sound sources, overlapping propagation areas, or changes in the environment. In this case, the fusion mapping relationship is satisfied. If an initial local sound field unit corresponds to only one subsequent initial local sound field unit, it indicates that the local sound field remains continuous and stable, and only undergoes normal temporal evolution. In this case, the continuous mapping relationship is satisfied. Through the above statistical analysis, the evolution pattern of the local sound field in continuous time can be accurately described, providing a basis for establishing a stable sound field temporal identifier.

[0028] Based on the split mapping relationship, the fusion mapping relationship and the continuous mapping relationship, the corresponding sound field temporal label is matched for the initial local sound field unit; Each initial local sound field unit within a preset time window is assigned a unique sound field timing label; wherein, the sound field timing label is a unique identifier used to identify the evolution process of the same local sound field within multiple consecutive preset time windows, and does not change with the position of the local sound field.

[0029] The initial local sound field units in each subsequent preset time window are processed sequentially according to a preset time order, and the inheritance method of the sound field timing label is determined according to the corresponding mapping relationship. When the continuous mapping relationship is satisfied, the sound field timing label of the initial local sound field unit corresponding to the previous time window is directly inherited to the corresponding subsequent initial local sound field unit, so that the same local sound field maintains a unified identifier throughout the evolution process. When the split mapping relationship is satisfied, the original sound field timing label is retained as the main label for the initial local sound field unit with the highest correlation to the original local sound field, and new sound field timing labels are generated for the remaining initial local sound field units generated by the split. At the same time, the split correlation relationship between the new label and the main label is recorded. When the fusion mapping relationship is satisfied, the sound field timing label with the longest duration or the highest temporal continuity among the multiple initial local sound field units participating in the fusion is used as the inherited label after fusion. The remaining sound field timing labels are recorded as fusion source labels, and corresponding fusion correlation relationships are established.

[0030] Each initial local sound field unit obtains a unique and continuous sound field temporal label, enabling the system not only to identify the current spatial location of the local sound field, but also to fully record the formation, splitting, fusion and continuous evolution of the local sound field in the entire time dimension, providing continuous and stable temporal basis for subsequent sound field identification of target speech, construction of local sound field models and propagation path analysis.

[0031] For example, after a smart speaker is installed in the living room of a residence, it does not immediately begin active interaction. Instead, during the initial modeling phase, it continuously collects environmental sound data from different times of day for a week, including environmental sound signals formed by different life scenarios such as family members' activities, TV playback, air conditioning operation, and cooking in the kitchen. The system divides all historical environmental sounds according to preset time windows. During the time when watching TV every night, the historical data from multiple days form local sound field units located near the sofa area.

[0032] Although the TV programs and the content of the speeches are different every day, their spatial location, propagation direction, sound energy distribution and reverberation characteristics are basically the same. Therefore, the system calculates that the local sound field units corresponding to different historical time windows have a high degree of temporal correlation and establishes the corresponding sound field mapping relationship, believing that these local sound fields belong to the same stable historical sound field evolution process.

[0033] Taking the living room example again, during the analysis of all historical data, the system found that during some historical periods, family members only watched TV in the sofa area, which formed a local sound field; while during other historical periods, one family member watched TV while another family member was talking in the dining table area, so the original local sound field in the sofa area corresponded to two local sound fields, thus the split mapping relationship was obtained.

[0034] The system further discovered that during certain historical periods, both family members eventually sat back in the sofa area to communicate, causing the two local sound fields that originally corresponded to the TV area and the dining table area respectively to re-form a unified local sound field, thus allowing for the statistical analysis of a fusion mapping relationship. For local sound fields that remained stable in position and exhibited minimal changes in propagation characteristics throughout most historical time windows, a continuous mapping relationship was obtained. Through the statistical analysis of a large amount of historical data, the system was able to fully grasp the long-term evolution patterns of various local sound fields in the living room, rather than making judgments based solely on a single sampling result.

[0035] Based on the historical analysis results above, the system first assigns a sound field temporal label L1 to the local sound field of the sofa area that first appears in the historical data; the sound field temporal label L1 is inherited between all historical local sound fields that have established continuous mapping relationships, so that local sound fields of different dates and times but belonging to the same sound field evolution process have the same label.

[0036] When a splitting mapping relationship appears in the historical data, the system continues to retain the sound field timing label L1 corresponding to the main sofa area, while assigning a new sound field timing label L2 to the local sound field of the newly added dining table area, and recording that L2 is formed by splitting L1; when a fusion mapping relationship appears in the historical data, the sound field timing label with the longest duration and the highest frequency of occurrence is retained as the inherited label after fusion, while recording the association information of other labels participating in fusion; the system establishes a complete sound field timing label library based on all historical environmental sound data, such as label L1 for the sofa area, label L2 for the dining table area, label L3 for the kitchen area, etc., and each label corresponds to the propagation characteristics and evolution relationship obtained from long-term statistics.

[0037] The target speech signal sequence is acquired in real time; the signal direction features of the target speech signal sequence are extracted; the corresponding initial local sound field unit is matched according to the signal direction features to obtain the target recognition sound field unit; and the sound field time sequence label corresponding to the target recognition sound field unit is used as the target sound field time sequence label.

[0038] The smart speaker continuously monitors sound signals in the target acoustic environment and synchronously collects sound data corresponding to each sound sampling point according to a preset sampling period. It performs voice activity detection on the continuously collected sound data using a preset voice detection threshold. When the voice energy exceeds the preset energy threshold and the spectral change conforms to human voice characteristics within multiple consecutive sampling periods, it determines that user voice has been detected. The starting time of the detected voice is taken as the starting point of the voice segment, and the end point is taken as the voice energy falling below the preset energy threshold within multiple consecutive sampling periods. The continuously collected voice data from the starting point to the end point is combined to form a target voice signal sequence. The target voice signal sequence refers to the set of voice data continuously collected during a complete interaction. It not only includes the target voice emitted by the user but may also include environmental noise, reverberation, and interference information from other sound sources, providing the original data basis for subsequently determining the local sound field to which the target voice belongs.

[0039] Based on the speech arrival time difference and speech energy distribution corresponding to each sound sampling point in the target speech signal sequence, the propagation time difference between the target speech and each sound sampling point is calculated; combined with the spatial location of each sound sampling point, a target speech propagation direction estimation model is established; by traversing the preset direction angle range, the theoretical propagation time difference corresponding to each candidate propagation direction is calculated, and compared with the actual propagation time difference respectively, the direction matching degree corresponding to each candidate propagation direction is calculated. The candidate propagation direction with the highest directional matching degree is selected as the main propagation direction of the target speech. Furthermore, the main propagation direction is corrected by combining the received energy distribution corresponding to each sound sampling point, resulting in the signal direction feature corresponding to the target speech signal. This signal direction feature describes the spatial propagation direction of the target speech when it reaches the smart speaker. It can be represented by azimuth, pitch, or the corresponding direction vector, reflecting the spatial direction from which the target speech originates in the target acoustic environment.

[0040] Read the spatial direction feature ranges corresponding to all initial local sound field units. The spatial direction feature ranges are the distribution of the main sound propagation directions obtained statistically from the local sound field division stage, which are used to characterize the main sound source directions corresponding to each initial local sound field unit over a long period of time. Compare the signal direction features corresponding to the target speech signal with the spatial direction feature ranges corresponding to each initial local sound field unit to calculate the directional matching degree between the target speech signal and each initial local sound field unit. For initial local sound field units whose directional matching degree meets the preset matching conditions, combine the spatial position distribution features corresponding to the target speech signal and the environmental acoustic features of the corresponding initial local sound field unit to calculate the comprehensive matching degree. The initial local sound field unit with the highest comprehensive matching degree is determined as the target recognition sound field unit.

[0041] Among them, the target recognition sound field unit refers to the local sound field region that is most consistent with the spatial source of the target speech and most consistent with the environmental acoustic characteristics, and is used to determine the local acoustic environment to which the target speech actually belongs.

[0042] It should be noted that after the smart speaker starts monitoring, an initial environmental sampling duration (e.g., 5-10 seconds) is first set, and the ambient sound signals in the target acoustic environment are continuously collected. The collected ambient sound is divided into multiple consecutive sampling frames. The short-time speech energy value corresponding to each sampling frame is calculated, and the mean and standard deviation of the short-time speech energy values ​​of all sampling frames are statistically analyzed. The short-time speech energy value is used to reflect the overall strength of the sound signal within the corresponding sampling frame. Based on this, the average energy of environmental noise is used as the benchmark value, and the standard deviation of a preset multiple is superimposed on the benchmark value to obtain the preset speech detection threshold. Since the standard deviation can reflect the degree of fluctuation of environmental noise energy, when the environmental noise is relatively stable, the obtained threshold is lower, which is beneficial for detecting weak human voices. When the environmental noise fluctuates greatly, the obtained threshold is automatically increased, which can reduce the probability of environmental noise falsely triggering speech detection.

[0043] During equipment operation, the environmental noise statistics are continuously updated using a sliding time window, and the average energy and standard deviation of the environmental noise are recalculated according to the preset update cycle. The preset speech detection threshold is dynamically corrected so that the speech detection threshold can be adaptively adjusted with changes in environmental noise.

[0044] The detected target speech signal sequence is divided into multiple consecutive speech frames of a fixed length, and the short-time speech energy value corresponding to each speech frame is calculated. The maximum and minimum values ​​of the short-time speech energy values ​​of all speech frames in the target speech signal sequence are counted, and the corresponding energy variation range is calculated. The energy variation range is divided according to a preset ratio to obtain the target speech preservation threshold as the preset energy threshold. The preset ratio can be determined according to the speech preservation requirements, for example, 20% to 30% of the energy variation range. The value can be adjusted according to the actual monitoring needs. This embodiment only shows one achievable value. The system iterates through each speech frame in chronological order. When the short-term speech energy corresponding to multiple consecutive speech frames is lower than a preset energy threshold, the target speech is determined to have ended, and the corresponding time is taken as the termination position of the target speech signal sequence. If subsequent speech energy is detected to be continuously higher than the preset speech detection threshold, a new target speech signal sequence extraction begins. This method avoids erroneous speech truncation due to short pauses by the user or energy fluctuations in a single sampling frame, improving the continuity and completeness of target speech extraction.

[0045] A local sound field model is constructed based on the target speech signal sequence collected within the target recognition sound field unit; The specific steps for constructing a local sound field model based on the target speech signal sequence collected within the target recognition sound field unit include: The target speech signal sequence is segmented according to a preset recognition window to obtain several target speech signals; the target temporal correlation between different target speech signals is extracted; based on the target temporal correlation and the signal direction features corresponding to the target speech signals, the propagation source and propagation direction of the target speech signals are identified; the target speech propagation path and the reflection propagation path associated with the target speech propagation path are established based on the propagation source and propagation direction. A continuous speech timeline is established based on the start and end times of the target speech signal sequence, and the target speech signal sequence is continuously divided according to the length of a preset recognition window. The length of the preset recognition window can be set according to the rhythm of normal speech pronunciation. For example, each preset recognition window corresponds to several tens to hundreds of milliseconds of continuous speech data, so that each time window contains relatively complete and stable speech information. A preset overlapping area is set between adjacent preset recognition windows to maintain the continuity of the same speech segment between adjacent time windows, thereby reducing the loss of speech information caused by window boundaries. The speech data corresponding to each time window is determined as a target speech signal, and a corresponding time number is assigned according to the time sequence to form a set of target speech signals, which provides a basis for establishing the temporal continuity of the speech propagation process.

[0046] The target speech signal refers to a continuous speech segment in the target speech signal sequence that corresponds to a preset time window. Each target speech signal retains the corresponding time information, signal direction features, and acoustic features.

[0047] The target speech signals are read sequentially according to their time numbers, and the corresponding speech energy change features, spectrum change features, and signal direction features are extracted respectively. The degree of change between the corresponding features of the target speech signals that are adjacent in time is calculated. The degree of change is used to reflect the continuity of the two target speech signals during the propagation process. Then, the time interval, the degree of change of speech energy, the degree of change of spectrum, and the degree of change of signal direction are normalized and weighted and fused according to preset weights to calculate the temporal correlation degree between adjacent target speech signals. When the temporal correlation degree meets the preset correlation conditions, the target temporal correlation relationship between the corresponding target speech signals is established and the correlation continues to be extended to the next target; otherwise, the current correlation relationship is terminated and a new target temporal correlation relationship is established.

[0048] Among them, the target temporal correlation refers to the continuous propagation relationship of multiple target speech signals in the time dimension. It is used to describe the evolution of the same speech propagation process in multiple consecutive time windows, rather than simply representing the order of time.

[0049] Based on the temporal correlation of the targets, multiple target speech signals belonging to the same continuous propagation process are grouped into a propagation analysis sequence, and the signal direction features corresponding to each target speech signal are read; where the propagation source refers to the actual spatial location that causes the target speech to be generated, and the propagation direction refers to the spatial propagation direction of the target speech from the propagation source to the smart speaker; The signal direction features corresponding to each target speech signal in the propagation analysis sequence are fitted and analyzed in chronological order to calculate the trend of propagation direction change over a continuous period of time. The rate of change of propagation direction is calculated in combination with the time interval between target speech signals. If the propagation direction remains stable, the propagation source is determined to be fixed. If the propagation direction changes continuously, the movement trend of the propagation source is inferred from the trajectory of the propagation direction change. Finally, the spatial location corresponding to the propagation source is correlated with the propagation direction of the target speech signal to obtain the propagation source and propagation direction of the target speech signal, providing a spatial propagation basis for subsequent propagation path construction.

[0050] Using the source of propagation as the starting point and the target recognition sound field unit corresponding to the smart speaker as the end point, a direct propagation path for the target speech is established based on the direction of propagation. The target speech propagation path refers to the trajectory of the target speech as it propagates directly from the air medium to the smart speaker. Combining the environmental acoustic characteristics corresponding to the target recognition sound field unit and the reverberation characteristics in historical environmental sound signals, the main reflection areas that may exist in the target acoustic environment are identified. Based on the spatial relationship between the target speech propagation path and each reflection area, the propagation trajectory of the target speech after reflection by each reflection area is calculated to reach the smart speaker. For each reflection propagation trajectory, the corresponding propagation distance, propagation delay, and propagation energy attenuation are calculated, and the corresponding reflection propagation path is established. The reflection propagation path refers to the propagation path formed after the target speech is reflected by walls, furniture, or other fixed obstacles. It is used to describe the propagation process of reverberant sound in complex acoustic environments, so that the subsequent local sound field model not only includes direct propagation characteristics but also reflects environmental reflection characteristics.

[0051] Propagation state analysis is performed on the target speech propagation path and reflection propagation path to obtain the background propagation path and dynamic propagation path; feature fusion is performed on the target speech propagation path, reflection propagation path, background propagation path and dynamic propagation path to generate a local sound field model; Statistical analysis is performed on the target speech propagation paths and reflection propagation paths corresponding to multiple consecutive preset recognition windows, and the changes in propagation direction, propagation delay, propagation energy, and frequency of occurrence for each propagation path are calculated. The magnitude of change of each propagation path is analyzed. When a propagation path remains stable in multiple consecutive time windows, and the changes in propagation direction, propagation delay, and propagation energy are all within a preset stable range, the propagation path is determined as a background propagation path. When a propagation path only appears in some time windows, or its propagation direction, propagation delay, or propagation energy changes continuously with time, the propagation path is determined as a dynamic propagation path.

[0052] Among them, the background propagation path is used to represent the stable propagation characteristics formed by a fixed environmental structure, such as the stable reflection path generated by a fixed wall or fixed furniture; the dynamic propagation path is used to represent the time-varying propagation characteristics caused by dynamic environmental changes such as the movement of people, the opening and closing of doors and windows, and the movement of objects, thereby realizing the distinction between the influence of static and dynamic environments.

[0053] For the target speech propagation path, reflection propagation path, background propagation path, and dynamic propagation path, corresponding propagation direction features, propagation distance features, propagation delay features, propagation energy features, and temporal continuity features are extracted respectively, and corresponding propagation path feature vectors are constructed. The feature vectors of each propagation path are uniformly normalized to eliminate the influence between different feature dimensions. According to the importance of different propagation paths to the propagation of target speech, corresponding weights are assigned, and all propagation path feature vectors are weighted and fused to generate a unified local sound field feature matrix. The local sound field feature matrix is ​​used as the local sound field model.

[0054] Among them, the local sound field model is a digital model used to describe the propagation law of speech within the target recognition sound field unit. It comprehensively reflects the direct propagation characteristics, environmental reflection characteristics, background propagation characteristics, and dynamic propagation characteristics of the target speech. It can realistically describe the propagation state of the target speech in the current complex acoustic environment, providing accurate propagation feature basis for subsequent acoustic compensation processing, thereby improving the accuracy of target speech enhancement and speech recognition in complex environments.

[0055] The target speech signal sequence is acoustically compensated using a local sound field model to obtain a target compensated speech sequence; speech recognition is then performed based on the target compensated speech sequence to obtain the target speech text. The specific steps for performing acoustic compensation processing on the target speech signal sequence using a local sound field model to obtain the target compensated speech sequence, and performing speech recognition based on the target compensated speech sequence to obtain the target speech text, include: In the local sound field model, the propagation relationships between different propagation paths are identified; feature components are extracted from the target speech signal to obtain the propagation components corresponding to each propagation path; propagation compensation relationships are established based on the propagation relationships corresponding to the propagation paths; each propagation component is independently compensated based on each propagation compensation relationship to obtain each propagation compensation signal; each propagation compensation signal is reconstructed to obtain the target compensated speech. The propagation feature information corresponding to the target speech propagation path, reflection propagation path, background propagation path, and dynamic propagation path stored in the local sound field model is read. The propagation relationship refers to the correspondence between different propagation paths in terms of spatial location, propagation delay, propagation energy, and propagation direction, reflecting the degree of mutual influence between different propagation paths. Subsequently, the propagation distance difference, propagation delay difference, propagation direction angle, and propagation energy difference between any two propagation paths are calculated, and each parameter is normalized. The normalized parameters are then fused according to preset weights to obtain the degree of propagation correlation between different propagation paths, and a corresponding propagation relationship matrix is ​​established based on this correlation. The propagation relationship matrix records the coupling relationship and mutual influence degree between each propagation path, providing a basis for subsequently establishing compensation strategies for different propagation paths.

[0056] Time-frequency analysis is performed on the target speech signal to convert the continuous speech signal into a time-frequency feature representation, so that the energy distribution of speech in the time and frequency dimensions can be represented separately; based on the propagation delay, propagation direction and propagation energy characteristics of each propagation path in the local sound field model, path attribution analysis is performed on each time-frequency component in the target speech signal. Specifically, for each time-frequency component, the degree of matching between it and the propagation features of each propagation path is calculated, and the propagation path with the highest degree of matching is determined as the propagation source corresponding to the time-frequency component; all time-frequency components belonging to the same propagation path are aggregated to form the propagation component corresponding to the propagation path; where a propagation component refers to the speech component formed by the same propagation path in the target speech, and each propagation component corresponds to a propagation path, providing a processing object for subsequent targeted path compensation.

[0057] Based on the propagation relationship matrix, the degree of propagation correlation between each propagation path is read, and the influence of each propagation path on the distortion of the target speech is calculated by combining the propagation delay, propagation energy attenuation degree and reflection number of each propagation path. Taking the target speech propagation path as the benchmark propagation path, each reflection propagation path, background propagation path and dynamic propagation path are compared with the benchmark propagation path respectively, and the propagation delay, propagation energy and frequency response differences that need to be corrected for the corresponding propagation path are calculated. Then, the above correction parameters are established with the corresponding propagation paths to form a propagation compensation relationship.

[0058] Among them, the propagation compensation relationship refers to the set of compensation parameters corresponding to each propagation path. It is used to describe the time delay deviation, amplitude deviation and frequency response deviation that need to be compensated for the corresponding propagation path, and provides a specific compensation basis for subsequent independent compensation of each propagation component.

[0059] The propagation compensation relationship corresponding to each propagation component is read, and the corresponding time delay compensation parameters, amplitude recovery parameters, and frequency response compensation parameters are extracted respectively. The time position of the corresponding propagation component is adjusted according to the time delay compensation parameters to keep each propagation component synchronized on the time axis. The amplitude of the propagation component is corrected according to the amplitude recovery parameters to compensate for signal attenuation or abnormal enhancement during propagation. The energy distribution of the propagation component in different frequency ranges is adjusted according to the frequency response compensation parameters to restore the spectral distortion caused by reflection, absorption, or blockage during propagation. Each propagation component after compensation forms a corresponding propagation compensation signal. The propagation compensation signal refers to the speech signal after path-independent correction, which has acoustic characteristics that are closer to the real speech propagation state than the original propagation component.

[0060] First, the propagation feature information corresponding to each propagation path in the local sound field model is read, and the propagation delay feature, propagation energy feature, and frequency response feature corresponding to each propagation path are extracted respectively. Taking the target speech propagation path as the reference propagation path, the delay difference, amplitude difference, and frequency response difference of each reflection propagation path, background propagation path, and dynamic propagation path relative to the reference propagation path are calculated respectively. The propagation features corresponding to each propagation path are matched with the historical propagation features saved in the local sound field model, and the historical propagation state with the highest similarity is selected. The compensation parameters saved in the corresponding historical propagation state are read as the initial propagation compensation parameters. According to the difference between the current propagation features and the historical propagation features, the initial propagation compensation parameters are proportionally corrected to generate the delay compensation parameters, amplitude recovery parameters, and frequency response compensation parameters of the corresponding propagation path respectively, and the corresponding propagation compensation relationship is established.

[0061] Based on the propagation delay relationship of each propagation compensation signal, all propagation compensation signals are time-aligned to keep effective speech information from different propagation paths synchronized on a unified time axis. Different fusion weights are assigned according to the propagation reliability of each propagation compensation signal. The propagation reliability can be determined comprehensively based on the stability of the propagation path, the propagation distance, the degree of propagation energy retention, and the consistency of historical propagation. The propagation compensation signals after time alignment are weighted and fused according to their corresponding fusion weights to restore the continuous speech waveform. The fused continuous speech is smoothed to eliminate possible splicing abrupt changes between different propagation compensation signals to obtain the target compensated speech. The target compensated speech refers to the speech signal after compensation by the local sound field model, which can effectively reduce the effects of reverberation, reflection, and background noise in complex acoustic environments, and improve the clarity and integrity of the speech.

[0062] The propagation quality features corresponding to each propagation compensation signal are extracted. These features include at least signal energy retention rate, signal-to-noise ratio (SNR), propagation stability, and time delay consistency. The signal energy retention rate characterizes the degree to which the effective energy of the propagated signal is preserved relative to the original target speech energy. The SNR characterizes the energy ratio between the effective speech and background noise. The propagation stability characterizes the stability of the propagation characteristics of the corresponding propagation path within multiple consecutive time windows. The time delay consistency characterizes the degree of time synchronization between the propagated signal and the target speech propagation path. Each propagation quality feature is normalized to eliminate the influence between different feature dimensions, and the corresponding propagation quality score is calculated based on each propagation quality feature. All propagation quality scores are then normalized so that the sum of all propagation quality scores equals 1, and the normalized propagation quality scores are determined as the fusion weights corresponding to each propagation compensation signal. A mapping relationship is established between the fusion weights and the corresponding propagation compensation signals for subsequent weighted fusion of propagation compensation signals. The higher the propagation quality score, the greater the fusion weight of the corresponding propagation compensation signal in the final speech reconstruction process, thereby enabling the propagation compensation signal with better propagation quality to occupy a higher proportion in the target compensated speech, improving the clarity and recognition accuracy of the reconstructed speech.

[0063] Speech sorting is performed on all target compensated speech to obtain multiple target speech segments; the target speech segments are then semantically reorganized to generate target speech text. The target compensated speech is further segmented to obtain multiple continuous speech segments. The start time, end time, duration, and completeness of each speech segment are calculated, and the continuous propagation relationship between the speech segments is analyzed in conjunction with the target temporal correlation. The multiple speech segments are sorted according to the target temporal correlation to maintain the original temporal order of speech segments belonging to the same continuous speech expression process. For speech segments whose order overlaps due to noise blockage or changes in propagation path, their arrangement is readjusted according to the target temporal correlation, finally obtaining multiple target speech segments arranged according to the actual speech expression order, providing a reliable temporal order basis for subsequent semantic continuous reconstruction.

[0064] Speech recognition is performed on each target speech segment to obtain the corresponding initial text segment. Text continuity is established based on the temporal order of the target speech segments, and word connections, grammatical structures, and semantic relationships are extracted between adjacent text segments. Semantic continuity refers to the degree of coherence in grammatical and semantic expression between adjacent text segments, used to determine whether multiple text segments belong to the same complete sentence. Text segments with pauses, repetitions, or partial omissions are automatically joined according to the degree of semantic continuity. Contextual completion is performed on text breaks caused by propagation interference, and duplicate text content is deleted. All text segments after semantic continuity reconstruction are sequentially concatenated to generate the target speech text, thereby ensuring that the output text accurately reflects the user's original speech expression and improving the completeness and accuracy of speech recognition results in complex acoustic environments.

[0065] Generate corresponding device interaction commands based on the target speech text, and control the corresponding target device to perform the corresponding interactive operation.

[0066] Semantic recognition is performed on the target speech text to obtain several independent semantic texts; semantic compensation is performed on the independent semantic texts to obtain the corresponding device operation texts; device operation texts are matched with device instructions to obtain the corresponding device interaction instructions; The target speech text refers to the continuous text content obtained after acoustic compensation and speech recognition. It may contain multiple device control intentions or multiple continuous operation statements, so further semantic recognition is required.

[0067] Semantic recognition refers to the process of segmenting different semantic content in continuous text based on the grammatical structure, semantic relationships, and device control logic of sentences, so that each segmentation result corresponds to a relatively independent control intention. In the specific implementation process, the target speech text is first segmented into multiple word units. A semantic association network is established based on the grammatical dependencies between words, where each word is a semantic node, and semantic connections are established between adjacent words based on subject-predicate, verb-object, modification, and parallel relationships. Then, the semantic association degree between each semantic node is calculated. When the association degree between adjacent semantic nodes is consistently higher than a preset association threshold, they are classified into the same semantic segment. When the association degree is lower than the preset association threshold or a new action word, device name, or control object appears, it is determined that the semantics have switched, and a new semantic segment is generated. Finally, each semantic segment generates a corresponding independent semantic text. An independent semantic text is a text unit that can independently express a device operation intention. For example, turning on the living room light and turning off the air conditioner correspond to two independent semantic texts, providing processing objects for subsequent semantic compensation.

[0068] Semantic compensation refers to the automatic recovery of semantic loss caused by omissions, spoken expressions, recognition errors, or contextual references during voice interaction, so that the final text can fully express the user's true device control intentions. In the specific implementation process, firstly, a device control semantic template library is established. The device control semantic template library stores the standard control statement structure corresponding to different devices. Each standard control statement includes at least four types of semantic elements: operation object, control action, control parameters, and control target. Semantic elements are extracted from each independent semantic text and matched with the device control semantic template library to identify the semantic elements already contained in the current independent semantic text and the missing semantic elements. For missing semantic elements, inferences are made by combining the current conversation history, the contextual relationship between independent semantic texts before and after, and the status of the smart speaker's currently connected devices. For example, if the current voice command is "turn up a little," and the previous voice command clearly states that the controlled object is the air conditioner temperature, then the missing controlled object is automatically compensated to be the air conditioner temperature. After completing the supplementation of all missing semantic elements, the arrangement order of each semantic element is reorganized according to the device control semantic template to form device operation text that conforms to the device control specifications. Device operation text refers to standardized text that can directly map device operation behavior. Its semantic expression is complete, its structure is unified, and it is convenient for subsequent device command generation.

[0069] In this embodiment, the device interaction command refers to the standard control command that can be directly recognized and executed by the smart speaker or smart home device. It includes not only device control actions, but also the corresponding device identifier and execution parameters.

[0070] In the specific implementation process, firstly, a device command mapping library is established. This library stores the correspondence between each type of device, its identifier, control actions, control parameters, and the actual executed commands. Then, semantic parsing is performed on the device operation text, extracting the device name, control actions, and control parameters, and matching them item by item with the device information in the device command mapping library. When multiple candidate devices exist, the matching degree of each candidate device is calculated by combining the current online status of the device, its location, the most recent interaction record, and the user's historical operation habits, and the device with the highest matching degree is selected as the target device. Next, the corresponding control command template is read according to the device type of the target device, and the device identifier, control actions, and control parameters are filled into the corresponding template to generate standard device interaction commands. Finally, the generated device interaction commands are validated for completeness, determining whether the device identifier, control actions, and control parameters all meet the device control requirements. Once all validations pass, the standard device interaction commands are sent to the corresponding target device for execution, thus completing the automatic conversion process from natural language to device control commands for the smart speaker.

[0071] In this embodiment, all preset models are established by combining historical acoustic data training with on-site adaptive correction. Specifically, environmental sound signals are first continuously collected in the target application environment for no less than seven days, and the activity status of personnel, equipment operation status, and changes in indoor spatial layout are recorded simultaneously in different time periods. The environmental sound signals are segmented according to preset time windows, and sound energy distribution characteristics, spectrum distribution characteristics, spatial direction characteristics, acoustic reverberation characteristics, propagation delay characteristics, and propagation energy change characteristics are extracted to form a historical acoustic sample library. Then, the historical acoustic sample library is used to construct an initial local sound field model, a propagation path model, and a semantic recognition model, and the models are deployed in the smart speaker. During the actual operation of the system, new environmental sound data is continuously collected and compared with the model prediction results. When the prediction error exceeds the preset allowable range for multiple consecutive preset time windows, the model parameters are updated again using the latest collected data, so that each model can continuously adapt to changes in the indoor sound field and improve the long-term stability of the model.

[0072] In this embodiment, during the implementation of the technical solution, continuous speech data of multiple typical users in the target acoustic environment are collected, and the duration and rhythm changes of each speech segment are statistically analyzed. The average and standard deviation of the duration of all speech segments are calculated, and combined with the minimum processing time allowed by the speech recognition system, a time length that can fully cover most of the continuous speech features and meet the real-time processing requirements is selected as the preset time window. A fixed proportion of overlapping area is set between continuous time windows, and the overlap length is verified according to the continuous speech recognition accuracy. When the speech recognition accuracy reaches the highest level, the corresponding time window and overlap proportion are determined as the final parameters.

[0073] A training sample set is established, including spatial distance, sound energy distribution characteristics, spectral distribution characteristics, spatial direction characteristics, and acoustic reverberation characteristics. Based on manual annotation or historical operation results, it is determined which sound sampling points belong to the same local sound field. The correlation between various features and real local sound field labels is calculated, and the importance of each feature is determined based on the correlation. Then, all feature weights are normalized so that the sum of the weights equals 1. Cross-validation is performed using historical environmental data, and the weights of each feature are continuously adjusted until the accuracy of local sound field segmentation reaches the preset requirements. The corresponding weights are then determined as the final weight parameters.

[0074] A sample set is constructed using historical environmental sound data from the target acoustic environment, and the spatial correlation and directional matching degree between all sound sampling points are calculated. Samples belonging to the same local sound field and different local sound fields are statistically analyzed to obtain the numerical distribution of the two types of samples. Then, using classification accuracy as the evaluation index, local sound field division is performed under different threshold conditions, and the misclassification rate and omission rate are statistically analyzed. The spatial correlation and directional matching degree corresponding to the minimum comprehensive error and the highest local sound field division accuracy are determined as the preset correlation threshold.

[0075] After the system starts up, it first continuously collects background sounds in the target acoustic environment, calculates the average value and standard deviation of the short-time energy corresponding to the background sounds, and generates a speech detection threshold by superimposing a preset multiple of the standard deviation on the average background energy as a benchmark. This threshold is used to determine whether the user has started speaking. When the target speech is detected, the system then counts the short-time energy values ​​corresponding to all speech frames in the target speech signal sequence, calculates the energy distribution range, and determines the preset energy threshold based on the energy distribution range. When the energy of multiple consecutive speech frames is lower than the preset energy threshold, the user's speech is considered to have ended. During the operation of the device, the background noise statistics and speech energy statistics are continuously updated according to a sliding time window, and the two thresholds are dynamically corrected synchronously.

[0076] Multiple continuous speech samples are established based on historical target speech signal sequences, and the temporal correlation between adjacent target speech signals is calculated. Standard samples are established based on manually annotated real continuous speech relationships. The target temporal correlation relationships established under different threshold conditions are compared with the standard samples, and the continuous recognition accuracy is statistically analyzed. The temporal correlation corresponding to the highest continuous recognition accuracy and the lowest false correlation rate is determined as the preset correlation condition, and the threshold is periodically recalibrated using newly added speech data.

[0077] In this embodiment, since environmental sound signals, spatial coordinates, propagation delay, propagation energy, and frequency characteristics each have different data units, data unification processing is performed before all calculations. Specifically, the unit information of various types of raw data is first read, and all similar data are converted to a unified unit according to a preset unit conversion rule. Time is uniformly converted to milliseconds, spatial distance to meters, sound energy to decibels, frequency to hertz, and spatial direction to angles. After unit unification, outlier detection is performed on all data. When data exceeding a reasonable range is detected, interpolation correction is performed based on the corresponding data in adjacent time windows to prevent outliers from participating in subsequent calculations.

[0078] To eliminate the impact of differences in the magnitude of different physical quantities on the comprehensive calculation, normalization is performed on all data involved in the fusion calculation. Specifically, firstly, the maximum, minimum, and average values ​​of each data point in the historical samples are statistically analyzed. Then, range normalization or standardization is used to uniformly map each data point to a dimensionless numerical range. After that, the normalized data is used to complete the calculation of propagation relationships, identity association, propagation priority, and device matching. After all calculations are completed, for data with physical meaning in the final output, the units are restored to actual engineering units according to the corresponding unit conversion rules. For intermediate calculation results used only for internal model analysis, the dimensionless form is maintained for subsequent calculations. Through the above three steps of unit unification, normalization, and result restoration, it is ensured that data from different sources and with different units can be fused and analyzed within the same calculation framework, improving the stability, accuracy, and repeatability of the overall system's calculation results.

[0079] Example 2: An active interaction system for a smart speaker based on a complex acoustic environment (see [link]). Figure 1 As shown, it includes: The acoustic environment construction module includes a sound field division unit. The sound field division unit is used to continuously collect ambient sound signals from each sound sampling point in the target acoustic environment using a smart speaker. It combines all ambient sound signals to perform local sound field division on the target acoustic environment, dividing the target acoustic environment into several initial local sound field units. It performs temporal sound field state analysis on the initial local sound field units to obtain the corresponding sound field temporal labels. The speaker's active interaction module includes a signal extraction unit, an interaction compensation unit, and an instruction execution unit. The signal extraction unit acquires the target speech signal sequence in real time; extracts the signal direction features of the target speech signal sequence; matches the corresponding initial local sound field unit based on the signal direction features to obtain the target recognition sound field unit; and uses the sound field timing label corresponding to the target recognition sound field unit as the target sound field timing label. The interaction compensation unit constructs a local sound field model based on the target speech signal sequence acquired within the target recognition sound field unit; performs acoustic compensation processing on the target speech signal sequence using the local sound field model to obtain the target compensated speech sequence; and performs speech recognition based on the target compensated speech sequence to obtain the target speech text. The instruction execution unit generates corresponding device interaction instructions based on the target speech text to control the corresponding target device to perform the appropriate interactive operation.

[0080] It should be understood that those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims. Parts not described in detail in this specification are prior art known to those skilled in the art.

Claims

1. A method for active interaction of a smart speaker based on a complex acoustic environment, characterized in that, Includes the following steps: The system continuously collects ambient sound signals from various sound sampling points within the target acoustic environment using a smart speaker; it then combines all ambient sound signals to perform local sound field segmentation of the target acoustic environment, dividing it into several initial local sound field units; and finally, it performs temporal sound field state analysis on the initial local sound field units to obtain the corresponding temporal labels of the initial local sound field units. Real-time acquisition of target speech signal sequence; extraction of signal direction features of target speech signal sequence; matching of corresponding initial local sound field units based on signal direction features to obtain target recognition sound field units; Use the sound field temporal label corresponding to the target recognition sound field unit as the target sound field temporal label; A local sound field model is constructed based on the target speech signal sequence collected within the target recognition sound field unit; acoustic compensation processing is performed on the target speech signal sequence using the local sound field model to obtain a target compensated speech sequence; speech recognition is performed based on the target compensated speech sequence to obtain the target speech text. Generate corresponding device interaction commands based on the target speech text, and control the corresponding target device to perform the corresponding interactive operation.

2. The method for active interaction of a smart speaker based on a complex acoustic environment according to claim 1, characterized in that, The specific steps for dividing the target acoustic environment into local sound fields by combining all ambient sound signals include: The ambient sound signal corresponding to each sound sampling point is extracted based on a preset time window; feature extraction is performed based on the ambient sound signal to obtain the ambient acoustic feature vector; the ambient acoustic feature vector includes at least sound energy distribution features, spectrum distribution features, spatial direction features and acoustic reverberation features; the spatial location of each sound sampling point is obtained; By combining the spatial location of each sampling point with the environmental acoustic feature vector, the spatial correlation between any two sound sampling points is calculated; based on the spatial correlation and spatial direction features, multiple sound sampling points are spatially clustered to obtain several initial local sound field units.

3. The method for active interaction of a smart speaker based on a complex acoustic environment according to claim 2, characterized in that, The specific steps for performing temporal sound field state analysis on the initial local sound field unit include: The initial local sound field units corresponding to each preset time window are traversed according to the preset time order. For any two adjacent preset time windows, the temporal correlation degree between the corresponding initial local sound field units is calculated. The sound field mapping relationship between the corresponding initial local sound field units is established based on the temporal correlation degree. Based on the sound field mapping relationship, the corresponding quantitative relationship between each initial local sound field unit is statistically analyzed: when one initial local sound field unit is mapped to multiple initial local sound field units, the split mapping relationship is satisfied; when multiple initial local sound field units are mapped to one initial local sound field unit, the fusion mapping relationship is satisfied; when one initial local sound field unit is mapped to only one initial local sound field unit, the continuous mapping relationship is satisfied. Based on the split mapping relationship, fusion mapping relationship and continuous mapping relationship, the corresponding sound field temporal label is matched for the initial local sound field unit.

4. The method for active interaction of a smart speaker based on a complex acoustic environment according to claim 3, characterized in that, The specific steps for constructing a local sound field model based on the target speech signal sequence collected within the target recognition sound field unit include: The target speech signal sequence is segmented according to a preset recognition window to obtain several target speech signals; the target temporal correlation between different target speech signals is extracted; based on the target temporal correlation and the signal direction features corresponding to the target speech signals, the propagation source and propagation direction of the target speech signals are identified; the target speech propagation path and the reflection propagation path associated with the target speech propagation path are established based on the propagation source and propagation direction. Propagation state analysis is performed on the target speech propagation path and reflection propagation path to obtain the background propagation path and dynamic propagation path; the features of the target speech propagation path, reflection propagation path, background propagation path and dynamic propagation path are fused to generate a local sound field model.

5. The method for active interaction of a smart speaker based on a complex acoustic environment according to claim 4, characterized in that, The specific steps for performing acoustic compensation processing on the target speech signal sequence using a local sound field model to obtain the target compensated speech sequence, and for performing speech recognition based on the target compensated speech sequence to obtain the target speech text, include: In the local sound field model, the propagation relationships between different propagation paths are identified; feature components are extracted from the target speech signal to obtain the propagation components corresponding to each propagation path; propagation compensation relationships are established based on the propagation relationships corresponding to the propagation paths; each propagation component is independently compensated based on each propagation compensation relationship to obtain each propagation compensation signal; each propagation compensation signal is reconstructed to obtain the target compensated speech. Speech sorting is performed on all target compensated speech to obtain multiple target speech segments; the target speech segments are then semantically reorganized to generate target speech text.

6. The method for active interaction of a smart speaker based on a complex acoustic environment according to claim 5, characterized in that, Semantic recognition is performed on the target speech text to obtain several independent semantic texts; semantic compensation is performed on the independent semantic texts to obtain the corresponding device operation texts; device operation texts are matched with device instructions to obtain the corresponding device interaction instructions.

7. A smart speaker active interaction system based on a complex acoustic environment, characterized in that, The system employs the active interaction method for smart speakers based on complex acoustic environments as described in any one of claims 1-6, comprising: The acoustic environment construction module includes a sound field division unit. The sound field division unit is used to continuously collect ambient sound signals from each sound sampling point in the target acoustic environment using a smart speaker. It combines all ambient sound signals to perform local sound field division on the target acoustic environment, dividing the target acoustic environment into several initial local sound field units. It performs temporal sound field state analysis on the initial local sound field units to obtain the corresponding sound field temporal labels. The speaker's active interaction module includes a signal extraction unit, an interaction compensation unit, and an instruction execution unit. The signal extraction unit acquires the target speech signal sequence in real time; extracts the signal direction features of the target speech signal sequence; matches the corresponding initial local sound field unit based on the signal direction features to obtain the target recognition sound field unit; and uses the sound field timing label corresponding to the target recognition sound field unit as the target sound field timing label. The interaction compensation unit constructs a local sound field model based on the target speech signal sequence acquired within the target recognition sound field unit; performs acoustic compensation processing on the target speech signal sequence using the local sound field model to obtain the target compensated speech sequence; and performs speech recognition based on the target compensated speech sequence to obtain the target speech text. The instruction execution unit generates corresponding device interaction instructions based on the target speech text to control the corresponding target device to perform the appropriate interactive operation.

Images