A voice instruction conflict processing method of a child interactive toy

Abstract

This invention relates to the field of interactive toy technology, specifically to a method for handling voice command conflicts in children's interactive toys. The method includes: acquiring audio signals from the same time period using a microphone array; determining the corresponding sound source location information based on each audio signal; performing audio separation processing on the sound source location information to obtain audio streams for each sound source; performing speech recognition and semantic analysis processing on each audio stream to obtain the command content and emotional state of each sound source; calculating priority based on each command content, emotional state, and preset user information to obtain a priority score for each sound source; sorting the sound sources according to the priority evaluation to obtain a response sequence; executing the command content of each sound source sequentially according to the response sequence; and controlling directional lighting based on the corresponding sound source location information to project light at the sound source location to indicate the current service target. This method effectively solves the voice command conflicts present in existing children's interactive toys.

Description

Technical Field

[0001] This invention relates to the field of interactive toy technology, and specifically to a method for handling voice command conflicts in children's interactive toys. Background Technology

[0002] With the increasing popularity of smart toys, the voice command conflict handling of interactive toys for children plays an important role in improving the fairness, sense of order, and user experience of interaction in multi-child families or group play scenarios.

[0003] Currently, common interactive toys for children typically use a first-come, first-served strategy based on time sequence or a strategy based on the maximum volume of the signal strength to determine the response target when handling voice commands issued by multiple users simultaneously. However, this approach cannot effectively separate and locate the different children corresponding to the concurrent voice commands in the acoustic space, resulting in a chaotic interaction process, a lack of reasonable basis for response decisions, and children's inability to clearly perceive the current service target, which seriously affects the experience in multi-user concurrent interaction scenarios. Summary of the Invention

[0004] To address the technical problem of voice command conflicts in existing interactive toys for children, this application provides a method for handling voice command conflicts in interactive toys for children.

[0005] The voice command conflict resolution method for a children's interactive toy provided in this application adopts the following technical solution: A method for handling voice command conflicts in a children's interactive toy, characterized by comprising: After acquiring audio signals from the same time period using a microphone array, the corresponding sound source location information is determined based on each audio signal. After performing audio separation processing on the sound source location information to obtain the audio streams of each sound source, speech recognition processing and semantic analysis processing are performed on each audio stream to obtain the instruction content and emotional state of each sound source. Priority is calculated based on the content of each instruction, each emotional state, and preset user information. After obtaining the priority score of each sound source, the sound sources are sorted according to the priority evaluation to obtain the response order sequence. According to the response sequence, the instructions of each sound source are executed in turn, and directional lighting is controlled according to the corresponding sound source location information to project light at the sound source location to indicate the current service object.

[0006] Furthermore, the steps for determining the corresponding sound source location information based on each audio signal include: After pairwise time delay estimation of each audio signal to obtain multiple time delay differences, spectral peak search is performed in the preset sound field space based on the multiple time delay differences to generate a spatial spectrum representing the probability intensity of each sound source. Local peak detection and cluster analysis are performed on the spatial spectrum to obtain spatial coordinates, which are then used as the sound source location information.

[0007] Furthermore, the steps for obtaining spatial coordinates by performing local peak detection and cluster analysis on the spatial spectrum include: Based on a preset judgment threshold, the spatial spectrum is filtered to obtain multiple candidate peak points and the spatial coordinates and energy values ​​corresponding to each candidate peak point; Based on the spatial coordinates, the energy distribution within a preset three-dimensional domain centered on each candidate peak point is extracted from the spatial spectrum. Then, based on the energy distribution, the features of the main lobe and side lobes are calculated to obtain the feature vectors corresponding to each candidate peak point. Similarity is calculated and groups are formed for each feature vector to obtain several cluster sets; The weighted centroids are calculated based on the spatial coordinates and energy values ​​of the corresponding candidate peak points included in each cluster set to obtain the spatial coordinates corresponding to each cluster set.

[0008] Furthermore, the steps for calculating the weighted centroids based on the spatial coordinates and energy values ​​of the corresponding candidate peak points included in each cluster set, to obtain the spatial coordinates corresponding to each cluster set, include: Nonlinear normalization is performed on each energy value in each cluster set to obtain the weight coefficient of each candidate peak point; Based on the weighting coefficients, the spatial coordinates of the candidate peak points in the corresponding cluster set are weighted and summed to obtain the weighted centroid coordinates. Based on the spatial coordinate distribution of each candidate peak point in each cluster set, the principal direction analysis and anisotropic covariance matrix calculation are performed to obtain the elliptical model parameters of the spatial distribution characteristics of each cluster set. Based on the parameters of the elliptical model, the distribution direction of the weighted centroid coordinates is corrected and offset. After obtaining the corrected weighted centroid coordinates, the corrected weighted centroid coordinates are determined as the corresponding spatial coordinates.

[0009] Furthermore, the steps of performing speech recognition and semantic analysis on each audio stream to obtain the instruction content and emotional state of each sound source include: The Mel frequency cepstral coefficients and fundamental frequency trajectories are extracted synchronously from each audio stream to obtain speech content features and paralinguistic features; The speech content features are decoded and word sequences are generated to obtain the text sequence of the corresponding audio stream; By performing dependency parsing and semantic role labeling on the text sequence, the core actions, objects of action, and constraints in the text sequence are parsed to form instruction content. Furthermore, by classifying the temporal patterns of paralinguistic features, sentiment category labels and sentiment dimension values ​​are obtained, thus constituting the sentiment state.

[0010] Furthermore, the steps for calculating the priority of each sound source based on the content of each instruction, each emotional state, and preset user information include: The instruction content is subjected to keyword matching and intent classification to obtain a quantitative value of the urgency of the instruction content and the category of the interaction intent. By using a preset emotion-weight mapping table, the emotion category labels and emotion dimension values ​​in the emotion state are transformed to obtain the emotion modulation coefficients of each sound source; Based on the pre-set user information, the user's age attribute, historical interaction frequency and identity tag are weighted and calculated to obtain the user's basic weight for each sound source. Based on the urgency quantification value and interaction intent category, the emotion modulation coefficient and user basic weight are fused and calculated to obtain the initial priority score of each sound source. Then, the initial priority score is corrected based on the historical interaction frequency to obtain the priority score.

[0011] Furthermore, the step of controlling directional lighting based on the corresponding sound source location information to project light at the sound source location to indicate the current service target includes: The sound source location information is mapped from the microphone array coordinate system to the light array coordinate system to obtain the target projection coordinates in the light array coordinate system. Control the horizontal and vertical deflection angles of at least one directional light source unit in the light array so that the optical axis of the directional light source unit is aligned with the target projection coordinates; Based on the current service target, the corresponding indication mode is selected from the preset mode library, and a corresponding pulse width modulation signal is generated according to the indication mode. Then, the directional light source unit is driven by the pulse width modulation signal to project light at the sound source position to indicate the current service target.

[0012] Beneficial effects achieved: This application provides a voice command conflict handling method for children's interactive toys, including: after collecting audio signals in the same time period through a microphone array, determining the corresponding sound source location information based on each audio signal; performing audio separation processing on the sound source location information to obtain the audio stream of each sound source, performing speech recognition processing and semantic analysis processing on each audio stream to obtain the command content and emotional state of each sound source; performing priority calculation based on each command content, each emotional state and preset user information to obtain the priority score of each sound source, sorting each sound source according to the priority evaluation to obtain a response order sequence; executing the command content of each sound source in sequence according to the response order sequence, and performing directional light control according to the corresponding sound source location information to project light at the sound source location to indicate the current service object.

[0013] In this application, audio signals are acquired through a microphone array to determine the location information of the sound sources. This physically separates the mixed voices from different children, laying the physical foundation for subsequent independent processing. Next, audio separation is performed based on the sound source location information to obtain audio streams from each source. Speech recognition and semantic analysis are then performed on each audio stream to obtain the instruction content and emotional state of each source. This allows for the interpretation of each child's specific intentions and emotional state from the audio signals, achieving a transformation from physical signals to semantic and emotional information. Then, based on the instruction content and emotional state of each source, combined with pre-set user information, a comprehensive priority calculation is performed to obtain a priority score. This score is then used to generate a response sequence. This process integrates multiple dimensions such as the urgency of the instruction, the user's emotional needs, and individual identity, thereby achieving a fairer and more reasonable intelligent decision-making process. Finally, instructions are executed sequentially according to the response order, and directional lighting is controlled simultaneously based on the corresponding sound source location information. Lights are projected onto the corresponding sound source locations to indicate the current service target. In this way, the internal decision-making results are presented in real time and intuitively through external visual signals that are precisely bound to the sound source location. This allows all participants to clearly know the current interaction target, thereby eliminating the problem of chaotic response caused by the inability to distinguish users, the single decision-making basis, and the lack of effective feedback at the technical level. This achieves orderly, fair, and intuitive multi-person voice interaction. Attached Figure Description

[0014] Figure 1 This is a flowchart illustrating the steps of a voice command conflict resolution method for a children's interactive toy according to this application. Figure 2 This is a flowchart illustrating the steps of obtaining the corresponding sound source location information based on the audio signal in this application. Figure 3 This is a flowchart illustrating the steps involved in obtaining the instruction content and emotional state of each sound source based on the audio stream in this application. Figure 4 A flowchart illustrating the steps for calculating the priority score of the sound source in this application; Figure 5 This is a flowchart illustrating the steps of this application to transmit light indications to the current service object based on the sound source location information. Detailed Implementation

[0015] The following combination Figures 1 to 5 This application will be described in further detail.

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0017] It should be noted that if the embodiments of the present invention involve directional indications (such as up, down, left, right, front, back, etc.), the directional indications are only used to explain the relative positional relationship and movement of the components in a specific posture. If the specific posture changes, the directional indications will also change accordingly.

[0018] Furthermore, if the embodiments of this invention involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the use of "and / or" or "and / or" throughout the text includes three parallel solutions. For example, "A and / or B" includes solution A, solution B, or a solution where both A and B are satisfied simultaneously. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this invention.

[0019] This application discloses a method for handling voice command conflicts in children's interactive toys.

[0020] Please refer to Figure 1 The voice command conflict handling method for children's interactive toys proposed in this embodiment includes steps S10 to S40: Step S10: After acquiring audio signals from the same time period through the microphone array, determine the corresponding sound source location information based on each audio signal.

[0021] By capturing and distinguishing different concurrent sound sources at the physical space level at the beginning of the processing flow, a spatial information foundation is provided for the entire conflict resolution method. Utilizing the spatial perception capability of the microphone array, a preliminary association is established between the collected mixed sound signals and specific spatial directions or locations. This enables the initial separation and marking of different children's voice points at the signal source. In other words, this embodiment does not process an indistinguishable mixed sound field, but rather obtains the source coordinates of each sound component in space. This allows all subsequent processing to be carried out based on the dimension of "where the sound comes from," laying a data foundation for subsequent audio separation, content understanding, and personalized feedback for each independent sound source.

[0022] It should be noted that a microphone array consisting of multiple microphone units arranged in a specific geometric configuration in space is used to synchronously acquire sound field data. Each microphone unit operates on the same time reference, and can simultaneously sense air vibrations in the sound field and convert sound pressure signals into corresponding analog electrical signals. Then, through a synchronous sample-and-hold circuit and an analog-to-digital converter, the analog electrical signals of each microphone unit are converted into audio signals with strictly aligned timestamps, thus obtaining a set of multiple audio signals corresponding to the same time period and having a definite spatial relationship and precise time synchronization relationship with each other.

[0023] Step S20: Perform audio separation processing on the sound source location information to obtain the audio stream of each sound source. Then, perform speech recognition processing and semantic analysis processing on each audio stream to obtain the instruction content and emotional state of each sound source.

[0024] By transforming the sound source location information obtained in the previous step into understandable and processable instruction content and emotional state, the leap from "perceiving where the sound comes from" to "understanding the content and emotion of each sound" is completed.

[0025] First, based on the sound source location information, the audio stream corresponding to each sound source is separated from the audio signal collected by the microphone array. This allows subsequent processing to be performed on the clear speech of individual speakers, creating conditions for high-precision recognition. Then, speech recognition is performed on each separated audio stream to convert it from an audio waveform into text-based instruction content. Semantic analysis is performed simultaneously to analyze its core intent and structure. At the same time, pattern analysis is performed on the acoustic features of the audio stream to identify its emotional state. This yields the structured and emotionally rich specific interactive intent corresponding to each concurrent sound source. In other words, the spatially separated individual sound sources are further endowed with instruction content and emotional state, providing input with clear semantic and emotional meaning for subsequent steps to make priority decisions based on multi-dimensional information.

[0026] It should be noted that the sound source location information, based on the direction or coordinates of each sound source in space determined in the previous step, is achieved through beamforming technology. Specifically, in the system's preset coordinate system, the three-dimensional spatial coordinates of each microphone unit in the microphone array are known and fixed. Simultaneously, the sound source location information determined in step S10 provides the three-dimensional spatial coordinates of each sound source within the same coordinate system. Based on this, the straight-line distance from the sound source to each microphone unit is calculated. This calculation is performed using the Euclidean distance formula in three-dimensional space, that is, using the coordinate difference between two points to obtain their straight-line distance. After obtaining the straight-line distance from the sound source to each microphone unit, the straight-line distance is converted into a time difference, i.e., absolute propagation delay, based on the speed of sound in the air. At the same time, a reference point is selected, which can be an actual microphone unit in the microphone array. The absolute propagation delay from the sound source to the reference point is calculated in the same way as above. Subsequently, the absolute propagation delay from the sound source to each other microphone unit is subtracted from the absolute propagation delay from the sound source to the reference point. The result is the relative time delay value required for each microphone unit relative to the reference point. This relative time delay value directly reflects the difference in the time of arrival of the sound wave at each microphone unit due to the different propagation path lengths.

[0027] Next, based on the calculated relative time delay value of each microphone unit, a fractional time delay filtering operation is applied to the audio signal of the corresponding microphone channel. This operation is achieved by setting a finite-length unit impulse response filter, whose frequency response is approximately a linear phase. This accurately shifts the audio signal of each microphone channel forward or backward on the time axis in the digital domain by the corresponding sampling point interval to compensate for the arrival time difference caused by the difference in the propagation path of the sound wave, so that the audio signal components from the sound source are phase aligned on all channels. Subsequently, a weighted summation operation is performed on the signals of each channel that have undergone time delay compensation. That is, the audio signal of each microphone channel is multiplied by a preset complex weight value and then accumulated. The weight value is set based on a specific beamforming criterion, which aims to maximize the gain of the audio signal from the sound source while suppressing interference from other directions. Finally, through the combination of the above time delay compensation and weighted summation operations, a spatial filter with clear spatial directivity is constructed at the signal processing level. Its maximum response direction strictly corresponds to the position of the sound source. The maximum gain direction of this spatial filter is precisely aligned with the position of the corresponding sound source, while suppressing interference from other directions and environmental noise. This yields the audio stream for each sound source. This process is applied sequentially to each located sound source position, thus separating the audio stream corresponding to each sound source from the audio signal mixed from multiple microphone channels.

[0028] Step S30: Priority calculation is performed based on the content of each instruction, each emotional state and preset user information. After obtaining the priority score of each sound source, the sound sources are sorted according to the priority evaluation to obtain the response order sequence.

[0029] Building upon the physical separation and semantic understanding of concurrent audio, a decision-making mechanism incorporating multiple dimensions is introduced to determine the execution order of voice commands. This step integrates and quantifies three types of heterogeneous information obtained in previous steps: command content representing different children's interactive intentions, emotional states reflecting their emotional states, and pre-set user information reflecting individual differences. This generates a priority score that comprehensively weighs the urgency of the command, the intensity of emotional expression, and the user's identity priority. All sound sources are then ranked according to this priority score. This approach departs from the simplistic decision-making logic of traditional methods that rely solely on a single signal feature (such as time or volume). Instead, it automatically generates a reasonable and interpretable response sequence based on the semantic importance of the command content, the child's emotional state, and the pre-set user information. This ensures that in conflict scenarios where multiple children simultaneously issue voice commands, the response behavior of this implementation is orderly and conforms to expected interaction ethics, providing a direct decision-making basis for ultimately achieving a fair and smooth multi-user interactive experience.

[0030] It should be noted that the priority score corresponding to each sound source is calculated as the key value for sorting. A comparison operation is performed on all sound sources. This comparison operation is based on the numerical value of the priority score and sorts all sound sources in descending order from high to low. When two or more sound sources have the same priority score, they are sorted according to the order in which the sound sources were located. This ensures that a definite total order relationship can be generated under any circumstances, resulting in an ordered response sequence.

[0031] Step S40: According to the response sequence, execute the instructions of each sound source in sequence, and perform directional lighting control according to the corresponding sound source location information to project light at the sound source location to indicate the current service object.

[0032] The system sequentially calls and executes the control logic or content playback corresponding to the instruction content of each sound source according to the order specified by the response sequence. This achieves a serialized and orderly response to concurrent conflicting instructions in the time dimension, avoiding response overlap and chaos. At the same time, when the instruction content of each sound source begins to be executed, the system synchronously projects a beam of light onto the spatial location of the sound source by controlling a directional light source unit based on the location information of the sound source. This provides a visual indication in the spatial dimension that is clearly associated with the currently executed action. This not only completes the logical sequential processing of multi-user instructions, but also clearly announces the current service object by combining the response object with a light indicator visible in the physical space. This makes the entire interaction process transparent, orderly, and spatially directional for children, fundamentally solving the confusion and interruption of the waiting experience caused by not knowing which system is responding to in multi-user scenarios, and achieving the unity of decision-making and feedback.

[0033] In one feasible implementation, refer to Figure 2 As shown, step S10 may specifically include steps S11 to S12: Step S11: Perform pairwise time delay estimation on each audio signal to obtain multiple time delay differences. Then, perform spectral peak search in the preset sound field space based on the multiple time delay differences to generate a spatial spectrum representing the probability intensity of each sound source.

[0034] First, for each pair of unique microphone units, after acquiring the two audio signals they collect, pre-emphasis processing is applied to each audio signal. This involves using a first-order high-pass filter to boost the high-frequency components of the audio signal. The transfer function of this first-order high-pass filter is typically expressed as: (in, It is the pre-weighting factor, and its value is between 0.9 and 0.97; This is a unit delay operator, which means delaying the audio signal by one sampling interval in audio signal processing. This processing can compensate for the attenuation of high-frequency components in the audio signal during acquisition or transmission. The pre-emphasized audio signal is then framed, that is, the continuous time-domain signal is divided into a series of short-time overlapping frames. The length of each frame is usually 20 to 40 milliseconds, and the frame shift is usually half the frame length to ensure the continuity between frames. Then, by multiplying the audio signal samples in the frame with the window function coefficients point by point, the two ends of the frame are smoothly transitioned to zero, thereby reducing the spectral leakage effect caused by the discontinuity of the frame boundaries when performing the Fast Fourier Transform in the future. After completing the above pre-emphasis, framing and windowing steps, the two pre-processed audio signals are obtained, which enhance the high-frequency components and reduce spectral leakage.

[0035] Next, a Fast Fourier Transform (FFT) is performed on each frame of the audio signal to convert the time-domain signal to the frequency domain, obtaining the complex spectra corresponding to the two audio signals. The cross-power spectral density (CPS) of these two complex spectra is calculated, which is the pointwise product of the complex conjugates of one complex spectrum and the other. Then, a specific weighting function is applied to the CPS, which is achieved by dividing the CPS by the amplitude spectrum of the CPS, thus obtaining the weighted CPS. Finally, an Inverse Fast Fourier Transform (IFFT) is performed on the weighted CPS to transform it back to the time domain, yielding the generalized cross-correlation function corresponding to the microphone unit pair. The peak position of this generalized cross-correlation function on the time delay axis represents the relationship between the two audio signals. To estimate the relative time delay difference, the above process is repeated for all non-repeating microphone unit pairs to obtain multiple generalized cross-correlation functions corresponding to all microphone unit pairs. By traversing the entire generalized cross-correlation function, the index number corresponding to the point with the largest function value is found. This index number represents the relative delay of the two audio signals in a sampling unit. Since the generalized cross-correlation function has been weighted to highlight the correlation peak corresponding to the true time delay, its peak position directly reflects the main time alignment offset between the audio signals. Multiplying the obtained peak index number by the system's sampling time interval converts it into a time delay difference in units of time. Repeating this operation for all microphone unit pairs yields multiple time delay differences.

[0036] Next, a dense grid of points is defined within the pre-defined three-dimensional sound field space. For each grid point, based on the geometric relationship between the grid point and each pair of microphone units in the microphone array, the theoretical time delay difference of sound propagation from that grid point to that pair of microphone units is calculated. Subsequently, the generalized cross-correlation function of each microphone unit pair is taken at its corresponding theoretical time delay difference. Since the peak value of the generalized cross-correlation function reflects the correlation of the audio signal at that time delay, this value represents the matching strength between the actual audio signal and the theoretical time delay. The values ​​of all microphone unit pairs are summed to obtain a total accumulated value, which represents the sum of the time delay difference set and the theoretical time delay difference. The consistency degree of the set can be used to evaluate the probability that the grid point is a real sound source location. The consistency degree is assigned to the grid point array as the energy value of each grid point. After traversing all grid points in the three-dimensional sound field space and completing the assignment, a three-dimensional spatial spectrum is obtained. The energy value of each grid point represents the probability intensity of the sound source being located at the grid point. This realizes the transformation of the time delay difference between discrete microphone pairs into a spatial spectrum that can cover the entire space to be detected. Thus, the sound source localization problem is transformed into the problem of finding the most likely point on the spatial spectrum, providing a global data foundation for subsequent determination of sound source location information.

[0037] It should be noted that window function coefficients refer to a set of numerical sequences pre-calculated and stored at discrete time points for, for example, a Hamming window. The length of this numerical sequence is the same as the length of an audio signal frame, and each value represents the magnitude weight of the window function at that point in the audio signal. For example, for a Hamming window of length N, the nth window function coefficient... The calculation formula is: The value of n ranges from 0 to N-1. During windowing, each audio signal x(n) within a frame is compared with the window function coefficient at the corresponding index n. Perform point-by-point multiplication to obtain the windowed audio signal. .

[0038] Step S12: Perform local peak detection and cluster analysis on the spatial spectrum to obtain spatial coordinates, and use the spatial coordinates as the sound source location information.

[0039] This step transforms a spatial spectrum, including noise and background interference, into specific sound source location information. Specifically, local peak detection identifies points in the spatial spectrum with energy values ​​significantly higher than their local neighbors. These points correspond to the locations where sound wave energy converges most concentratedly, i.e., candidate locations where sound sources exist. However, due to factors such as reverberation, noise, or algorithm resolution, a real sound source may generate multiple adjacent peak points in the spectrum, forming peak clusters. Therefore, further cluster analysis is needed to group these multiple neighboring peak points belonging to the same sound source into one class and calculate their weighted centroids to eliminate redundant spurious peaks. This process also converges multiple responses corresponding to a sound source into a single most representative spatial coordinate, outputting one or more precise spatial coordinates. Each spatial coordinate corresponds to the sound source location information of an identified sound source, thus completing the transformation from audio signal to final sound source localization result.

[0040] Furthermore, step S12 also includes steps S121 to S124: Step S121: Based on a preset judgment threshold, the spatial spectrum is filtered to obtain multiple candidate peak points and the spatial coordinates and energy values ​​corresponding to each candidate peak point.

[0041] First, a preset threshold for judging energy value is set. This preset threshold is usually determined in advance based on the background noise level, signal strength, or empirical value.

[0042] Next, the energy values ​​of all grid points in the spatial spectrum are traversed, and the energy value of each grid point is compared with a preset judgment threshold. Only when the energy value of a grid point is greater than or equal to the preset judgment threshold is the grid point retained and marked as a candidate peak point. For each retained candidate peak point, its corresponding three-dimensional spatial coordinates (i.e., the position of the candidate peak point in the spatial spectrum) and its energy value are recorded. This is to initially filter out low-energy response areas in the spatial spectrum caused by noise, reverberation, or calculation errors, thereby reducing the amount of data that needs to be processed for subsequent peak detection and cluster analysis, and improving the significance of the peak points corresponding to the real sound source, laying the foundation for accurate identification of the sound source location.

[0043] Step S122: Based on each spatial coordinate, extract the energy distribution within a preset three-dimensional domain centered on each candidate peak point in the spatial spectrum, and then perform feature calculations on the main lobe and side lobes based on the energy distribution to obtain the feature vectors corresponding to each candidate peak point.

[0044] First, based on the spatial coordinates of each candidate peak point obtained through screening, in the generated spatial spectrum, with the three-dimensional coordinates of each candidate peak point as the center, extract the energy values ​​of all grid points within a three-dimensional neighborhood of a preset size (e.g., a cubic region with a side length of several grid units), thereby obtaining the local energy distribution of the candidate peak point. When calculating the features of the main lobe and side lobes based on this energy distribution, it should be noted that the main lobe refers to the energy value corresponding to the candidate peak point itself, while the side lobes refer to other energy values ​​existing in the energy distribution of the three-dimensional neighborhood besides the energy value corresponding to the candidate peak point itself. The specific process of feature calculation includes, but is not limited to: calculating the ratio of the main lobe energy value to the average of all side lobe energy values ​​to measure the prominence of the main lobe; calculating the ratio of the main lobe energy value to the maximum... The ratio of sidelobe energy values ​​is used to measure attenuation characteristics; the number of sidelobes is counted; and the spatial distance distribution between each sidelobe and the main lobe is calculated. Combining these calculated feature parameters forms a multidimensional feature vector. This feature vector quantitatively describes the local morphological characteristics of the energy distribution centered on the candidate peak point. This generates a digital description containing local energy structure information for each candidate peak point, thus transforming an energy point on the spatial spectrum into a set of quantifiable features. This set of features can effectively distinguish the energy distribution generated by real sound sources, which usually has a sharp main lobe and weak side lobes, from pseudo-peaks generated by noise, reverberation, or other interference, which often have multiple strong side lobes or indistinct main lobes. This provides a basis for subsequent clustering analysis or classification decisions.

[0045] Step S123: Calculate the similarity of each feature vector and group them based on the similarity to obtain several cluster sets.

[0046] By calculating the Euclidean distance between two feature vectors, the smaller the Euclidean distance, the more similar the candidate peak points represented by the two feature vectors are in terms of energy distribution. Based on all the calculated pairwise Euclidean distances, all candidate peak points are grouped. For example, a density-based spatial clustering algorithm is applied to group candidate peak points in space whose Euclidean distance to each other is less than a preset neighborhood radius into the same cluster. Through this process, all candidate peak points are divided into several cluster sets with high internal similarity and low mutual similarity. This automatically identifies and groups multiple candidate peak points with high similarity in local energy distribution features extracted in the previous step into the same group, laying the foundation for the next step of calculating the spatial coordinates of each sound source.

[0047] Step S124: Calculate the weighted centroid based on the spatial coordinates and energy values ​​of the corresponding candidate peak points included in each cluster set to obtain the spatial coordinates corresponding to each cluster set.

[0048] Weighted centroids are calculated based on the spatial coordinates and energy values ​​of the corresponding candidate peak points included in each cluster set. By using the energy value as a weighting factor, a weighted average is performed on the spatial coordinates of all candidate peak points within the cluster set. This ensures that candidate peak points with higher energy values ​​have a greater weight in the final coordinate calculation. In this way, while integrating multiple candidate peak points, the interference of noise points or side lobes with lower energy is reduced. A unique and more robust spatial coordinate is derived for each cluster set representing a potential sound source, improving the accuracy of sound source location estimation and its robustness to noise. This provides a foundation for subsequent sound source identification, tracking, or output.

[0049] Furthermore, step S124 also includes steps S1241 to S1244: Step S1241: Perform nonlinear normalization on each energy value in each cluster set to obtain the weight coefficient of each candidate peak point.

[0050] First, the energy values ​​of all candidate peak points within a cluster are combined into an energy set. Then, a nonlinear monotonically increasing function, such as an exponential function, is applied to each energy value in this energy set. This amplifies the relative differences between different energy values ​​through exponential operations with the natural constant e as the base and the energy value as the exponent, significantly boosting high energy values ​​and relatively suppressing low energy values. Next, the energy values ​​after nonlinear transformation are summed to obtain a total. Finally, the nonlinearly transformed energy value corresponding to each candidate peak point is divided by this total, and the resulting value is the weight coefficient of the corresponding candidate peak point. At the same time, a normalization operation is performed so that the sum of the weight coefficients of all candidate peak points within the same cluster is 1. The magnitude of each weight coefficient not only reflects the relative level of the energy value of the candidate peak point, but also further strengthens the contribution ratio of high-energy points through nonlinear transformation.

[0051] Step S1242: Based on the weight coefficients, the spatial coordinates of the candidate peak points in the corresponding cluster set are weighted and summed to obtain the weighted centroid coordinates.

[0052] Each coordinate component (e.g., X, Y, Z components) of the spatial coordinates of each candidate peak point in the cluster set is multiplied by the weight coefficient corresponding to that candidate peak point to obtain the weighted coordinate components of that candidate peak point. Then, the weighted X coordinate components, weighted Y coordinate components, and weighted Z coordinate components of all candidate peak points in the same cluster set are summed to obtain three sums. These three sums constitute the weighted centroid coordinates. By incorporating the energy value into the average calculation of the spatial coordinates in the form of weights, the weighted centroid coordinates represented by the calculation result are no longer a simple geometric center, but are more biased towards the spatial region where high-energy candidate peak points are clustered.

[0053] Step S1243: Based on the spatial coordinate distribution of each candidate peak point in each cluster set, perform principal direction analysis and anisotropic covariance matrix calculation to obtain the elliptical model parameters of the spatial distribution characteristics of each cluster set.

[0054] Let the weighted centroid coordinates be The spatial coordinates of each candidate peak point are: Calculate the deviation vector of the spatial coordinates of each candidate peak point relative to the weighted centroid coordinates, i.e. Next, the weighted covariance matrix is ​​calculated using the weighting coefficients. Each element of this weighted covariance matrix is ​​composed of the weighted sum of the products of the corresponding deviation vectors. Specifically, the (1,1) element C_xx of the matrix equals... The element C_xy in the matrix (1,2) equals The element C_xz in the matrix (1,3) equals The element C_yy in the matrix (2,2) equals The element C_yz at (2,3) of the matrix is ​​equal to The element C_zz at (3,3) of the matrix is ​​equal to Based on the symmetry of the covariance matrix, let C_y = C_xy, C_zx = C_xz, and C_zy = C_yz, thus obtaining a 3x3 three-dimensional covariance matrix. Each element of this three-dimensional covariance matrix represents the covariance relationship and dispersion of the point set along different coordinate axes. It is the weight coefficient of the i-th candidate peak point.

[0055] Next, eigenvalue decomposition is performed on the three-dimensional covariance matrix to obtain three eigenvalues ​​and their corresponding eigenvectors. Specifically, the three eigenvalues ​​are found... , , and the corresponding three feature vectors , , Such that the product of the three-dimensional covariance matrix C and each eigenvector is equal to the product of that eigenvalue and the eigenvector, i.e., satisfying... To achieve this decomposition, it is necessary to solve the characteristic equation. ,in, It is a 3-order identity matrix. The characteristic equation represents the determinant operation of a matrix, and it is a function of the matrix determinant. The cubic polynomial equation is solved by finding its characteristic equation, yielding three eigenvalues ​​arranged in descending order. ≥ ≥ For each obtained eigenvalue, substitute it into the homogeneous linear equation system. In this process, by solving the homogeneous linear equation system, a non-zero solution vector is obtained. Due to the symmetry of the covariance matrix, these non-zero solution vectors are orthogonal to each other. Finally, each non-zero solution vector is divided by its corresponding length to obtain the eigenvectors corresponding to the eigenvalues.

[0056] Among them, the eigenvector corresponding to the largest eigenvalue indicates the direction in which the point set is most dispersed in space, i.e., the principal direction. The other two eigenvectors are orthogonal to the principal direction, and together they define the three mutually perpendicular principal axes describing the distribution of the point set. The directions of these eigenvectors define the three principal axis directions of the elliptical model, and the square root of the eigenvalue is proportional to the length of the corresponding principal axis, i.e., the decomposition yields three eigenvalues. , , and the corresponding three feature vectors , , Then, the ellipse model parameters are composed of these two parts: the directions of the three eigenvectors directly define the three principal axes of the ellipse in space; and the lengths of the three principal axes (i.e., the semi-axis lengths of the ellipse) are proportional to the square root of the corresponding eigenvalues, usually calculated as follows: , , , where k is a constant related to confidence or scaling factor, together forming the elliptical model parameters that describe the spatial distribution characteristics of the cluster set. By quantitatively obtaining the directional characteristics and anisotropy of the spatial distribution of candidate peak points within each cluster set, a precise geometric basis is provided for the next step of correcting the weighted centroid coordinates based on physical distribution characteristics.

[0057] It should be noted that the length in "non-zero solution vector divided by its corresponding length" is For a non-zero solution vector v, its components are represented as The length of a non-zero solution vector is defined as the distance from the origin (0,0,0) to the point... According to the Pythagorean theorem in three-dimensional space, the straight-line distance is equal to the square root of the sum of the squares of the components along each coordinate axis. Therefore, its calculation formula is: .

[0058] Step S1244: Based on the elliptical model parameters, the distribution direction of the weighted centroid coordinates is corrected and offset. After obtaining the corrected weighted centroid coordinates, the corrected weighted centroid coordinates are determined as the corresponding spatial coordinates.

[0059] Using the parameters of the elliptic model, based on the degree of dispersion of the point set along each principal axis (proportional to...) This involves applying a small offset to the weighted centroid coordinates along one or more principal axes, for example, calculating the points along the principal direction where the distribution is most dispersed (corresponding to the largest eigenvalue). eigenvectors The centroid offset on the axis can be along the axis. Direction, and A certain proportional value, the sign of which can be determined based on the specific distribution (such as skewness) of candidate peak points in that direction, is used to adjust the weighted centroid coordinates towards regions with higher point density. Next, this calculated offset is vector-added with the weighted centroid coordinates to obtain the corrected weighted centroid coordinates. These corrected weighted centroid coordinates are then used as the spatial coordinates corresponding to the cluster set. By incorporating the anisotropic geometric characteristics of the point set's spatial distribution, the weighted centroid is fine-tuned based on its physical distribution, thereby improving the accuracy of the located spatial coordinates when the point set distribution is non-uniform or elongated.

[0060] In one feasible implementation, refer to Figure 3 As shown, step S20 may specifically include steps S21 to S23: Step S21: Perform synchronous extraction operations on Mel frequency cepstral coefficients and fundamental frequency trajectories for each audio stream to obtain speech content features and paralinguistic features.

[0061] Each continuous audio stream signal is preprocessed by framing and windowing to obtain a series of short-duration, stable audio stream signal frames.

[0062] For each frame of the audio stream signal, two parallel processing flows are executed simultaneously: The first flow is used to extract Mel frequency cepstral coefficients. The steps include performing a fast Fourier transform on each frame of the audio stream signal to obtain the amplitude spectrum, passing the amplitude spectrum through a set of triangular bandpass filters uniformly distributed on the Mel frequency scale, taking the logarithm of the output energy of each triangular bandpass filter, and finally performing a discrete cosine transform on the obtained logarithmic energy sequence. The first few coefficients are taken to form a Mel frequency cepstral coefficient vector representing the short-time spectral envelope features of the audio stream signal. This Mel frequency cepstral coefficient vector is the speech content feature.

[0063] The second process is used to extract the fundamental frequency estimate, and its steps include processing the audio stream signal sequence obtained after framing and windowing. (Where n = 0, 1, ..., N-1, and N is the frame length), calculate its autocorrelation function at time delay m. Its calculation formula is This calculation reflects the similarity between the audio stream signal and its own delay. When the audio stream signal is periodic, the autocorrelation function will peak at a position where the delay is an integer multiple of the signal period; after calculating... Then, within a preset reasonable fundamental frequency range (this range is determined by the upper and lower limits of the possible speaker's fundamental frequency, for example, 70Hz to 350Hz, and converted into the corresponding time delay range), a search is performed. The first significant peak position m_peak corresponds to an integer multiple (usually 1, i.e., T0 itself) of the fundamental frequency period T0 of the audio stream signal. Therefore, the estimated fundamental frequency period of that frame of the audio stream signal is... (in (where is the sampling rate), and its reciprocal is the estimated fundamental frequency corresponding to that frame of audio stream signal. The fundamental frequency estimate is the secondary language feature.

[0064] By processing each frame of the audio stream signal sequentially, the two processes described above output a time-varying sequence of Mel frequency cepstral coefficients and a time-varying sequence of fundamental frequency estimates, respectively. These two time-aligned sequences together constitute the speech content features and paralinguistic features extracted synchronously from the corresponding audio streams. This ensures that the representation of "what to say" and the representation of "how to say" have strict synchronicity and frame-level correspondence in the time dimension, providing consistent and correlated underlying feature inputs for subsequent steps of speech recognition and sentiment analysis.

[0065] Step S22: Decode the speech content features and generate word sequences to obtain the text sequence of the corresponding audio stream.

[0066] The Mel-frequency cepstral coefficient sequence corresponding to the language content features is used as input and fed into a trained speech recognition system. This speech recognition system first defines a deep neural network model (a nonlinear computational model composed of multiple hidden layers). During modeling, the Mel-frequency cepstral coefficients of each frame and the context features of several frames before and after are concatenated into a fixed-dimensional vector as input to the deep neural network model. After nonlinear transformation through multiple hidden layers of the deep neural network model, this vector is mapped to a probability distribution vector in the output layer through a Softmax function. Each dimension of this vector corresponds to a phoneme state to be modeled (e.g., the start, middle, and end states of a phoneme). Its value represents the posterior probability that the Mel-frequency cepstral coefficients of the current frame belong to a specific phoneme state given the current input features. The entire deep neural network model is trained in a supervised manner using a large amount of labeled speech data (i.e., audio signals and their corresponding phoneme labels for each frame). The cross-entropy loss between the network output probability and the real label is minimized through the backpropagation algorithm, thereby learning the accurate mapping relationship from the Mel-frequency cepstral coefficient feature sequence to the posterior probability of the phoneme state.

[0067] Then, the posterior probability of the phoneme state is used as the acoustic observation probability. The decoder uses a pre-defined pronunciation dictionary (a predefined mapping table that maps each word in the vocabulary to its corresponding phoneme sequence, thereby establishing a correspondence between words and phonemes) to convert the candidate word sequence into a phoneme state sequence hypothesis. Combined with the prior probability of the word sequence provided by the neural network language model (a model trained through a neural network structure to calculate the joint probability or conditional probability of the word sequence to evaluate the linguistic rationality of the word sequence), the cumulative probability of each path is calculated through dynamic programming in the search space consisting of all possible word sequences, phoneme state sequences, and observation sequences. This cumulative probability is jointly determined by the acoustic observation probability and the prior probability of the word sequence. The optimal path and its probability for each time frame and each state are iteratively retained. Finally, at the end of the search, the path with the highest cumulative probability is selected. The word sequence corresponding to this path is the word sequence with the highest matching degree with the input Mel frequency cepstral coefficient sequence, thus obtaining the text sequence.

[0068] It should be noted that during the training phase of the deep neural network model, the Mel-frequency cepstral coefficient feature sequence is used as the input for forward propagation. This means the data sequentially passes through the input layer, multiple hidden layers undergoing nonlinear transformations (such as using the ReLU activation function), and the output layer's Softmax function. This results in the output probability distribution of each frame's Mel-frequency cepstral coefficient features belonging to various phoneme states. Then, the cross-entropy loss between this output probability distribution and the corresponding phoneme state is calculated. The cross-entropy loss function measures the difference between the output probability distribution and the phoneme state; a smaller value indicates more accurate model predictions. Finally, the backpropagation algorithm is used to calculate the cross-entropy loss across all... The gradients of the weights and bias parameters are calculated. The backpropagation algorithm, based on the chain rule, propagates the error backward layer by layer from the output layer, calculating the partial derivative of the loss with respect to the parameters of each layer. After obtaining the gradients, the stochastic gradient descent algorithm is used to iteratively update the model parameters according to the gradient direction and learning rate to reduce the cross-entropy loss value. By repeating the forward propagation, loss calculation, backpropagation, and parameter update process on the entire training dataset, the parameters of the deep neural network model are continuously adjusted, so that its output probability distribution gradually approximates the true phoneme state distribution. This allows the model to learn the accurate mapping relationship from the Mel frequency cepstral coefficient feature sequence to the posterior probability of the phoneme state, ultimately enabling the model to accurately predict the posterior probability of the corresponding phoneme state for a new Mel frequency cepstral coefficient feature sequence.

[0069] Step S23 involves performing dependency parsing and semantic role labeling on the text sequence to parse out the core actions, objects of action, and constraints in the text sequence, forming instruction content. Additionally, by classifying the temporal patterns of paralinguistic features, sentiment category labels and sentiment dimension values ​​are obtained, thus constituting the sentiment state.

[0070] By inputting a text sequence into a deep learning-based natural language processing model, the model first performs word segmentation and part-of-speech tagging on the text sequence, then performs dependency parsing to identify grammatical dependencies between words and construct a dependency tree. For example, a transformation-based parser is used to determine the subject-verb-object structure. Next, semantic role tagging is performed to identify the predicates in the text sequence and their related semantic roles such as agent, patient, time, place, and instrument, thereby extracting core actions such as verb phrases, objects such as noun phrases, and constraints such as modifiers or prepositional phrases, and integrating them into structured instruction content.

[0071] Meanwhile, by inputting the fundamental frequency estimation sequence of paralinguistic features into a recurrent neural network classification model, this model learns the dynamic patterns in the fundamental frequency estimation sequence and outputs the probability distribution of emotional category labels such as happy, sad, or angry through fully connected layers and a softmax function. It also outputs continuous values ​​of emotional dimensions such as arousal, valence, and dominance through a regression layer. This enables the system to accurately parse executable instructions and perceive the user's emotional state from the user's voice input, thereby achieving human-computer interaction that integrates semantic understanding and emotional perception, and improving the system's context adaptability and response accuracy.

[0072] In one feasible implementation, refer to Figure 4 As shown, step S30 may specifically include steps S31 to S34: Step S31: Perform keyword matching and intent classification on the instruction content to obtain the urgency quantification value and interaction intent category of the instruction content.

[0073] The command text is scanned and matched using a pre-defined dictionary containing keywords corresponding to different urgency levels. For example, a string matching algorithm is used to identify high-urgency keywords such as "immediately," "as soon as possible," and "urgent," or low-urgency keywords such as "later" and "when available." A pre-defined weight value is assigned to each matched keyword. If frequency is taken into account, the weight of each keyword is multiplied by the number of times it appears in the command content or its usage frequency for weighting. Finally, the weight values ​​of all keywords are summed or averaged to obtain a quantitative value of urgency. The keyword weight values ​​are usually calculated using statistical methods such as TF-IDF or manually set based on experience.

[0074] Simultaneously, the instruction text is input into a pre-trained intent classification model. This intent classification model is typically a classifier based on a deep neural network architecture such as Transformer. By analyzing the semantics of the instruction content, it maps the instruction content to predefined interactive intent categories such as "device control," "information query," "setting modification," or "help request." This provides structured and quantitative input regarding the urgency and purpose of the instruction for subsequent priority calculation, enabling the system to make preliminary distinctions and assessments based on the semantics and urgency attributes of the instruction itself. The intent classification model is a deep learning model trained on a large amount of labeled data, used to automatically identify the predefined intent category to which the input text belongs.

[0075] Step S32: By using a preset emotion-weight mapping table, the emotion category label and emotion dimension value in the emotion state are converted to obtain the emotion modulation coefficient of each sound source.

[0076] It should be noted that the preset emotion-weight mapping table is a predefined lookup table that maps emotion category labels and emotion dimension values ​​to specific numerical coefficients. Structurally, it consists of two core parts. The first part is a direct mapping relationship, predefining a corresponding basic emotion weight value for each emotion category label. For example, "anger" is mapped to a higher basic weight, and "calm" is mapped to a lower basic weight. During operation, the basic emotion weight value is obtained by directly looking up the table based on the identified emotion category label. The second part is the adjustment rule for emotion dimension values. This adjustment rule defines how to use the specific values ​​of emotion dimension values ​​to correct the basic emotion weight values. For example, it stipulates that for every unit increase in the emotion dimension value, the basic emotion weight value is multiplied by a coefficient greater than one to obtain the emotion adjustment coefficient. This enables dynamic adjustment of the processing priority of voice commands from different sound sources based on the user's real-time emotional state, thereby improving the intelligence and thoughtfulness of human-computer interaction.

[0077] Step S33: Based on the preset user information, perform weighted calculations on the age attribute, historical interaction frequency, and identity tags of the corresponding user to obtain the basic user weight of each sound source.

[0078] It should be noted that the pre-set user information is pre-stored static attributes and dynamic behavioral data about each user, mainly including age attributes, historical interaction frequency, and identity tags.

[0079] First, the information in the three dimensions of age attribute, historical interaction frequency, and identity tag is numerically processed. For example, an age grouping rule is predefined, such as dividing age into children, teenagers, adults, and seniors, and a preset value is assigned to each age group as the age attribute value. During operation, the user's actual age is directly mapped to the corresponding age attribute value based on the group they fall into. The number of interactions of the user within a specific time period is counted as the historical interaction frequency. Then, through min-max normalization, the historical interaction frequency is linearly transformed to the standard range of 0 to 1, thereby obtaining a normalized frequency score as the historical interaction frequency value. A predefined identity tag-weight mapping table is maintained, in which a fixed identity tag value is assigned to each possible identity tag such as "administrator", "VIP user" or "ordinary user". During operation, the corresponding identity tag value is directly obtained by looking up the table based on the user's identity tag.

[0080] Then, a weight is preset for each dimension. Using a weighted summation formula, such as User Basic Weight = (Age Attribute Weight * Age Attribute Value) + (Historical Interaction Frequency Weight * Historical Interaction Frequency Value) + (Identity Tag Weight * Identity Tag Value), the user basic weight corresponding to each sound source is calculated. Based on the individual characteristics of users, different basic priorities are assigned to different users, so that the system can implement personalized response strategies in subsequent priority ranking, prioritize the handling of important users or high-frequency users' instructions, and improve the fairness and efficiency of the overall interaction.

[0081] Step S34: Based on the urgency quantification value and the interaction intent category, the emotion adjustment coefficient and the user basic weight are fused and calculated to obtain the initial priority score of each sound source. Then, the initial priority score is corrected based on the historical interaction frequency to obtain the priority score.

[0082] The urgency quantification value is used as a multiplicative factor, and the corresponding intent weight coefficient is found by using the interaction intent category as the query key through a pre-maintained interaction intent category-weight mapping table (which assigns a fixed value to each predefined interaction intent category as its corresponding intent weight coefficient). Then, the sentiment adjustment coefficient, user base weight, intent weight coefficient, and urgency quantification value are multiplied by a formula to obtain the initial priority score, specifically: Initial priority score = urgency quantification value * intent weight coefficient * sentiment adjustment coefficient * user base weight.

[0083] Next, a decay factor related to historical interaction frequency is introduced. This decay factor decreases as the historical interaction frequency increases, specifically through an exponential decay function. The initial priority score is multiplied by this decay factor, thereby moderately suppressing the scores of high-frequency interaction users and obtaining a corrected initial priority score, i.e., the priority score. By integrating and balancing multiple dimensions from instruction content, user sentiment, user static attributes, and dynamic behavior, and by suppressing high-frequency interaction users to prevent a single user from monopolizing system resources, a dynamic, fair, and comprehensive priority score is obtained, providing a direct and reasonable decision-making basis for the final response ranking.

[0084] In one feasible implementation, refer to Figure 5 As shown, step S40 may specifically include steps S41 to S43: Step S41: Map the sound source location information from the microphone array coordinate system to the light array coordinate system to obtain the target projection coordinates on the light array coordinate system.

[0085] This is achieved through a pre-calibrated three-dimensional spatial coordinate transformation matrix, which describes the rotation and translation relationship between the microphone array coordinate system and the light array coordinate system. The sound source location information in the microphone array coordinate system is represented in homogeneous coordinate form, that is, from a three-dimensional coordinate point. Expanded into a four-dimensional vector The superscript T indicates transpose. This three-dimensional spatial coordinate transformation matrix is ​​a 4x4 homogeneous transformation matrix. The 3x3 submatrix in its upper left corner (i.e., the region consisting of the nine elements from the first row to the third row and the first column to the third column of the homogeneous transformation matrix) represents the rotation transformation from the microphone array coordinate system to the light array coordinate system. The first three elements on its rightmost side (i.e., the first three numerical elements from top to bottom in the fourth column of the homogeneous transformation matrix) represent the translation vector from the origin of the microphone array coordinate system to the origin of the light array coordinate system.

[0086] When performing matrix multiplication, a 4x4 homogeneous transformation matrix is ​​multiplied by a four-dimensional column vector representing the sound source location information. The operation rule is that each row of the homogeneous transformation matrix is ​​multiplied by the corresponding element of the four-dimensional column vector, and then the results are summed to obtain a new four-dimensional result vector. The first three elements of this new four-dimensional result vector are the target projection coordinates in the light array coordinate system. Its fourth element is theoretically 1, and is discarded after the calculation is completed. This calculation process is mathematically equivalent to first applying a rotation transformation to the original three-dimensional coordinates, and then applying a translation transformation, thereby realizing the mapping of the coordinate system. This provides spatial coordinate input for directly controlling the deflection angle of the light unit in subsequent steps, ensuring that the light beam can accurately point to the actual physical location of the sound source.

[0087] Step S42: Control the horizontal and vertical deflection angles of at least one directional light source unit in the light array so that the optical axis of the directional light source unit is aligned with the target projection coordinates.

[0088] In the light array coordinate system, the fixed installation coordinates of the directional light source unit are known three-dimensional coordinate points, denoted as [reference needed]. The target projection coordinates are three-dimensional coordinate points obtained after mapping in the previous step, denoted as [reference needed]. During calculation, each component of the target projection coordinates is subtracted from the corresponding component of the fixed installation coordinates of the directional light source unit to obtain the three-dimensional direction vector. .

[0089] Next, based on the three-dimensional direction vector In the coordinate system of the light array, the horizontal plane is defined as the xy plane, and the vertical direction as the z-axis. Then, the azimuth component in the horizontal direction and the pitch component in the vertical direction are calculated. Specifically, the azimuth component is determined by finding the angle between the projection of the three-dimensional direction vector onto the horizontal plane and the positive x-axis, using the arctangent function. To calculate this angle, the arctangent function determines the correct quadrant based on the signs of dx and dy, thus obtaining the azimuth component; the pitch component is obtained by determining the angle between the three-dimensional direction vector itself and the horizontal plane, using the arcsine function. Or the arctangent function, where It is the magnitude of the three-dimensional direction vector, i.e. The calculated azimuth component is the required horizontal deflection angle, and the calculated pitch component is the required vertical deflection angle.

[0090] Finally, the control circuit generates corresponding drive signals (such as pulse width modulation signals or stepper motor control pulses) to drive the horizontal and vertical rotation motors of the directional light source unit, so that they rotate to the calculated horizontal and vertical deflection angles, respectively, so that the optical axis of the directional light source unit is aligned with the target projection coordinates in three-dimensional space.

[0091] Step S43: Select the corresponding indication mode from the preset mode library according to the current service object, generate the corresponding pulse width modulation signal according to the indication mode, and drive the directional light source unit to project light at the sound source position to indicate the current service object through the pulse width modulation signal.

[0092] First, obtain the identification information of the current service object (such as user ID or priority category). Then, use this identification information as an index to query the preset pattern library. The preset pattern library is a pre-configured database or file that stores the mapping relationship between different service object identifiers and specific indication pattern parameters. Therefore, the indication pattern associated with the current service object can be obtained by querying and matching.

[0093] The indicator mode defines the visual characteristic parameters of the light projection, such as color value, brightness level, flicker frequency or dynamic mode. Based on these visual characteristic parameters, the corresponding pulse width modulation signal waveform is calculated. For example, color control may involve generating independent pulse width modulation signals for red, green and blue LEDs to mix the target color. Brightness or flicker control is achieved by adjusting the signal duty cycle and period. The generation process is usually completed by a microcontroller or a dedicated hardware PWM generator.

[0094] The generated pulse width modulation signal is output to the driving circuit of the directional light source unit. The driving circuit controls the light intensity, color and timing of the light source according to the pulse width modulation signal, so that the directional light source unit, which has been aligned with the target projection coordinates in the previous step, projects a light pattern that conforms to the indication pattern. This light pattern forms a visual mark at the sound source location, clearly indicating the object that is currently being responded to or served. This provides differentiated and customizable visual feedback for different service objects. By intuitively distinguishing service targets through preset diverse light patterns, the clarity of human-computer interaction and user experience are enhanced.

[0095] The above are all preferred embodiments of this application, and are not intended to limit the scope of protection of this application. Therefore, all equivalent changes made in accordance with the structure, shape and principle of this application should be covered within the scope of protection of this application.

Claims

1. A voice instruction conflict processing method of a child interactive toy, characterized by, include: After acquiring audio signals from the same time period using a microphone array, the corresponding sound source location information is determined based on each audio signal. After performing audio separation processing on the sound source location information to obtain the audio stream of each sound source, speech recognition processing and semantic analysis processing are performed on each audio stream to obtain the instruction content and emotional state of each sound source. Priority calculations are performed based on the content of each instruction, the emotional state of each instruction, and preset user information to obtain priority scores for each sound source. Then, the sound sources are sorted according to the priority evaluation to obtain a response order sequence. According to the response sequence, the instructions of each sound source are executed in sequence, and directional lighting control is performed according to the corresponding sound source location information to project light at the sound source location to indicate the current service object.

2. The voice instruction conflict processing method of the child interactive toy according to claim 1, characterized in that, The step of determining the corresponding sound source location information based on each of the audio signals includes: After pairwise time delay estimation of each audio signal to obtain multiple time delay differences, spectral peak search is performed in a preset sound field space based on the multiple time delay differences to generate a spatial spectrum characterizing the probability intensity of each sound source. Local peak detection and cluster analysis are performed on the spatial spectrum to obtain spatial coordinates, which are then used as the location information of the sound source.

3. The voice instruction conflict processing method of the child interactive toy according to claim 2, characterized in that, The steps of performing local peak detection and cluster analysis on the spatial spectrum to obtain spatial coordinates include: The spatial spectrum is filtered based on a preset judgment threshold to obtain multiple candidate peak points and the spatial coordinates and energy values ​​corresponding to each candidate peak point; Based on the spatial coordinates, after extracting the energy distribution in a preset three-dimensional region centered on each candidate peak point in the spatial spectrum, the feature calculation of the main lobe and side lobe is performed based on the energy distribution to obtain the feature vectors corresponding to each candidate peak point. The similarity of each feature vector is calculated and grouped to obtain several cluster sets; The spatial coordinates of each cluster set are obtained by calculating the weighted centroid based on the spatial coordinates and energy values ​​of the corresponding candidate peak points included in each cluster set.

4. The voice instruction conflict processing method of the child interactive toy according to claim 3, characterized in that, The step of calculating the weighted centroid based on the spatial coordinates and energy values ​​of the corresponding candidate peak points included in each cluster set to obtain the spatial coordinates corresponding to each cluster set includes: The energy values ​​in each of the cluster sets are nonlinearly normalized to obtain the weight coefficients of each of the candidate peak points; Based on the weighting coefficients, the spatial coordinates of the candidate peak points in the corresponding cluster set are weighted and summed to obtain the weighted centroid coordinates. Based on the spatial coordinate distribution of each candidate peak point in each cluster set, the principal direction analysis and anisotropic covariance matrix calculation are performed to obtain the elliptical model parameters of the spatial distribution characteristics of each cluster set. Based on the ellipse model parameters, the distribution direction of the weighted centroid coordinates is corrected and offset to obtain the corrected weighted centroid coordinates, and the corrected weighted centroid coordinates are then determined as the corresponding spatial coordinates.

5. The voice instruction conflict processing method of the child interactive toy according to claim 1, characterized in that, The step of performing speech recognition processing and semantic analysis on each of the audio streams to obtain the instruction content and emotional state of each sound source includes: The Mel frequency cepstral coefficients and fundamental frequency trajectories of each audio stream are extracted synchronously to obtain speech content features and paralinguistic features; The speech content features are decoded and word sequences are generated to obtain the text sequence of the corresponding audio stream; By performing dependency parsing and semantic role labeling on the text sequence, the core actions, objects of action, and constraints in the text sequence are parsed out to form instruction content. Furthermore, by classifying the temporal patterns of the paralinguistic features, sentiment category labels and sentiment dimension values ​​are obtained to constitute the sentiment state.

6. The voice instruction conflict processing method of the child interactive toy according to claim 1, characterized in that, The step of calculating the priority of each sound source based on the content of each instruction, the emotional state of each sound source, and preset user information includes: The instruction content is subjected to keyword matching and intent classification operations to obtain a quantitative value of the urgency of the instruction content and the category of the interaction intent. By using a preset emotion-weight mapping table, the emotion category labels and emotion dimension values ​​in the emotion states are converted to obtain the emotion modulation coefficients of each sound source; Based on the preset user information, the age attribute, historical interaction frequency and identity tag of the corresponding user are weighted and calculated to obtain the user basic weight of each sound source. Based on the urgency quantification value and the interaction intent category, the emotion adjustment coefficient and the user basic weight are fused and calculated to obtain the initial priority score of each sound source. Then, the initial priority score is corrected based on the historical interaction frequency to obtain the priority score.

7. The voice instruction conflict processing method of the child interactive toy according to claim 1, characterized in that, The step of performing directional lighting control based on the corresponding sound source location information to project light at the sound source location to indicate the current service object includes: The sound source location information is mapped from the microphone array coordinate system to the light array coordinate system to obtain the target projection coordinates on the light array coordinate system. Control the horizontal and vertical deflection angles of at least one directional light source unit in the light array so that the optical axis of the directional light source unit is aligned with the target projection coordinates; Based on the current service object, a corresponding indication mode is selected from the preset mode library, and a corresponding pulse width modulation signal is generated according to the indication mode. Then, the directional light source unit is driven by the pulse width modulation signal to project light at the sound source position to indicate the current service object.

Images