Lamplight interaction method and device based on audio data recognition and storage medium

Abstract

The invention is suitable for the technical field of computers, and provides a light interaction method and device based on audio data recognition and a storage medium, and the light interaction method based on audio data recognition comprises the steps: obtaining first audio data output by a first device and second audio data output by a target user; performing attention coding on the first text data and the second text data to obtain text coding data; performing similarity weighted coding on the first acoustic data and the second acoustic data to obtain acoustic coded data; obtaining audio state recognition information according to a pre-trained audio state recognition model; and controlling a light parameter of the second equipment according to the audio state identification information. According to the method, the stability is ensured, flexible response can be realized according to the emotion of the user and the environment change, the lamplight can provide more accurate and more personalized response in different environments and situations, the lamplight interaction effect is improved, and the more intelligent and humanized interaction experience is realized.

Description

Technical Field

[0001] This application belongs to the field of computer technology, and in particular relates to a light interaction method, device and storage medium based on audio data recognition. Background Technology

[0002] With the development of artificial intelligence technology and advancements in audio processing, speech recognition, and natural language processing, the interactive experience of smart devices has become increasingly rich and complex. In intelligent interactive systems, existing lighting control is still relatively simple, adapting to the environment by relying on fixed audio templates or adjusting solely based on real-time user commands. However, this approach lacks responsiveness to changes in user emotions and context, making the interaction feel stiff, inflexible, and unintelligent, resulting in poor performance. Summary of the Invention

[0003] This application provides a light interaction method, device, and storage medium based on audio data recognition, which can solve the above-mentioned problems.

[0004] In a first aspect, embodiments of this application provide a light interaction method based on audio data recognition, comprising: Acquire first audio data output by a first device and second audio data output by a target user; wherein, the first audio data includes first acoustic data and first text data, and the second audio data includes second acoustic data and second text data; Attention encoding is performed on the first text data and the second text data to obtain text-encoded data; The first acoustic data and the second acoustic data are weighted by similarity to obtain acoustic coded data; Based on the text-encoded data, acoustic-encoded data, and the pre-trained audio state recognition model, audio state recognition information is obtained; Based on the audio status recognition information, the lighting parameters of the second device are controlled; wherein the second device is linked with the first device.

[0005] Further, the attention encoding of the first text data and the second text data to obtain text-encoded data includes: Obtain the first text representation corresponding to the first text data and the second text representation corresponding to the second text data, and obtain the syntactic representation corresponding to the first text data; Multi-head interactive attention encoding is performed on the syntactic representation corresponding to the first text data and the first text representation to obtain the first text encoded representation; Multi-head interactive attention encoding is performed on the syntactic representation corresponding to the first text data and the second text representation to obtain the second text encoded representation; The text encoded data is obtained based on the first text encoding representation and the second text encoding representation.

[0006] Further, obtaining the syntactic representation corresponding to the first text data includes: Extract the text feature representation corresponding to the first text representation; Obtain the temporal adjacency matrix corresponding to the first text data, and obtain the temporal weighted graph corresponding to the first text data; wherein, the temporal adjacency matrix is ​​the adjacency relationship between text nodes in the temporal dimension of the first text data, and the temporal weighted graph includes the probability of the text node being passed to the adjacent text node in the temporal dimension of the first text data; Based on the text feature representation corresponding to the first text representation, the temporal adjacency matrix corresponding to the first text data, the temporal weighted graph corresponding to the first text data, and the pre-trained syntactic extraction model, the syntactic representation corresponding to the first text data is obtained.

[0007] Further, obtaining the time-series weighted graph corresponding to the first text data includes: Obtain the adjacency type between the text nodes in the temporal adjacency matrix; Extract the adjacency type representation corresponding to the adjacency type between the text nodes in the temporal adjacency matrix; Based on the text feature representation corresponding to the first text representation, the adjacency type representation corresponding to the adjacency type between the text nodes in the temporal adjacency matrix, and the preset weighted calculation method, a temporal weighted graph corresponding to the first text data is obtained.

[0008] Further, the step of performing similarity-weighted encoding on the first acoustic data and the second acoustic data to obtain acoustic encoded data includes: Obtain the first acoustic representation corresponding to the first acoustic data and the second acoustic representation corresponding to the second acoustic data; Extract the first state information corresponding to the first acoustic representation and the second state information corresponding to the second acoustic representation; Analyze the acoustic state similarity between the first state information and the second state information; Based on the acoustic state similarity, the first acoustic representation and the second acoustic representation are weighted by similarity to obtain the acoustic coded data; Further, the extraction of the first state information corresponding to the first acoustic representation and the second state information corresponding to the second acoustic representation includes: The first acoustic representation and the second acoustic representation are input into a pre-trained acoustic state recognition model to obtain first emotional state information corresponding to the first acoustic representation and second emotional state information corresponding to the second acoustic representation; wherein, the pre-trained acoustic state recognition model includes several acoustic state recognition modules, each of which includes a sub-module for processing the temporal features of the acoustic representation, a sub-module for processing the frequency features of the acoustic representation, and a self-attention encoding module.

[0009] Furthermore, the second device comprises several bulbs linked to the first device. The audio state recognition information includes intensity state information and emotional state information. Controlling the lighting parameters of the second device based on the audio state recognition information includes: Based on the intensity status information, control the brightness and flicker parameters of several of the bulb lamps; Based on the emotional state information, the color temperature parameters of several of the bulb lamps are controlled.

[0010] Secondly, embodiments of this application provide a light interaction device based on audio data recognition, comprising: The acquisition unit is used to acquire first audio data output by a first device and second audio data output by a target user; wherein the first audio data includes first acoustic data and first text data, and the second audio data includes second acoustic data and second text data; The first processing unit is used to perform attention encoding on the first text data and the second text data to obtain text-encoded data. The second processing unit is used to perform similarity-weighted encoding on the first acoustic data and the second acoustic data to obtain acoustic encoded data; The third processing unit is used to obtain audio state recognition information based on the text encoded data, acoustic encoded data and the pre-trained audio state recognition model; The fourth processing unit is used to control the lighting parameters of the second device based on the audio status recognition information; wherein the second device is linked with the first device.

[0011] Thirdly, embodiments of this application provide a light interaction device based on audio data recognition, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the method described in the first aspect above.

[0012] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect above.

[0013] In this embodiment, first audio data output by a first device and second audio data output by a target user are acquired; attention encoding is performed on the first and second text data to obtain text-encoded data; similarity-weighted encoding is performed on the first and second acoustic data to obtain acoustic-encoded data; audio state recognition information is obtained based on the text-encoded data, acoustic-encoded data, and a pre-trained audio state recognition model; and the lighting parameters of the second device are controlled based on the audio state recognition information. This method effectively enhances the interactive capabilities of smart devices, ensuring stability while flexibly responding to user emotions and environmental changes. It enables lighting to provide more accurate and personalized responses in different environments and situations, improving the interactive effect of lighting and achieving a more intelligent and humanized interactive experience. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is a schematic flowchart of the light interaction method based on audio data recognition provided in the first embodiment of this application; Figure 2 This is a schematic flowchart of S102 in the light interaction method based on audio data recognition provided in the first embodiment of this application; Figure 3 This is a schematic flowchart of steps S10211 to S10213 in the light interaction method based on audio data recognition provided in the first embodiment of this application; Figure 4 This is a schematic flowchart of S103 in the light interaction method based on audio data recognition provided in the first embodiment of this application; Figure 5 This is a schematic flowchart of S105 in the light interaction method based on audio data recognition provided in the first embodiment of this application; Figure 6 This is a schematic diagram of a light interaction device based on audio data recognition provided in the second embodiment of this application; Figure 7 This is a schematic diagram of an interactive bulb light device based on audio recognition provided in the third embodiment of the present invention. Detailed Implementation

[0016] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0017] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0018] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0019] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0020] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0021] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0022] Please see Figure 1 , Figure 1This is a schematic flowchart of the lighting interaction method based on audio data recognition provided in the first embodiment of this application. In this embodiment, the executing entity of the lighting interaction method based on audio data recognition is a device with lighting interaction functionality based on audio data recognition, such as a server, computer, etc. Figure 1 The light interaction method based on audio data recognition shown includes: S101: Obtain first audio data output by the first device and second audio data output by the target user; wherein, the first audio data includes first acoustic data and first text data, and the second audio data includes second acoustic data and second text data.

[0023] The device acquires first audio data output by the first device and second audio data output by the target user.

[0024] In this embodiment, the first device is a smart device capable of outputting audio data, such as a smart speaker, a smart assistant, or any electronic device capable of emitting sound. It typically provides a fixed audio template to provide background information or create a specific scene atmosphere.

[0025] The first audio data is the audio data output by the first device, and it consists of two parts: The first acoustic data, which is the raw audio signal, such as a waveform signal, can be further processed through feature extraction, such as Mel-frequency cepstral coefficients (MFCCs) or time-frequency domain features.

[0026] The first text data is the text form converted from the first acoustic data through a speech recognition algorithm. It is often used to describe the environment or indicate the status, such as "Welcome to the Home Karaoke Platform" or a fixed playback track.

[0027] The second device is a device that is linked to the first device, such as a smart lighting controller or a device equipped with bulb lamps.

[0028] The second audio data is the audio input from the target user, including: The second acoustic data is the user's voice signal, which may contain information such as spoken expression and emotional fluctuations.

[0029] The second text data is user input text obtained through speech recognition, which may involve instructions to the first device, expressions of user feelings, or lyrics sung by the user.

[0030] For example, in entertainment settings such as KTV, parties, or bars, the first audio data output by the first device is a fixed playback track, and the second audio data output by the target user is the user's singing.

[0031] S102: Perform attention encoding on the first text data and the second text data to obtain text encoded data.

[0032] The device performs attention encoding on the first and second text data to obtain text-encoded data.

[0033] Specifically, natural language processing (NLP) techniques, such as BERT or the Transformer architecture, can be used to perform attention encoding on the first text data (e.g., "Welcome here") and the second text data (e.g., what the user says, such as "I like blue").

[0034] During processing, the system will segment the text and filter stop words, finally generating more refined text encoding data to help identify user emotions in subsequent steps.

[0035] In one implementation, such as Figure 2 As shown, S102 may include S1021 to S1024, and the specific details of S1021 to S1024 are as follows: S1021: Obtain the first text representation corresponding to the first text data and the second text representation corresponding to the second text data, and obtain the syntactic representation corresponding to the first text data.

[0036] The device acquires a first text representation corresponding to the first text data and a second text representation corresponding to the second text data. The first text representation converts the first text data into a vector form for subsequent calculations and processing. Common representation methods include word embeddings, such as Word2Vec and GloVe. The second text representation converts the second text data into a corresponding vector form to represent the user input.

[0037] The device can calculate a first text representation by inputting the first text data from the first device into a text embedding model. The same method is applied to the user's second text data, and the second text representation is obtained by passing it through the same text embedding model.

[0038] The device uses syntactic analysis tools in Natural Language Processing (NLP) (such as Stanford Parser or SpaCy) to perform syntactic analysis on the first text data, obtain syntactic representations, and display the words in the text and their relationships.

[0039] In one implementation, such as Figure 3 As shown, the method for obtaining the syntactic representation corresponding to the first text data can be S10211~S10213, and S10211~S10213 are as follows: S10211: Extract the text feature representation corresponding to the first text representation.

[0040] Text feature representation is a further processing result that transforms the initial text data into vector form. It is typically used to extract key information and features from the text to facilitate subsequent model processing and analysis.

[0041] Text embedding techniques (such as Word2Vec, GloVe, BERT, etc.) are used to transform the first text data into its feature representation. This representation captures the semantic information and contextual relationships of the text content.

[0042] For example, processing the first text data "Welcome" might result in an output vector containing feature representations that reflect the correlation between "welcome" and "visit".

[0043] S10212: Obtain the temporal adjacency matrix corresponding to the first text data, and obtain the temporal weighted graph corresponding to the first text data; wherein, the temporal adjacency matrix is ​​the adjacency relationship between text nodes in the temporal dimension of the first text data, and the temporal weighted graph includes the probability of the text node being passed to the adjacent text node in the temporal dimension of the first text data.

[0044] The device acquires the temporal adjacency matrix corresponding to the first text data. The temporal adjacency matrix is ​​a syntactic adjacency matrix, which defines the connection relationship of text nodes based on the text order. Assuming several words in the text are nodes, forming an adjacency matrix, if a word is connected to its subsequent words (for example, "welcome" and "visit" are directly connected), then the corresponding value is set to 1, and the rest are 0.

[0045] The device acquires the time-weighted graph corresponding to the first text data. The time-weighted graph is based on the adjacency matrix, assigning weights to the connections between text nodes to represent the probability of transmission. For example, the connection weight between "welcome" and "visit" can be calculated based on their common frequency of occurrence, forming a weighted graph that represents the possibility of information transmission between nodes.

[0046] Specifically, when obtaining the time-series weighted graph corresponding to the first text data, the device obtains the adjacency type between text nodes in the time-series adjacency matrix; extracts the adjacency type representation corresponding to the adjacency type between text nodes in the time-series adjacency matrix; and obtains the time-series weighted graph corresponding to the first text data based on the text feature representation corresponding to the first text representation, the adjacency type representation corresponding to the adjacency type between text nodes in the time-series adjacency matrix, and the preset weighting calculation method.

[0047] The device acquires the adjacency types between text nodes in the temporal adjacency matrix. By analyzing the temporal adjacency matrix, the device determines the connection relationships between text nodes. For example, if node A and node B are directly connected, their adjacency type can be labeled as "direct relationship." These adjacency types can be identified and labeled using a rule engine or machine learning methods.

[0048] The temporal adjacency matrix is ​​a matrix that represents the connection relationship between text nodes in the temporal dimension of the text. It can be represented by 1 and 0 to indicate whether the nodes are connected or not.

[0049] A text node refers to the basic unit of analysis in a text, usually a word, phrase, or sentence, and can have different definitions depending on the system design.

[0050] Adjacency type refers to the nature or type of connection between text nodes, such as direct connection, synonym relationship, hierarchical relationship, etc. Different connection types reflect different ways of transmitting text information.

[0051] The device extracts the adjacency type representations corresponding to the adjacency types between text nodes in the temporal adjacency matrix. The identified adjacency types are then converted into vector representations. For example, if there are three adjacency types—"direct relation," "synonymous relation," and "hyper- ...

[0052] Based on the text feature representation corresponding to the first text representation, the adjacency type representation corresponding to the adjacency type between text nodes in the temporal adjacency matrix, and a preset weighted calculation method, a temporal weighted graph corresponding to the first text data is obtained. Using the text feature representation and adjacency type representation, and through the preset weighted calculation method, the relationships between nodes are weighted and transformed into specific association values. For example, each corresponding connection can be weighted according to the similarity of text features or the importance of adjacency type.

[0053] For example, if the adjacency between node A and node B is a "direct relationship," then the weighting value can be set to 0.8; if it is a synonym relationship, then it is set to 0.6. This forms the final time-series weighted graph, which can dynamically reflect the transmission and changes of text features.

[0054] This embodiment enhances the ability to understand text content and model relationships, and also provides the entire system with higher intelligence in information processing, thereby accurately obtaining audio status recognition information, controlling the lighting parameters of the second device, and improving the interactive effect.

[0055] S10213: Based on the text feature representation corresponding to the first text representation, the temporal adjacency matrix corresponding to the first text data, the temporal weighted graph corresponding to the first text data, and the pre-trained syntactic extraction model, obtain the syntactic representation corresponding to the first text data.

[0056] The text feature representation extracted in step S10211, the temporal adjacency matrix and temporal weighted graph in step S10212, and the pre-trained syntactic extraction model are input for comprehensive analysis.

[0057] The model identifies the core sentence structure and lexical relationships in the text, and outputs the final syntactic representation. For example, for the sentence "Welcome," the model might output a syntactic relationship graph containing the subject, verb, and object, enabling the system to better understand the sentence structure.

[0058] In this embodiment, by combining text feature representation, temporal adjacency matrix, temporal weighted graph and syntactic extraction model, the system's ability to process text data can be significantly improved, and the syntactic representation corresponding to the first text data can be obtained more accurately. This allows for precise acquisition of audio state recognition information, thereby controlling the lighting parameters of the second device and improving the interactive effect.

[0059] S1022: Perform multi-head interactive attention encoding on the syntactic representation corresponding to the first text data and the first text representation to obtain the first text encoded representation.

[0060] The device inputs the first text representation and the syntactic representation of the first text data into the multi-head interactive attention encoding module. This module displays the input vector and syntactic structure, and uses a multi-head attention mechanism for parallel processing to extract important features and information from the text. Multi-head interactive attention encoding is a deep learning technique that utilizes a multi-head attention mechanism to capture different features and relationships in the input data. By processing multiple "attention heads" in parallel, it can simultaneously focus on multiple aspects of the text input.

[0061] After encoding, a first text encoding representation is generated, which records the overall meaning and grammatical structure of the first text.

[0062] S1023: Perform multi-head interactive attention encoding on the syntactic representation corresponding to the first text data and the second text representation to obtain the second text encoded representation.

[0063] The second text representation is combined with the syntactic representation and input into the multi-head interactive attention encoding module. This process is similar to step S1022, where the system extracts valuable information from the user's expression.

[0064] After encoding, a second text encoding representation is generated, which integrates information such as user emotions and intentions.

[0065] S1024: Obtain the text encoding data based on the first text encoding representation and the second text encoding representation.

[0066] Finally, the first and second text-encoded representations obtained in steps S1022 and S1023 are integrated to form the final text-encoded data. This data can capture basic information of the first device and real-time user input, which can then be used for subsequent analysis, decision-making, or control.

[0067] The above implementation method, utilizing a multi-head interactive attention mechanism and syntactic representation, can significantly improve the processing capability of text data. It not only achieves an effective combination of the first device and user input, but also enhances the system's understanding of user emotions and intentions, ultimately resulting in a more intelligent and seamless interactive experience.

[0068] S103: Perform similarity-weighted encoding on the first acoustic data and the second acoustic data to obtain acoustic encoded data.

[0069] The device performs similarity-weighted encoding on the first and second acoustic data to obtain acoustically encoded data. Specifically, the device extracts features (e.g., MFCC or pitch analysis) from the first and second acoustic data, and then calculates their similarity. Based on the differences in similarity, the acoustic features are weighted and fused to generate acoustically encoded data.

[0070] For example, when a user says "The atmosphere here is great," the system will identify the user's emotional changes and increase the weight of the audio information expressed by the user by comparing the corresponding frequency characteristics of the background music played by the smart speaker.

[0071] In one implementation, such as Figure 4 As shown, S103 may include S1031~S1034, and S1031~S1034 may include: S1031: Obtain the first acoustic representation corresponding to the first acoustic data and the second acoustic representation corresponding to the second acoustic data.

[0072] In this embodiment, the first acoustic data is an audio output signal from the first device, such as speech, music, or ambient sound. In implementation, this is typically background audio provided by the device, which may be some preset voice prompts or background music.

[0073] The second acoustic data is the audio input signal from the target user. This could be the user's voice commands, dialogue, or feedback, and typically contains some uncertainty and noise.

[0074] The first acoustic representation is obtained by converting the first acoustic data into vector form and using acoustic feature extraction methods (such as MFCC or spectral analysis), which reflects its acoustic feature information.

[0075] The second acoustic representation involves transforming the second acoustic data into the same vector form as the first acoustic representation. The same acoustic feature extraction method is used to ensure that the acoustic features can be compared in the same dimension.

[0076] For the first acoustic data, acoustic features are extracted using acoustic feature extraction algorithms (such as MFCC, Chroma, Spectrogram, etc.) to generate a first acoustic representation. For example, the frequency domain information of the signal is transformed into a computable feature vector, representing the characteristics of this acoustic data.

[0077] For the second acoustic data, the same method is used to extract the acoustic representation, ensuring that the two representations are of the same form for subsequent comparison.

[0078] For example, if the first acoustic data is background music, its acoustic representation is obtained by extracting the spectral features of the content, while the input user speech is parsed to extract the features of the user's voice using the same algorithm.

[0079] S1032: Extract the first state information corresponding to the first acoustic representation and the second state information corresponding to the second acoustic representation.

[0080] Using an emotion classification algorithm, first state information is extracted from the first acoustic representation. This information can describe the emotion conveyed by the acoustic data (such as happiness, sadness, etc.) or specific operation instructions (such as "turn up the volume").

[0081] For the second acoustic representation, the same state information extraction is performed to capture the user's emotions and intentions.

[0082] For example, a user's voice input might convey a happy emotion, while the background music suggests a relaxed and confident state.

[0083] In one embodiment, the first acoustic representation and the second acoustic representation are input into a pre-trained acoustic state recognition model to obtain first emotional state information corresponding to the first acoustic representation and second emotional state information corresponding to the second acoustic representation; wherein, the pre-trained acoustic state recognition model includes several acoustic state recognition modules, each of which includes a sub-module for processing the temporal features of the acoustic representation, a sub-module for processing the frequency features of the acoustic representation, and a self-attention encoding module.

[0084] Specifically, this involves collecting acoustic data containing multiple emotional states (e.g., emotionally rich dialogue audio). Preprocessing the audio signals, including denoising and normalization, improves the effectiveness of subsequent feature extraction. The collected audio data is then analyzed to extract first and second acoustic representations. Features can be obtained using methods such as Mel-frequency cepstral coefficients (MFCC) and spectral features. When extracting temporal features, temporal analysis techniques (such as sliding windows) are used to capture the dynamic changes in the audio signal. When extracting frequency features, spectral analysis is used to obtain the energy distribution of different frequency components.

[0085] The extracted first and second acoustic representations are input into the pre-trained acoustic state recognition model, respectively. The model integrates different acoustic state recognition modules, including a sub-module for processing temporal features, a sub-module for processing frequency features, and a self-attention encoding module, which can effectively capture complex emotional features.

[0086] After processing by the pre-trained model, the output is the first emotional state information corresponding to the first acoustic representation and the second emotional state information corresponding to the second acoustic representation.

[0087] This emotional state information can be emotional categories (such as happiness, sadness, etc.) and probability distributions (confidence level of each emotion).

[0088] S1033: Analyze the acoustic state similarity between the first state information and the second state information.

[0089] By using a predefined similarity calculation method, such as cosine similarity, the similarity between the first-state information and the second-state information is analyzed and compared. For example: [ \text{Similarity} = \frac{A \cdot B}{||A|| ||B||} ] Here, (A) and (B) represent the state information vectors. If the two are similar in emotion (e.g., both represent "happiness"), then the similarity will be high.

[0090] S1034: Based on the acoustic state similarity, perform similarity-weighted encoding on the first acoustic representation and the second acoustic representation to obtain the acoustic encoded data.

[0091] The obtained acoustic state similarity is used to weight the first and second acoustic representations. For example, by setting weighting criteria: If the similarity is high, give it a higher weight; If the similarity is low, a lower weight is given.

[0092] Finally, the weighted feature data is combined to generate acoustically coded data, making it more targeted and able to better reflect the user's state and intent.

[0093] In this embodiment, through real-time analysis, similarity analysis, and a weighting mechanism of user-input acoustic data, the system can make more human-like responses based on user emotions and background information output by the first device, thereby improving the interactive experience. It can accurately understand user emotions and intentions, and thus more precisely adjust response behavior. This optimizes the accuracy of the system's response behavior, driving a more interactive and human-like user experience.

[0094] S104: Obtain audio state recognition information based on the text encoding data, acoustic encoding data, and pre-trained audio state recognition model.

[0095] The text-encoded data and acoustic-encoded data generated in steps S102 and S103 are input into a pre-trained audio state recognition model. This model can infer the user's emotional state through deep learning analysis.

[0096] For example, when a user exhibits excitement, the audio state recognition information output by the state recognition model might be "user is emotionally excited".

[0097] S105: Control the lighting parameters of the second device according to the audio status recognition information; wherein the second device is linked with the first device.

[0098] By interpreting the obtained audio status recognition information, the system controls the lighting parameters of the second device (bulb lamp).

[0099] If the user appears relaxed and happy, the lights may switch to a warm yellow mode and increase their brightness; while when the user is quiet, the lights will dim and switch to a soft blue.

[0100] This kind of lighting adjustment based on user emotions not only enhances the atmosphere of the scene, but also improves the user experience.

[0101] In one implementation, such as Figure 5As shown, S105 may include S1051~S1052, the second device is a plurality of bulb lights linked with the first device, and the audio state recognition information includes intensity state information and emotion state information. S1051: Based on the intensity status information, control the brightness parameters and flicker parameters of several of the bulb lamps.

[0102] Based on the extracted intensity status information, a control signal is sent to the second device (bulb). For example, if the intensity status information indicates a high audio intensity, the bulb's brightness parameter can be set to high brightness mode, and the flicker parameter may be set to a stable state; conversely, if the intensity is low, the brightness is reduced and the flicker frequency is increased.

[0103] S1052: Based on the emotional state information, control the color temperature parameters of several of the bulb lamps.

[0104] Based on the extracted emotional state information, adjust the color temperature parameters of the bulb. For example, when the emotional state information is "happy", the light can be adjusted to a warm tone (2700K); if the emotional state information is "sad", it can be adjusted to a cool tone (6500K).

[0105] The following are some specific application scenarios of this application: 1. Meeting room scene In a meeting room setting, the primary device might output "Meeting begins, please pay attention," while the user's real-time input could be "I think this solution needs improvement." The system can recognize the user's positive emotions and appropriately adjust the lighting to a bright, cool tone to enhance engagement; conversely, as the discussion deepens, the lighting dims to reinforce the atmosphere of reflection.

[0106] 2. Entertainment Scenes In entertainment settings such as karaoke bars, parties, or bars, fixed audio tracks and real-time user input, such as feedback from singing or emotional expressions, can be used to adjust lighting in real time. When the background music reaches a climax, the lights can flash or even change color to attract users' attention and enhance the atmosphere.

[0107] 3. Smart healthcare scenarios In a medical setting, such as a smart interactive device for rehabilitation, the first device's sound can be gentle background music, while the second audio data comes from the patient's emotional expressions. Through these methods, the system can adjust the lighting in real time to alleviate the patient's anxiety.

[0108] In this embodiment, first audio data output by a first device and second audio data output by a target user are acquired; attention encoding is performed on the first and second text data to obtain text-encoded data; similarity-weighted encoding is performed on the first and second acoustic data to obtain acoustic-encoded data; audio state recognition information is obtained based on the text-encoded data, acoustic-encoded data, and a pre-trained audio state recognition model; and the lighting parameters of the second device are controlled based on the audio state recognition information. This method effectively enhances the interactive capabilities of smart devices, ensuring stability while flexibly responding to user emotions and environmental changes. It enables lighting to provide more accurate and personalized responses in different environments and situations, improving the interactive effect of lighting and achieving a more intelligent and humanized interactive experience.

[0109] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0110] Please see Figure 6 , Figure 6 This is a schematic diagram of a light interaction device based on audio data recognition provided in the second embodiment of this application. The included units are used to perform... Figures 1-5 The steps in the corresponding embodiments. Please refer to the details. Figures 1-5 The relevant descriptions in the corresponding embodiments are shown below. For ease of explanation, only the parts relevant to this embodiment are shown. See also... Figure 6 The light interaction device 6 based on audio data recognition includes: The acquisition unit 61 is used to acquire first audio data output by the first device and second audio data output by the target user; wherein, the first audio data includes first acoustic data and first text data, and the second audio data includes second acoustic data and second text data; The first processing unit 62 is used to perform attention encoding on the first text data and the second text data to obtain text encoded data; The second processing unit 63 is used to perform similarity-weighted encoding on the first acoustic data and the second acoustic data to obtain acoustic encoded data; The third processing unit 64 is used to obtain audio state recognition information based on the text encoded data, acoustic encoded data and the pre-trained audio state recognition model; The fourth processing unit 65 is used to control the lighting parameters of the second device based on the audio status recognition information; wherein the second device is linked with the first device.

[0111] Furthermore, the first processing unit is specifically used for: Obtain the first text representation corresponding to the first text data and the second text representation corresponding to the second text data, and obtain the syntactic representation corresponding to the first text data; Multi-head interactive attention encoding is performed on the syntactic representation corresponding to the first text data and the first text representation to obtain the first text encoded representation; Multi-head interactive attention encoding is performed on the syntactic representation corresponding to the first text data and the second text representation to obtain the second text encoded representation; The text encoded data is obtained based on the first text encoding representation and the second text encoding representation.

[0112] Furthermore, the first processing unit is specifically used for: Extract the text feature representation corresponding to the first text representation; Obtain the temporal adjacency matrix corresponding to the first text data, and obtain the temporal weighted graph corresponding to the first text data; wherein, the temporal adjacency matrix is ​​the adjacency relationship between text nodes in the temporal dimension of the first text data, and the temporal weighted graph includes the probability of the text node being passed to the adjacent text node in the temporal dimension of the first text data; Based on the text feature representation corresponding to the first text representation, the temporal adjacency matrix corresponding to the first text data, the temporal weighted graph corresponding to the first text data, and the pre-trained syntactic extraction model, the syntactic representation corresponding to the first text data is obtained.

[0113] Furthermore, the first processing unit is specifically used for: Obtain the adjacency type between the text nodes in the temporal adjacency matrix; Extract the adjacency type representation corresponding to the adjacency type between the text nodes in the temporal adjacency matrix; Based on the text feature representation corresponding to the first text representation, the adjacency type representation corresponding to the adjacency type between the text nodes in the temporal adjacency matrix, and the preset weighted calculation method, a temporal weighted graph corresponding to the first text data is obtained.

[0114] Furthermore, the second processing unit is specifically used for: Obtain the first acoustic representation corresponding to the first acoustic data and the second acoustic representation corresponding to the second acoustic data; Extract the first state information corresponding to the first acoustic representation and the second state information corresponding to the second acoustic representation; Analyze the acoustic state similarity between the first state information and the second state information; Based on the acoustic state similarity, the first acoustic representation and the second acoustic representation are weighted by similarity to obtain the acoustic coded data.

[0115] Furthermore, the second processing unit is specifically used for: The first acoustic representation and the second acoustic representation are input into a pre-trained acoustic state recognition model to obtain first emotional state information corresponding to the first acoustic representation and second emotional state information corresponding to the second acoustic representation; wherein, the pre-trained acoustic state recognition model includes several acoustic state recognition modules, each of which includes a sub-module for processing the temporal features of the acoustic representation, a sub-module for processing the frequency features of the acoustic representation, and a self-attention encoding module.

[0116] Furthermore, the second device comprises several bulbs linked to the first device, and the audio state recognition information includes intensity state information and emotional state information. The fourth processing unit is specifically used for: Based on the intensity status information, control the brightness and flicker parameters of several of the bulb lamps; Based on the emotional state information, the color temperature parameters of several of the bulb lamps are controlled.

[0117] Figure 7 This is a schematic diagram of an interactive bulb light device based on audio recognition provided in the third embodiment of the present invention. Figure 7 As shown, a terminal device 7 in this embodiment includes: a processor 70, a memory 71, and a computer program 72 stored in the memory 71 and executable on the processor 70, such as a program for an interactive method for a bulb lamp based on audio recognition.

[0118] When processor 70 executes computer program 72, it implements the steps in the above-described embodiments of the audio recognition-based bulb light interaction method, for example... Figure 1 The steps S101 to S105 are shown. Alternatively, when the processor 70 executes the computer program 72, it implements the functions of each module / unit in the above-described device embodiments, for example... Figure 6 The functions of modules 61 to 65 are shown.

[0119] For example, the computer program 72 can be divided into one or more modules / units, which are stored in the memory 71 and executed by the processor 70 to complete this application. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program 72 in the audio recognition-based bulb interactive device. For example, the computer program 72 can be divided into an acquisition unit, a first processing unit, a second processing unit, a third processing unit, and a fourth processing unit, with the specific functions of each unit as follows: The first processing unit is used to perform attention encoding on the first text data and the second text data to obtain text-encoded data. The second processing unit is used to perform similarity-weighted encoding on the first acoustic data and the second acoustic data to obtain acoustic encoded data; The third processing unit is used to obtain audio state recognition information based on the text encoded data, acoustic encoded data and the pre-trained audio state recognition model; The fourth processing unit is used to control the lighting parameters of the second device based on the audio status recognition information; wherein the second device is linked with the first device.

[0120] The audio recognition-based interactive bulb device 7 provided in this application may include, but is not limited to, a processor and a memory. Those skilled in the art will understand that... Figure 7 This is merely an example of an interactive bulb light device based on audio recognition and does not constitute a limitation on interactive bulb light devices based on audio recognition. It may include more or fewer components than shown in the figure, or combine certain components, or different components. For example, the interactive bulb light device based on audio recognition may also include input / output devices, network access devices, buses, etc.

[0121] The processor 70 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0122] The memory 71 can be an internal storage unit of the audio recognition-based interactive bulb device, such as a hard drive or memory. The memory 71 can also be an external storage device, such as a plug-in hard drive, smart media card (SMC), secure digital (SD) card, or flash card. Furthermore, the audio recognition-based interactive bulb device may include both internal and external storage units. The memory 71 is used to store the computer program and other programs and data required by the audio recognition-based interactive bulb device. The memory 71 can also be used to temporarily store data that has been output or will be output.

[0123] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0124] This application also provides a network device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor executes the computer program to implement the steps in any of the above method embodiments.

[0125] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0126] This application provides a computer program product that, when run on a mobile terminal, enables the mobile terminal to implement the steps described in the above-described method embodiments.

[0127] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographing device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0128] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0129] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0130] In the embodiments provided in this application, it should be understood that the disclosed apparatus / network devices and methods can be implemented in other ways. For example, the apparatus / network device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0131] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0132] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A light interaction method based on audio data recognition, characterized in that, include: Acquire first audio data output by a first device and second audio data output by a target user; wherein, the first audio data includes first acoustic data and first text data, and the second audio data includes second acoustic data and second text data; Attention encoding is performed on the first text data and the second text data to obtain text-encoded data; The first acoustic data and the second acoustic data are weighted by similarity to obtain acoustic coded data; Based on the text-encoded data, acoustic-encoded data, and the pre-trained audio state recognition model, audio state recognition information is obtained; Based on the audio status recognition information, the lighting parameters of the second device are controlled; wherein the second device is linked with the first device.

2. The light interaction method based on audio data recognition as described in claim 1, characterized in that, The step of performing attention encoding on the first text data and the second text data to obtain text-encoded data includes: Obtain the first text representation corresponding to the first text data and the second text representation corresponding to the second text data, and obtain the syntactic representation corresponding to the first text data; Multi-head interactive attention encoding is performed on the syntactic representation corresponding to the first text data and the first text representation to obtain the first text encoded representation; Multi-head interactive attention encoding is performed on the syntactic representation corresponding to the first text data and the second text representation to obtain the second text encoded representation; The text encoded data is obtained based on the first text encoding representation and the second text encoding representation.

3. The light interaction method based on audio data recognition as described in claim 2, characterized in that, The step of obtaining the syntactic representation corresponding to the first text data includes: Extract the text feature representation corresponding to the first text representation; Obtain the temporal adjacency matrix corresponding to the first text data, and obtain the temporal weighted graph corresponding to the first text data; wherein, the temporal adjacency matrix is ​​the adjacency relationship between text nodes in the temporal dimension of the first text data, and the temporal weighted graph includes the probability of the text node being passed to the adjacent text node in the temporal dimension of the first text data; Based on the text feature representation corresponding to the first text representation, the temporal adjacency matrix corresponding to the first text data, the temporal weighted graph corresponding to the first text data, and the pre-trained syntactic extraction model, the syntactic representation corresponding to the first text data is obtained.

4. The light interaction method based on audio data recognition as described in claim 3, characterized in that, The step of obtaining the time-series weighted graph corresponding to the first text data includes: Obtain the adjacency type between the text nodes in the temporal adjacency matrix; Extract the adjacency type representation corresponding to the adjacency type between the text nodes in the temporal adjacency matrix; Based on the text feature representation corresponding to the first text representation, the adjacency type representation corresponding to the adjacency type between the text nodes in the temporal adjacency matrix, and the preset weighted calculation method, a temporal weighted graph corresponding to the first text data is obtained.

5. The light interaction method based on audio data recognition as described in any one of claims 1 to 4, characterized in that, The step of performing similarity-weighted encoding on the first acoustic data and the second acoustic data to obtain acoustic encoded data includes: Obtain the first acoustic representation corresponding to the first acoustic data and the second acoustic representation corresponding to the second acoustic data; Extract the first state information corresponding to the first acoustic representation and the second state information corresponding to the second acoustic representation; Analyze the acoustic state similarity between the first state information and the second state information; Based on the acoustic state similarity, the first acoustic representation and the second acoustic representation are weighted by similarity to obtain the acoustic coded data.

6. The light interaction method based on audio data recognition as described in claim 5, characterized in that, The extraction of the first state information corresponding to the first acoustic representation and the second state information corresponding to the second acoustic representation includes: The first acoustic representation and the second acoustic representation are input into a pre-trained acoustic state recognition model to obtain first emotional state information corresponding to the first acoustic representation and second emotional state information corresponding to the second acoustic representation; wherein, the pre-trained acoustic state recognition model includes several acoustic state recognition modules, each of which includes a sub-module for processing the temporal features of the acoustic representation, a sub-module for processing the frequency features of the acoustic representation, and a self-attention encoding module.

7. The light interaction method based on audio data recognition as described in any one of claims 1 to 4, characterized in that, The second device consists of several bulb lights linked to the first device. The audio state recognition information includes intensity state information and emotional state information. Controlling the lighting parameters of the second device based on the audio state recognition information includes: Based on the intensity status information, control the brightness and flicker parameters of several of the bulb lamps; Based on the emotional state information, the color temperature parameters of several of the bulb lamps are controlled.

8. A light interaction device based on audio data recognition, characterized in that, include: The acquisition unit is used to acquire first audio data output by a first device and second audio data output by a target user; wherein the first audio data includes first acoustic data and first text data, and the second audio data includes second acoustic data and second text data; The first processing unit is used to perform attention encoding on the first text data and the second text data to obtain text-encoded data. The second processing unit is used to perform similarity-weighted encoding on the first acoustic data and the second acoustic data to obtain acoustic encoded data; The third processing unit is used to obtain audio state recognition information based on the text encoded data, acoustic encoded data and the pre-trained audio state recognition model; The fourth processing unit is used to control the lighting parameters of the second device based on the audio status recognition information; wherein the second device is linked with the first device.

9. A light interaction device based on audio data recognition, characterized in that, include: Processor, memory, and computer programs stored in said memory and executable on said processor; When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

Images