TinyML-based music interaction response method and system, application and storage medium

Abstract

The invention relates to the technical field of artificial intelligence, and discloses a TinyML-based music interaction response method and system, an application and a storage medium, the method is based on an end-side TinyML technology, music content can be analyzed in real time, actions and results required to be responded by a device can be given, the constraint of manual or semi-manual making actions can be eliminated, waiting for too much processing time is not needed, and the efficiency is improved. According to the method, input and analysis are synchronously carried out, the whole system can greatly reduce the computing power requirement, the reliability and universality of interactive response are greatly improved, and the practical value in practical significance is achieved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to a music interaction response method, system, application, and storage medium based on TinyML. Background Technology

[0002] Currently, most music-based VAD (Voice Activity Detection) applications on the market are analysis software aimed at professionals, with a high barrier to entry. They often require algorithm models with a large number of parameters and high computing power to ensure the accuracy of the algorithm. In embedded platforms with limited hardware resources, such as in the field of smart toys for children and parents, it is difficult to meet the high resource requirements for such solutions to operate.

[0003] The two most common solutions to this problem are the MCU-based pre-recording solution and the PC+Arduino-based solution. The MCU-based pre-recording solution pre-creates the songs and toy device actions and stores them in the MCU chip. During actual operation, it plays songs and responds to actions based on the pre-recorded data. However, due to the limited storage space of the MCU and the need for manual creation of the pre-recorded data, the number of songs that can be displayed is extremely limited. Manual creation is also time-consuming and labor-intensive, and it is helpless when dealing with songs not pre-recorded, making large-scale application difficult. The PC+Arduino-based solution modifies the original device, connecting the control wires to an external system. The PC controls and plays the music, and the original device only responds to the PC's control signals. Compared to the first solution, it can play a wider variety of music, increasing its versatility. However, the action response still requires PC control, limiting it to semi-manual action creation. This modification method still has a high technical threshold and requires a PC, making widespread application difficult. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a music interactive response method, system, application, and storage medium based on TinyML, which can analyze music content in real time, realize synchronous input parsing, improve parsing efficiency, and greatly reduce computing power requirements.

[0005] The present invention solves the technical problem by adopting the following technical solution: This invention provides a music interaction response method based on TinyML, the method comprising the following steps: Acquire and configure TinyML models for audio data analysis and output results for various tasks, including creating datasets and labels, model training, model quantization, and deployment. Acquire audio data, cache, integrate, and transform the audio data before outputting it in blocks; The configured TinyML model is used to analyze the segmented audio data and output recognition results, specifically including: Data preprocessing involves performing feature transformation on the data to converge the audio time-domain signal to a feature signal; Extract common features from feature signals, and use one or more identical or different head structures to complete and output result data for different tasks; The results data from different tasks are processed according to single-component classification tasks, multiple-choice classification tasks, or regression tasks, and the data to be arbitrated is output, specifically including: The results of the single-component classification task are calculated using the sigmoid type, converted into corresponding probability values, and then compared with a preset threshold before being output. After performing softmax calculations on the results data of the multiple-choice classification task and converting them into corresponding probability values, the data with the largest value is output. The results of the regression task are output directly. Arbitration is conducted based on the data to be arbitrated to control music playback and action response, specifically including: Acquire the configuration and status of the action device, confirm the arbitration data that needs to be responded to, and filter out useless arbitration data; Based on the pre-defined logic of the action device, the response priority of the data to be arbitrated is determined and the action sequence is generated and output sequentially to complete the control or response.

[0006] Preferably, the creation of the dataset and labels includes: Obtain the raw audio data and its lyrics timestamp information, and create a tag indicating whether there is human voice. After the sound source is separated, the data of each audio track is obtained, and the target label is created by collecting frame-level information based on the energy of each audio track. The target labels are subjected to de-extreme operations including smoothing, filtering, and merging. The model training includes: Frame-level supervised learning is performed on the acquired raw audio data and target labels; The loss function is selected based on the task type, and supervised learning is performed based on the total loss value after weighted summation. The model quantization and deployment include: Quantize the floating-point model into a fixed-point model; Matching determines whether dynamic quantization or static quantization is used.

[0007] Preferably, the step of caching, integrating, and converting audio data before outputting it in blocks specifically includes: The audio data stream is buffered into the target data block and then output for post-processing. The acquired audio data is converted into the target format data using encoding and decoding methods; Convert non-target sampling rate or precision data to target sampling rate or precision data.

[0008] Preferably, the recognition results include human voice recognition results, accompaniment recognition results, target instrument sound recognition results, drum beat recognition results, rhythm recognition results, chord type recognition results, sound energy recognition results, and lip shape or amplitude recognition results. The single-component classification tasks are independent but not mutually exclusive tasks, including human voices, accompaniment, instrument sounds, and drum beats. The multiple-choice classification task is a mutually influential and mutually exclusive task, including chord type and rhythm; The regression task is a non-binary or discrete task, including sound energy, lip shape, or amplitude.

[0009] Preferably, the step of processing the result data of different tasks according to single-component classification tasks, multiple-choice classification tasks, or regression tasks and outputting the data to be arbitrated further includes: The result data of the classification task is filtered or smoothed in the time dimension to make a comprehensive judgment based on the information data of multiple frames before and after, or to output the average or median of the results of multiple frames. High and low thresholds are used simultaneously to trigger or convert the result data into invalid data.

[0010] Preferably, the matching task type selection loss function specifically includes: For single-component classification tasks, choose a binary classification loss function related to sigmoid; For multiple-choice classification tasks, choose a softmax-related loss function; For regression tasks, the MSE loss function is selected.

[0011] Preferably, the model training further includes: Advanced optimizations are introduced, including FocalLoss, pseudo-labels, masking techniques, self-distillation mechanisms, or external distillation mechanisms. Supervised learning performed by the matching optimizer.

[0012] This invention also provides a music interaction response system based on TinyML, the system comprising a mechanical motion module, a controlled drive module, a music flow control module, a data analysis module, and an arbitration control module, wherein... The mechanical motion module is configured to complete a specified preset action under the drive of the controlled drive module; The controlled drive module is built into the mechanical motion module and is used to receive instructions from the arbitration control module and drive the mechanical motion module. The music stream control module is configured to acquire audio data, complete audio conversion, and ultimately perform music playback interaction. The data analysis module is configured to analyze audio data and output analysis results, specifically including: Acquire and configure TinyML models for audio data analysis and output results for various tasks, including creating datasets and labels, model training, model quantization, and deployment. Acquire audio data, cache, integrate, and transform the audio data before outputting it in blocks; The configured TinyML model is used to analyze the segmented audio data and output recognition results, specifically including: Data preprocessing involves performing feature transformation on the data to converge the audio time-domain signal to a feature signal; Extract common features from feature signals, and use one or more identical or different head structures to complete and output result data for different tasks; The results data from different tasks are processed according to single-component classification tasks, multiple-choice classification tasks, or regression tasks, and the data to be arbitrated is output, specifically including: The results of the single-component classification task are calculated using the sigmoid type, converted into corresponding probability values, and then compared with a preset threshold before being output. After performing softmax calculations on the results data of the multiple-choice classification task and converting them into corresponding probability values, the data with the largest value is output. The results of the regression task are output directly. The arbitration control module is configured to output a sequence of response actions based on the analysis results, and to complete the final action control and interactive behavior, specifically including: Acquire the configuration and status of the action device, confirm the arbitration data that needs to be responded to, and filter out useless arbitration data; Based on the pre-defined logic of the motion device, the response priority of the data to be arbitrated is determined and the action sequence is generated for sequential output. The controlled drive module completes the control or response to the mechanical motion module or music flow control module.

[0013] The present invention also provides an electronic device, comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which enables the at least one processor to execute the aforementioned TinyML-based music interaction response method.

[0014] The present invention also provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the aforementioned TinyML-based music interactive response method.

[0015] Compared with the prior art, the present invention has the following beneficial effects: The TinyML-based music interaction response method of this invention immediately initiates a low-computational-consumption streaming processing mechanism upon receiving audio data. It analyzes the continuous audio data stream segment by segment in real time, without waiting for all audio data to load. This simultaneous input and parsing not only improves parsing efficiency but also reduces computational requirements and pressure. For the extracted key features such as audio rhythm, melody, and chords, it provides corresponding analysis results, which are sent to the control module to perform corresponding actions or responses. This eliminates the constraints of manual or semi-manual action creation and eliminates excessive processing time. The system lowers the user threshold, greatly increases the reliability and versatility of the interaction response, and enables offline multi-device collaboration. Furthermore, it is easier to port to embedded and other mobile devices, effectively expanding the application scenarios and scope.

[0016] Other outstanding substantive features and significant advancements of this invention compared to the prior art are further described in detail in the Embodiments section. Attached Figure Description

[0017] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a flowchart of the TinyML-based music interaction response method in Example 1; Figure 2 This is a system structure diagram of Example 2. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] It should be noted that certain names are used in the specification and claims to refer to specific components. It should be understood that those skilled in the art may use different names to refer to the same component. This specification and claims do not distinguish components based on differences in name, but rather on substantial differences in function. As used in this specification and claims, "comprising" or "including" is an open-ended term and should be interpreted as "comprising but not limited to" or "including but not limited to". The embodiments described in the Detailed Description section are preferred embodiments of the present invention and are not intended to limit the scope of the invention.

[0020] Furthermore, those skilled in the art will recognize that various aspects of the present invention can be implemented as a system, method, or computer program product. Therefore, various aspects of the present invention can be specifically implemented in a combination of hardware and software, collectively referred to herein as a "circuit," "module," or "system." Additionally, in some embodiments, various aspects of the present invention can also be implemented as a computer program product contained in one or more microcontroller-readable media, the microcontroller-readable medium containing microcontroller-readable program code. Example 1

[0021] like Figure 1 As shown, this embodiment provides a music interaction response method based on TinyML, which includes the following steps: Acquire and configure TinyML (Tiny Machine Learning) models for audio data analysis and output results for various tasks, including creating datasets and labels, model training, model quantization, and deployment. The creation of the dataset and labels in this embodiment includes: Obtain the raw audio data and its lyrics timestamp information, and create a tag indicating whether there is human voice. After the sound source is separated, the data of each audio track is obtained. Based on the energy of each audio track, combined with the energy of each audio track, speech recognition algorithm, large language model, etc., frame-level information is obtained and target labels are created. The target labels are subjected to de-extreme operations including smoothing, filtering, and merging. Obtaining raw audio data includes, but is not limited to, collaborating with copyright holders of legitimate music to acquire legitimate music data; using open-source datasets; and hiring amateur or professional musicians to produce new songs, etc., which will not be elaborated here. The model training in this embodiment includes: Frame-level supervised learning is performed on the acquired raw audio data and target labels; The loss function is selected based on the task type, and the results are weighted and summed before supervised learning is performed based on the total loss value. Specifically, for the aforementioned single-component classification task, a binary classification loss function related to sigmoid can be selected, such as BCE; for multiple-choice classification tasks, a softmax-related loss function, such as CE, can be selected; and for regression tasks, the MSE loss function can be selected. Since the multiple tasks are independent of each other, appropriate loss functions should be selected for each task, weighted and summed, and finally supervised learning is performed based on the total loss value. In this embodiment, to effectively address the poor performance issues caused by long-tail effects, inaccurate labels, and non-specific features, FocalLoss, pseudo-labels, masking techniques, self-distillation mechanisms, and external distillation mechanisms are introduced to improve model performance. Simultaneously, based on the output results, the structure of the target model is determined while ensuring that the total number of output categories is not less than the required number of categories. Furthermore, a matched optimizer is used for common supervised learning. When the data volume is sufficient, unsupervised learning and semi-supervised learning methods are added. Common data augmentation techniques, such as mixup and SpecAugment, can also be employed, but will not be elaborated upon here.

[0022] The model quantization and deployment in this embodiment include: Quantizing a floating-point model to a fixed-point model, typically from fp32 to int8, can reduce the model size to about one-quarter of its original size. The quantization method can be either dynamic or static, depending on the specific situation. For static quantization, if the performance loss is significant, quantization-aware training can be used for fine-tuning. This is because the aforementioned fixed-point and sparsification processes introduce computational errors, which are compensated for by quantization-aware training. This allows the model to naturally perceive the computational noise introduced by fixed-point and sparsification operations during training and adapt its parameters to these changes, achieving almost lossless model performance.

[0023] Matching determines whether dynamic quantization or static quantization is used.

[0024] Acquire audio data, cache, integrate, and transform the audio data before outputting it in blocks; In this embodiment, the specific steps for outputting audio data in blocks after caching, integrating, and converting the data include: After the audio data stream is buffered to the target data block, it is output for post-processing. In this embodiment, the audio signal is divided into 4-second units. After receiving a 4-second audio stream, the audio waveform is finely processed, merging multiple consecutive peaks and filtering individual narrow peaks. Then, the audio is pre-emphasized to enhance the high-frequency components of the signal. The acquired audio data is converted into the target format data using encoding and decoding methods, such as converting Opus data into PCM data; Non-target sampling rate or precision data is converted to target sampling rate or precision data. Specifically, the sampling rate of the input audio data is analyzed. Since the sound quality of different audio data varies, including 8K, 16K, 24K, 44.1K, 48K, etc., and the data precision also varies, such as 16bit, 20bit, 24bit, etc., the sampling rate and precision need to be unified. For non-target sampling rate and precision, resampling methods such as interpolation and averaging are used to convert them to target sampling rate and precision. After the data is buffered, integrated, and converted, the data to be processed is divided into blocks and input into subsequent modules in sequence. PCEN (Per-Channel Energy Normalization) technology can also be used to normalize the spectrum, suppress background noise and energy fluctuations in the audio signal, and make the spectral characteristics more stable and consistent. The TinyML model is configured to analyze the segmented audio data and output recognition results. These results include voice recognition, accompaniment recognition, target instrument sound recognition, drum beat recognition, rhythm recognition, chord type recognition, sound energy recognition, and lip-sync or amplitude recognition. Specifically, the following steps are included: Data preprocessing involves performing feature transformation on the data to converge the audio time-domain signal into a feature signal, which facilitates further feature extraction, feature analysis, and final judgment by subsequent algorithms. The extraction of common features from feature signals, using single or multiple identical or different head structures to complete and output results for different tasks, can be divided into two parts in this embodiment: First, a feature extraction sharing module, whose main function is to extract common features from the feature signals in the preprocessing results. Second, a multi-result detection module for different target results, whose main function is to use single or multiple identical or different head structures to complete the output of results for different tasks; specifically, this includes a shared CNN Backbone and multiple LSTM (Long Short Term Memory) structure heads. The shared Backbone extracts common features from the spectrum, and each LSTM structure head predicts in real time the start and end times of a specific signal component (such as human voice, accompaniment, drum sounds, strings, etc.) in the audio. Of course, this structure can be reasonably reduced or recombinated according to the device model, which will not be elaborated here.

[0025] The results data from different tasks are processed according to single-component classification tasks, multiple-choice classification tasks, or regression tasks, and the data to be arbitrated is output, specifically including: The results of the single-component classification task are calculated using the sigmoid type, converted into corresponding probability values, and then compared with a preset threshold before being output. After performing softmax calculations on the results data of the multiple-choice classification task and converting them into corresponding probability values, the data with the largest value is output. The results of the regression task are output directly. In this embodiment, the single-component classification task is an independent but not mutually exclusive task, including human voice, accompaniment, instrumental sound, and drum beat; Multiple-choice classification tasks are mutually influential and mutually exclusive tasks, including chord type and rhythm; The regression task is a non-binary or discrete task, including sound energy, lip shape, or amplitude. This step will be explained in further detail: For classification tasks involving single components, such as whether there is a human voice, accompaniment, musical instrument, or drumbeat, since these categories are independent and do not have a mutual exclusion relationship, they may exist simultaneously or not. Therefore, sigmoid calculations can be performed on these results separately to convert the AI ​​results into their respective probability values, which are then compared with their respective thresholds. If the probability is higher than the threshold, the result is considered to exist; otherwise, it is considered not to exist. The threshold is generally 50%, but it can also be selected based on the posterior results of specific situations to better balance performance.

[0026] Furthermore, to avoid glitch-like or random results in the model output, which could lead to false positives or false negatives, the following measures can be taken: 1. Filter the results over time, combining information from multiple frames before and after the test. For example, only if the results in most or all frames show an existence should the result be considered valid; or use median filtering or other smoothing techniques to use the average or median of multiple frames as the final result. 2. Use high and low thresholds for conversion. Only results above the high confidence threshold should trigger a switch, while results below the low confidence threshold should fail. Results in between are considered transitional states and no state switching should occur. For multiple-choice classification tasks, such as determining chord types or musical beats, since there is usually only one complete chord type at a given time point, the final result will be chosen from chord types such as C, D, and E, as well as no chord. Similarly, for beats, the basic beats are finite, and each part will only have one beat. For tasks where types are mutually exclusive and influence each other, softmax calculations can be performed on these mutually exclusive results to convert the AI ​​results into probability values, ensuring the total probability is 1. The case with the highest probability is then selected as the final result. Likewise, to avoid spikes caused by extreme cases, the results can be smoothed or filtered as described above to obtain more stable results.

[0027] For regression tasks, such as measuring vocal energy and lip movement amplitude, the results are not binary or discrete states with multiple possibilities, but rather nearly continuous numerical results. For such numerical regression tasks, the results can be directly used as the final output, or the results can be simply categorized and manually merged into the most recent state for output. To ensure more stable results, smoothing and filtering are still preferable additional methods.

[0028] Arbitration is conducted based on the data to be arbitrated to control music playback and action response, specifically including: Acquire the configuration and status of the action device, confirm the arbitration data that needs to be responded to, and filter out useless arbitration data; Based on the pre-defined logic of the motion device, the response priority of the data to be arbitrated is determined and the action sequence is generated and output sequentially to complete the control or response; Further examples are given below: The system acquires the device's configuration and status, identifies the required response signals, and filters out useless results. For example, if the target motion device is positioned as a guitarist, takes the form of a cat, and has movable parts such as its mouth, tail, and hands, then it can activate responses to the following results: mouth shape and amplitude, chord type, drum beat, and whether there is a guitar sound; and filter out the following results (repeated functions or not supported): whether there is a human voice, accompaniment, or other instruments.

[0029] Based on the activation results and the device's preset logic, the response priority of each result is determined, the required action sequence is generated, and the results are sent to the control modules of each action in an appropriate form. For example: singing lip movements take priority over drum beats; the drum beats will only respond when the singer is not singing; chord results always take priority and can coexist with other results; to increase the fun, the cat singer will periodically stop all actions, make a meowing sound, and yawn, with the highest priority. The generation of action sequences and their sequential output to complete control or response is specifically as follows: The action control module drives the motors to complete the required actions according to the action sequence provided in the output and the time rhythm. For example, when singing, the motor controlling the mouth is driven to simulate the mouth shape and amplitude based on the lip shape and amplitude, thus showing the singing form; when chords are present, preset guitar playing actions are performed according to the chord type, such as the right hand strumming speed and the left hand changing the guitar pressing position, etc.

[0030] At the same time, based on the preset action logic, specific audio is played, and the device's light, electricity, heat and other related peripherals are controlled to improve the interactive experience, which will not be elaborated here.

[0031] The TinyML music interaction response method provided in this embodiment is applied to embedded platforms with limited hardware resources. It can reliably cooperate with the control system, operate stably and efficiently under low computing power conditions, and provide reliable human-computer interaction. In particular, when the algorithm accuracy is insufficient, it can analyze the music content in real time and provide the action and result required by the device, freeing it from the constraints of manual or semi-manual action creation. Moreover, it does not require waiting for excessive processing time, realizing synchronous input parsing, which can greatly reduce computing power requirements, greatly increase the reliability and versatility of interactive response, and has practical value. Example 2

[0032] like Figure 2 As shown, this embodiment provides a music interaction response system based on TinyML. The system includes a mechanical motion module, a controlled drive module, a music flow control module, a data analysis module, and an arbitration control module. The mechanical motion module is configured to complete specified preset actions under the drive of the controlled drive module; The controlled drive module is built into the mechanical motion module and is used to receive instructions from the arbitration control module and drive the mechanical motion module. The music stream control module is configured to acquire audio data, perform audio conversion, and ultimately handle music playback interaction. The data analysis module is configured to analyze audio data and output the analysis results, specifically including: Acquire and configure TinyML models for audio data analysis and output results for various tasks, including creating datasets and labels, model training, model quantization, and deployment. Acquire audio data, cache, integrate, and transform the audio data before outputting it in blocks; The configured TinyML model is used to analyze the segmented audio data and output recognition results, specifically including: Data preprocessing involves performing feature transformation on the data to converge the audio time-domain signal to a feature signal; Extract common features from feature signals, and use one or more identical or different head structures to complete and output result data for different tasks; The results data from different tasks are processed according to single-component classification tasks, multiple-choice classification tasks, or regression tasks, and the data to be arbitrated is output, specifically including: The results of the single-component classification task are calculated using the sigmoid type, converted into corresponding probability values, and then compared with a preset threshold before being output. After performing softmax calculations on the results data of the multiple-choice classification task and converting them into corresponding probability values, the data with the largest value is output. The results of the regression task are output directly. The arbitration control module is configured to output a sequence of response actions based on the analysis results, and to complete the final action control and interactive behavior, specifically including: Acquire the configuration and status of the action device, confirm the arbitration data that needs to be responded to, and filter out useless arbitration data; Based on the pre-defined logic of the motion device, the response priority of the data to be arbitrated is determined and the action sequence is generated for sequential output. The controlled drive module completes the control or response to the mechanical motion module or music flow control module.

[0033] For example, in this embodiment, the convolutional layers in the data analysis module can use traditional convolution operators, depth-wise convolution, point-wise convolution, etc., with dimensions not limited to 1-dimensional or 2-dimensional. Specific model configurations are used to extract features, further integrating low-dimensional features into more unique high-dimensional information. The time-series analysis module can use RNN, GRU, LSTM, and their variants, mainly for time-related contextual changes and feature analysis, capable of focusing on a broader time span and historical information. Here, it can better analyze the intrinsic features and changes in music, such as whether a song is gentle or rock, and adaptively judge the results based on intrinsic features, improving the model's generalization and universality. The attention module can use Transformer and its variant structures, including dot product self-attention, sparse attention, linear / kernel attention, cross attention, multi-head attention, positional encoding, frequency domain attention, etc. In scenarios with sufficient computing power, it can achieve better results than RNN / GRU / LSTM. Attention mechanisms are employed, for example... Non-Transformer and its variants refer to methods that adjust feature influence by extracting features from different dimensions and generating attention coefficients for those dimensions. Examples include: SE modules (channel dimension), scale awareness, spatial awareness, and task awareness, as well as time awareness, which can be derived in this application. Linear layers can use fully connected layers and their variants to integrate the high-level features extracted by the aforementioned modules and perform the final classification and / or regression tasks. Normalization layers, activation functions, and residual connections: Normalization layers are mainly used to adjust the distribution of data, accelerate model learning and convergence, and improve model performance to some extent. In practice, they are often integrated into operators such as convolution and linear layers. Activation functions mainly introduce new features through non-linear operations. Residual connections mainly allow input information to bypass several layers and be directly passed to subsequent layers through skip connections, improving model performance and accelerating convergence. It should be noted that TinyML models can use multiple data analysis modules in combination. The combination of the same or different operators can be in sequential, nested, or parallel structures, which will not be elaborated here. Example 3

[0034] This embodiment provides an electronic device, including: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which enables the at least one processor to perform the TinyML-based music interaction response method described in Embodiment 1. Example 4

[0035] This embodiment provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the TinyML-based music interactive response method as described in Embodiment 1.

[0036] Various implementations of the systems and techniques described in the above embodiments can be implemented in digital electronic circuit systems, integrated circuit systems, application-specific integrated circuits (ASICs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0037] These computational programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0038] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0039] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0040] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

[0041] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0042] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A music interaction response method based on TinyML, characterized in that, The method includes the following steps: Acquire and configure TinyML models for audio data analysis and output results for various tasks, including creating datasets and labels, model training, model quantization, and deployment. Acquire audio data, cache, integrate, and transform the audio data before outputting it in blocks; The configured TinyML model is used to analyze the segmented audio data and output recognition results, specifically including: Data preprocessing involves performing feature transformation on the data to converge the audio time-domain signal to a feature signal; Extract common features from feature signals, and use one or more identical or different head structures to complete and output result data for different tasks; The results data from different tasks are processed according to single-component classification tasks, multiple-choice classification tasks, or regression tasks, and the data to be arbitrated is output, specifically including: The results of the single-component classification task are calculated using the sigmoid type, converted into corresponding probability values, and then compared with a preset threshold before being output. After performing softmax calculations on the results data of the multiple-choice classification task and converting them into corresponding probability values, the data with the largest value is output. The results of the regression task are output directly; Arbitration is conducted based on the data to be arbitrated to control music playback and action response, specifically including: Acquire the configuration and status of the action device, confirm the arbitration data that needs to be responded to, and filter out useless arbitration data; Based on the pre-defined logic of the action device, the response priority of the data to be arbitrated is determined and the action sequence is generated and output sequentially to complete the control or response.

2. The music interaction response method based on TinyML according to claim 1, characterized in that, The creation of the dataset and labels includes: Obtain the raw audio data and its lyrics timestamp information, and create a tag indicating whether there is human voice. After the sound source is separated, the data of each audio track is obtained, and the target label is created by collecting frame-level information based on the energy of each audio track. The target labels are subjected to de-extreme operations including smoothing, filtering, and merging. The model training includes: Frame-level supervised learning is performed on the acquired raw audio data and target labels; The loss function is selected based on the task type, and supervised learning is performed based on the total loss value after weighted summation. The model quantization and deployment include: Quantize the floating-point model into a fixed-point model; Matching determines whether dynamic quantization or static quantization is used.

3. The music interaction response method based on TinyML according to claim 1, characterized in that, The specific steps of caching, integrating, and converting audio data before outputting it in blocks include: The audio data stream is buffered into the target data block and then output for post-processing. The acquired audio data is converted into the target format data using encoding and decoding methods; Convert non-target sampling rate or precision data to target sampling rate or precision data.

4. The music interaction response method based on TinyML according to claim 1, characterized in that, The recognition results include human voice recognition results, accompaniment recognition results, target instrument sound recognition results, drum beat recognition results, rhythm recognition results, chord type recognition results, sound energy recognition results, and lip shape or amplitude recognition results. The single-component classification tasks are independent but not mutually exclusive tasks, including human voices, accompaniment, instrument sounds, and drum beats. The multiple-choice classification task is a mutually influential and mutually exclusive task, including chord type and rhythm; The regression task is a non-binary or discrete task, including sound energy, lip shape, or amplitude.

5. The music interaction response method based on TinyML according to claim 4, characterized in that, The process of processing and outputting the data to be arbitrated based on the results of different tasks, categorized as single-component classification tasks, multiple-choice classification tasks, or regression tasks, also includes: The result data of the classification task is filtered or smoothed in the time dimension to make a comprehensive judgment based on the information data of multiple frames before and after, or to output the average or median of the results of multiple frames. High and low thresholds are used simultaneously to trigger or convert the result data into invalid data.

6. The music interaction response method based on TinyML according to claim 5, characterized in that, The loss function for selecting the matching task type specifically includes: For single-component classification tasks, choose a binary classification loss function related to sigmoid; For multiple-choice classification tasks, choose a softmax-related loss function; For regression tasks, the MSE loss function is selected.

7. The music interaction response method based on TinyML according to claim 6, characterized in that, The model training also includes: Advanced optimizations are introduced, including FocalLoss, pseudo-labels, masking techniques, self-distillation mechanisms, or external distillation mechanisms. Supervised learning performed by the matching optimizer.

8. A music interactive response system based on TinyML, characterized in that, The system includes a mechanical motion module, a controlled drive module, a music flow control module, a data analysis module, and an arbitration control module, wherein... The mechanical motion module is configured to complete a specified preset action under the drive of the controlled drive module; The controlled drive module is built into the mechanical motion module and is used to receive instructions from the arbitration control module and drive the mechanical motion module. The music stream control module is configured to acquire audio data, complete audio conversion, and ultimately perform music playback interaction. The data analysis module is configured to analyze audio data and output analysis results, specifically including: Acquire and configure TinyML models for audio data analysis and output results for various tasks, including creating datasets and labels, model training, model quantization, and deployment. Acquire audio data, cache, integrate, and transform the audio data before outputting it in blocks; The configured TinyML model is used to analyze the segmented audio data and output recognition results, specifically including: Data preprocessing involves performing feature transformation on the data to converge the audio time-domain signal to a feature signal; Extract common features from feature signals, and use one or more identical or different head structures to complete and output result data for different tasks; The results data from different tasks are processed according to single-component classification tasks, multiple-choice classification tasks, or regression tasks, and the data to be arbitrated is output, specifically including: The results of the single-component classification task are calculated using the sigmoid type, converted into corresponding probability values, and then compared with a preset threshold before being output. After performing softmax calculations on the results data of the multiple-choice classification task and converting them into corresponding probability values, the data with the largest value is output. The results of the regression task are output directly; The arbitration control module is configured to output a sequence of response actions based on the analysis results, and to complete the final action control and interactive behavior, specifically including: Acquire the configuration and status of the action device, confirm the arbitration data that needs to be responded to, and filter out useless arbitration data; Based on the pre-defined logic of the motion device, the response priority of the data to be arbitrated is determined and the action sequence is generated for sequential output. The controlled drive module completes the control or response to the mechanical motion module or music flow control module.

9. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the TinyML-based music interactive response method according to any one of claims 1-7.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to execute the TinyML-based music interactive response method according to any one of claims 1-7.

Images