LLM-based intelligent toy control method and device, and storage medium

Abstract

The application discloses an intelligent toy control method and device based on an LLM, and a storage medium. The method comprises the following steps: performing speech endpoint detection and speech recognition on a child voice signal to obtain a plurality of text segments; extracting a fundamental frequency sample value sequence in a specified time interval at the tail of each voice segment, and obtaining a fundamental frequency slope through linear fitting; storing the text segments in a segment buffer in chronological order and splicing the text segments into accumulated text; determining a tone termination according to the fundamental frequency slope of the latest voice segment and a pre-labeled tone termination judgment threshold, and simultaneously determining whether the accumulated text contains a complete predicate structure by using a pre-trained syntactic integrity classifier; when both determinations pass, submitting the accumulated text to a large language model to generate a reply text, converting the reply text into speech and playing the speech, and extracting action keywords to control the intelligent toy to perform corresponding actions.

Description

Technical Field

[0001] This disclosure relates to the field of speech processing technology, specifically to an LLM-based intelligent toy control method, device, and storage medium. Background Technology

[0002] In recent years, smart toys with voice interaction capabilities have gradually adopted large language models to perform semantic understanding and response generation for user input. Existing smart toys typically collect speech through microphones, use a speech endpoint detection module to segment speech into segments based on the duration of silence, then a speech recognition module converts each speech segment into text, inputs the recognized text into a large language model to generate a response, and then a speech synthesis module plays the response content, or controls actuators such as servos and lights to complete corresponding actions based on the response content.

[0003] However, children's language organization and continuous expression abilities are not yet mature. When expressing a complete intention, they are prone to pauses, repetitions, hesitation words, and segmented additions. Existing speech endpoint detection typically uses a uniform silence duration threshold. When a child's pause in the middle of a sentence exceeds this threshold, the complete sentence is segmented into multiple independent speech segments. If the texts obtained from each speech segment are submitted separately to a large language model, the model may generate responses prematurely based on incomplete words or sentences, leading to misunderstandings of intent, frequent follow-up questions, interruptions in dialogue, and even mismatches between speech responses and toy actions. Simply extending the silence detection time increases the interaction waiting time and may incorrectly merge adjacent different instructions. Therefore, how to accurately determine whether a child's intermittent speech constitutes a complete expression while maintaining real-time interaction remains a technical problem that needs to be solved in the control of intelligent toys using large language models. Summary of the Invention

[0004] The purpose of this application is to provide an LLM-based intelligent toy control method, device, and storage medium to solve the problems mentioned in the background art.

[0005] According to a first aspect of this application, an LLM-based intelligent toy control method is provided, comprising the following steps: performing speech endpoint detection on a child's speech signal collected by a microphone, and dividing the speech stream into multiple speech segments; performing speech recognition on each speech segment to obtain corresponding text segments; whenever a speech segment ends, extracting the fundamental frequency sampling value sequence within a specified time interval at the end of the speech segment, and performing linear fitting on the fundamental frequency sampling value sequence to obtain the fundamental frequency slope; storing the text segments in a segment buffer, and concatenating all accumulated text segments in the segment buffer in chronological order to obtain accumulated text; performing dual integrity judgment on the accumulated text: comparing the fundamental frequency slope corresponding to the latest speech segment with a pre-defined intonation termination judgment threshold to determine whether the intonation has terminated; sending the accumulated text into a pre-trained syntactic integrity classifier to determine whether the accumulated text contains a complete predicate structure; when both the intonation termination judgment and the predicate structure integrity judgment pass, submitting the accumulated text to a large language model, which generates a response text; converting the response text into speech playback, and extracting action keywords from the response text to control the intelligent toy to perform corresponding actions.

[0006] Preferably, the intonation termination discrimination threshold is determined by the following method: collecting multiple complete sentences spoken by the target child, performing linear fitting on the fundamental frequency sampling value sequence within the specified time interval at the end of each complete sentence to obtain the fundamental frequency slope corresponding to each sentence, forming a set of slope samples; calculating the mean and standard deviation of the set of slope samples, adding the mean and the standard deviation to obtain the intonation termination discrimination threshold; when the fundamental frequency slope at the end of a certain speech segment is lower than the intonation termination discrimination threshold, it is determined that the intonation has terminated.

[0007] Preferably, the syntactic integrity classifier uses the BERT model, which includes a word embedding layer, a position embedding layer, a multi-layer Transformer encoder, and a linear classification head connected in sequence. The word embedding layer maps the input word identifier sequence into a fixed-dimensional vector representation. The position embedding layer generates a position vector with the same dimension as the output of the word embedding layer, adds it element-wise to the word embedding vector, and then feeds it into the multi-layer Transformer encoder. After processing by the multi-layer Transformer encoder, the output vector corresponding to the position of the classification marker word is taken and fed into the linear classification head, which outputs the probability values ​​of the two categories. The category with the larger probability value is taken as the classification result.

[0008] Preferably, the BERT model adopts a BERT-tiny structure, the Transformer encoder has 2 layers and a hidden dimension of 128; each Transformer encoder layer includes a multi-head self-attention sublayer and a feedforward network sublayer, the multi-head self-attention sublayer has 2 attention heads, each attention head has a dimension of 64, the intermediate hidden dimension of the feedforward network sublayer is 512, and the activation function is GELU; residual connections and layer normalization are set after each sublayer; the linear classification head has an input dimension of 128 and an output dimension of 2.

[0009] Preferably, the training data for the BERT model is constructed as follows: positive samples are children's sentences containing at least one complete predicate structure; negative samples include prefix segments obtained by truncating the positive samples at random positions before the predicate appears, and hesitant filler word segments in the children's speech; the ratio of positive samples to negative samples is 1:2 to 1:3.

[0010] Preferably, a forced submission condition is set for the fragment buffer: when the entry time of the earliest fragment in the fragment buffer exceeds a preset timeout period from the current time, or when the number of accumulated text fragments in the fragment buffer exceeds a preset upper limit for the number of fragments, the accumulated text is directly submitted to the large language model; the fragment buffer is cleared after submission.

[0011] Preferably, when submission is triggered by the forced submission condition, a marker text is appended to the beginning of the accumulated text, indicating that the current input may be incomplete; the large language model provides an understanding response to the input with the marker text based on preset instructions in the system prompts.

[0012] Preferably, before submitting the accumulated text to the large language model, the accumulated text is cleaned according to a preset hesitant filler word list, and text segments that match the entries in the hesitant filler word list are deleted by string matching, and the cleaned text is used as input to the large language model.

[0013] A second aspect of this application also provides an LLM-based intelligent toy control device, comprising: a voice processing module for performing voice endpoint detection on a child's voice signal collected by a microphone, dividing the voice stream into multiple voice segments; performing voice recognition on each voice segment to obtain corresponding text fragments; a fundamental frequency analysis module for extracting a fundamental frequency sampling value sequence within a specified time interval at the end of each voice segment, performing linear fitting on the fundamental frequency sampling value sequence to obtain the fundamental frequency slope; and an integrity discrimination module for storing the text fragments in a fragment buffer, and sorting all accumulated text fragments in the fragment buffer according to time order. The text is concatenated to obtain accumulated text. A dual integrity check is performed on the accumulated text: the fundamental frequency slope corresponding to the latest speech segment is compared with a pre-defined intonation termination threshold to determine if intonation has terminated; the accumulated text is fed into a pre-trained syntactic integrity classifier to determine if it contains a complete predicate structure; an output control module is used to submit the accumulated text to a large language model when both intonation termination and predicate structure integrity checks pass, and the large language model generates a response text; the response text is converted into speech for playback, and action keywords are extracted from the response text to control the smart toy to perform corresponding actions.

[0014] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform the above-described method for controlling an LLM-based intelligent toy.

[0015] This application establishes a speech segment buffering and integrity discrimination mechanism between speech endpoint detection and a large language model. Multiple speech segments formed by children due to pauses, hesitations, or segmented additions are spliced ​​together chronologically. The mechanism combines fundamental frequency changes at the end of speech segments with the predicate structure of the accumulated text to determine whether the current expression has ended, allowing the large language model to generate a response only after obtaining a relatively complete intent. This processing method reduces misunderstandings, premature responses, and invalid follow-up questions caused by incomplete content in individual speech segments, while avoiding increased interaction waiting caused by simply extending the silence discrimination time. By calibrating the intonation termination discrimination threshold based on complete sentence samples from the target children, it can adapt to individual differences among children in terms of fundamental frequency range and sentence-end intonation changes. The segment buffer has timeout and segment number limits, and can also trigger subsequent processing in a timely manner when the expression is not completed for a long time, thus balancing the recognition accuracy of children's intermittent expressions with the response efficiency of intelligent toy voice interaction, and providing a more reliable text input for response voice playback and action control.

[0016] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0017] The accompanying drawings exemplify embodiments and form part of the specification, serving together with the textual description to explain exemplary implementations of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of the claims. Throughout the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0018] Figure 1 A schematic diagram of an LLM-based intelligent toy control method according to an exemplary embodiment of the present disclosure is shown; Figure 2 A flowchart illustrating the dual integrity determination of accumulated text according to an exemplary embodiment of this disclosure is shown. Figure 3 A schematic diagram of the BERT-tiny model structure according to an exemplary embodiment of the present disclosure is shown; Figure 4 A block diagram of an LLM-based intelligent toy control device is shown according to an exemplary embodiment of the present disclosure; Figure 5 A schematic block diagram of a computing device according to an exemplary embodiment of the present disclosure is shown. Detailed Implementation

[0019] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0020] In this disclosure, unless otherwise stated, the use of terms such as "first," "second," etc., to describe various elements is not intended to limit the positional, temporal, or importance relationships of these elements; such terms are merely used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of that element, while in other cases, based on the context, they may refer to different instances.

[0021] The terminology used in the description of the various examples described in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context explicitly indicates otherwise, an element may be one or more unless the number of elements is specifically limited. Furthermore, the term "and / or" as used in this disclosure covers any one of the listed items and all possible combinations thereof.

[0022] It should be noted that all user information (including but not limited to user device information, user personal information, object information corresponding to device usage data, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, device usage data, etc.) involved in all embodiments of this application are information and data authorized by the user or fully authorized by all parties.

[0023] According to embodiments of this disclosure, a method for controlling intelligent toys based on a Large Language Model (LLM) is provided. This method is particularly suitable for voice-interactive intelligent toys designed for children (e.g., 3 to 7 years old). The toy is equipped with at least a microphone acquisition module, a Voice Activity Detection (VAD) module, an Automatic Speech Recognition (ASR) module, a Large Language Model inference module, a Text-to-Speech (TTS) module, and an action execution module. The Large Language Model inference module can be deployed on a cloud server, and the toy interacts with it via network communication; in other embodiments, an edge model with 1B to 3B parameters can be used, with all processing completed locally on the toy. After receiving action commands, the action execution module controls internal actuators such as servos or LEDs to produce limb movements or light changes.

[0024] The embodiments of this disclosure are described in detail below with reference to the accompanying drawings.

[0025] Figure 1 A schematic diagram of an LLM-based smart toy control method according to an exemplary embodiment of the present disclosure is shown, including the following method steps: Step S101, performing voice endpoint detection on the child's voice signal collected by the microphone, dividing the voice stream into multiple voice segments; performing voice recognition on each voice segment to obtain the corresponding text segment.

[0026] In a typical interactive process, the microphone continuously collects the child's voice signal. The VAD module determines the start and end boundaries of the voice segment based on the duration of continuous silence, divides the audio stream into several voice segments, and then sends them to the ASR module for recognition. The recognized text is directly input into the large language model to generate a response. Children aged 3 to 7 often break their speech into multiple segments when expressing a complete intention, interspersed with pauses and hesitation words of 1 to 3 seconds. These pauses exceed the silence detection threshold of the VAD, and the VAD will independently classify each segment as a voice segment. After ASR recognition, each segment is sent to the large language model one by one. A single segment often does not constitute a complete instruction, and the large language model generates follow-up questions or misunderstandings based on incomplete input. This method adds an intention integrity judgment step between the ASR module output and the large language model input, buffering and judging the integrity of text segments, and submitting them all at once after the accumulated text constitutes a complete intention.

[0027] The microphone captures the child's voice signal, which is then fed into the VAD module. The VAD module performs speech endpoint detection on the continuous audio stream and segments the audio stream into multiple speech segments based on the duration of silence. Each speech segment carries a start and end timestamp. The silence detection threshold can be configured, for example, from 500 milliseconds to 1000 milliseconds. Each speech segment is then fed into the ASR module for speech recognition, and the ASR module outputs a text fragment corresponding to each speech segment.

[0028] In step S102, whenever a speech segment ends, the fundamental frequency sampling value sequence within a specified time interval at the end of the speech segment is extracted, and the fundamental frequency sampling value sequence is linearly fitted to obtain the fundamental frequency slope.

[0029] According to embodiments of this disclosure, whenever the VAD module determines the end of a speech segment, it extracts a sequence of fundamental frequency (F0) samples from a specified time interval at the end of the speech segment while simultaneously sending the speech segment to the ASR module. Fundamental frequency refers to the basic frequency of vocal cord vibration in a speech signal, reflecting the speaker's intonation. The intonation trend at the end of a sentence is related to whether the speaker has completed the expression of the current statement: the intonation at the end of declarative and imperative sentences usually decreases, while the intonation is mostly flat or rising when the speaker is pausing or hesitating.

[0030] Specifically, a specified time interval, such as the last 300 milliseconds of a speech segment, is used. Within this interval, fundamental frequency samples are extracted at a fixed frame rate, for example, 100 frames per second. Approximately 30 sampling points are obtained within the 300-millisecond interval. Fundamental frequency extraction can be performed using methods commonly used in this field, such as the autocorrelation method or the YIN algorithm. Some frames may not have detectable valid fundamental frequency values ​​due to factors such as breath sounds or environmental noise. Frames without detected valid fundamental frequency values ​​are discarded, and only valid sampling points are retained.

[0031] For example, the fundamental frequency extraction module and the ASR module multiplex the same PCM audio stream, with a sampling rate of 16kHz, a bit depth of 16bit, and mono. Fundamental frequency extraction uses the YIN algorithm, with a frame length of 25 milliseconds (corresponding to 400 sampling points) and a frame shift of 10 milliseconds (corresponding to 160 sampling points), thus the fixed frame rate is 100 frames per second. The YIN algorithm outputs a fundamental frequency estimate and its periodic confidence level (range 0~1) for each frame. Only frames with a confidence level ≥ 0.6 are retained as valid sampling points, and the remaining frames are discarded. This threshold is based on a preset engineering calibration (test results of 10 children's voices in an anechoic chamber environment) to ensure that the F0 estimation error of valid points is less than ±5Hz. Within a 300-millisecond window at the end of each speech segment, the fundamental frequency extraction module outputs a two-dimensional array, namely, the first dimension is the timestamp array t[i] (unit: seconds, backtracked relative to the end time of the speech segment), and the second dimension is the corresponding fundamental frequency value array f[i] (unit: Hz). This two-dimensional array serves as the input to the linear fitting module.

[0032] The retained effective fundamental frequency sample value sequence is subjected to least-squares linear fitting to obtain the slope of the fitted line, i.e., the fundamental frequency slope. Let there be a total of [number missing] samples within a specified time interval. The effective sampling point, the first The time corresponding to each sampling point is (Unit: seconds), fundamental frequency value is (Unit: Hertz), Fundamental frequency slope Calculate using the following formula: ;in This represents the number of valid sampling points; The dimension is seconds (s); The dimensionless quantity is Hertz (Hz). ), The dimensions are (Right now (), which represents the rate at which the fundamental frequency changes over time. Indicates a drop in tone. It indicates a rising or level tone.

[0033] As an example, if 5 valid sampling points are retained within a 300-millisecond interval, the times are respectively seconds, corresponding to the fundamental frequency values ​​are respectively Hertz, calculated by substituting into the formula. Hz / s indicates that the intonation at the end of the speech segment shows a clear downward trend.

[0034] Optionally, when the number of effective fundamental frequency sampling points is less than 3, the reliability of linear fitting is insufficient. In this case, the fundamental frequency slope is not calculated, and the speech segment is marked as "indeterminate intonation". In the subsequent dual integrity judgment, the speech segment marked as "indeterminate intonation" is treated as if the intonation has not been terminated, and the judgment result of the syntactic integrity classifier determines whether to submit the accumulated text to the large language model.

[0035] In some embodiments, the fundamental frequency range of adult speech is approximately between 85 and 255 Hz, while the fundamental frequency range of children aged 3 to 7 is typically between 200 and 500 Hz, and the variation in fundamental frequency within the same sentence is much greater than that of adults. Significant differences also exist in fundamental frequency characteristics among individual children. Using a uniform fixed slope threshold for intonation termination determination has poor applicability when dealing with different children. This embodiment sets up personalized calibration for each child, establishing a unique intonation termination discrimination threshold for that child using a small sample size.

[0036] In this embodiment, the calibration process is performed when the toy first interacts with a child, or it can be completed by a parent guiding the child. The child is asked to repeat or naturally say several complete sentences, such as short commands like "Hello, little bear," "Tell me a story," or "I want to listen to a song." The number of sentences collected is, for example, 5 to 10. The fundamental frequency sampling value sequence within a specified time interval at the end of each complete sentence is linearly fitted according to the aforementioned steps to obtain the fundamental frequency slope corresponding to each sentence, forming a set of slope samples. ,in The number of complete statements collected.

[0037] Calculate the mean of this set of slope samples. and standard deviation The threshold for determining tone termination is obtained by adding the mean and the standard deviation. : ; Dimensions and fundamental frequency slope Consistent, at Hz / s. Due to the falling intonation at the end of the sentence, It is a negative value; plus back A shift towards a larger numerical value forms a dividing point: in subsequent interactions, the fundamental frequency slope at the end of a certain speech segment is lower than... When the pitch drop reaches or exceeds the child's typical termination level, the pitch is considered to have stopped; when it exceeds... The tone of voice was not terminated at that time.

[0038] For example, if the fundamental frequency slopes at the end of five complete sentences are collected during the calibration phase, respectively... Hz / s, then the mean Hz / s, standard deviation Hz / s, threshold Hz / s. For a certain speech segment in subsequent interactions, if the fundamental frequency slope at the end is... Hz / s, lower than The tone of voice has been stopped; if it is Hz / s, higher than The tone of voice was not terminated.

[0039] The calibrated intonation termination threshold is stored in the local storage of the smart toy and can be directly read and used in subsequent interactions. Optionally, after accumulating a certain amount of interaction data, the fundamental frequency slope at the end of the speech segment that is successfully judged as a complete sentence in daily interactions can be added as a new sample to the original slope sample set, and the mean and standard deviation can be recalculated to update the intonation termination threshold, so that it gradually adapts to the changes in children's speech as they grow.

[0040] In step S103, the text segments are stored in a segment buffer, and a dual integrity check is performed on the accumulated text. Specifically, the text segments are stored in the segment buffer, and all accumulated text segments in the segment buffer are concatenated in chronological order to obtain the accumulated text; a dual integrity check is performed on the accumulated text: the fundamental frequency slope corresponding to the latest speech segment is compared with a pre-defined intonation termination threshold to determine whether the intonation has terminated; the accumulated text is then fed into a pre-trained syntactic integrity classifier to determine whether the accumulated text contains a complete predicate structure.

[0041] Figure 2 A flowchart illustrating the performance of dual integrity determination on accumulated text according to an exemplary embodiment of this disclosure is shown, such as... Figure 2 As shown, in step S201, the text fragment is stored in a fragment buffer, and all accumulated text fragments in the fragment buffer are concatenated in chronological order to obtain accumulated text. According to an embodiment of this disclosure, a fragment buffer is set up to temporarily store text fragments that have not yet been determined to be complete intents. The fragment buffer stores each text fragment and its corresponding fundamental frequency slope value in chronological order.

[0042] After each speech segment is determined by VAD, the text segment output by the ASR module is stored in the segment buffer, and the fundamental frequency slope of that speech segment generated in the previous steps is stored synchronously. Then, all the accumulated text segments in the segment buffer are concatenated in chronological order to obtain the accumulated text, which is then subjected to the following two checks. In one implementation, for the same speech segment, the extraction of the fundamental frequency sampling value sequence and speech recognition are performed in parallel. Both receive the same speech segment audio data, complete their processing independently, and then synchronously write their respective outputs to the segment buffer.

[0043] In step S202, dual integrity discrimination is performed on the accumulated text. Specifically, regarding intonation termination check: the fundamental frequency slope corresponding to the latest speech segment in the segment buffer is taken, and compared with the intonation termination discrimination threshold obtained by the aforementioned calibration . If the fundamental frequency slope is lower than , it is determined that the intonation has terminated; if the fundamental frequency slope is greater than or equal to , or the speech segment is marked as "intonation undetermined" due to insufficient effective sampling points, it is determined that the intonation has not terminated.

[0044] In this embodiment, the method further includes discrimination by a syntactic integrity classifier, that is, the accumulated text is fed into a pre-trained syntactic integrity classifier to determine whether the accumulated text contains a complete predicate structure, that is, whether it contains the backbone component of an action or a request. The predicate structure in this embodiment refers to a structural unit in Chinese syntactic analysis that includes a verb or verb phrase and its necessary arguments. For example, "let the bear" lacks an action predicate and is judged not to contain a complete predicate structure; "let the bear sing" contains a complete predicate structure; a segment containing only hesitation words such as "um that" does not contain any predicate component. The classifier outputs probability values of two categories, and the category with the larger probability value is taken as the discrimination result.

[0045] According to some embodiments, the syntactic integrity classifier adopts a BERT model, specifically a BERT-tiny structure. Children's utterances are usually short and have simple syntactic structures, so the complexity of this classification task is limited. The parameter scale of BERT-tiny is sufficient for the task, and it has fast inference speed, which is suitable for running on the embedded processor at the toy end side.

[0046] Figure 3 shows a schematic diagram of the BERT-tiny model structure according to an exemplary embodiment of the present disclosure, and the overall architecture of the model is as follows: the model is composed of a word embedding layer, a position embedding layer, 2 layers of Transformer encoders, and 1 linear classification head connected in sequence.

[0047] Specifically, the word embedding layer maps the input token ID sequence to a 128-dimensional vector representation. The position embedding layer generates a 128-dimensional position vector for each position in the sequence, which is added element-wise with the word embedding vector, and the obtained result is used as the input representation sequence of the Transformer encoder. The maximum length of the input sequence is set to 32 tokens, the classification marker token ([CLS]) is fixedly placed at the first position; positions with less than 32 tokens are filled with zero vectors, and sequences with more than 32 tokens are truncated to 32 tokens.

[0048] A two-layer Transformer encoder is stacked serially. Each layer contains a multi-head self-attention sublayer and a feedforward sublayer. The multi-head self-attention sublayer uses two attention heads, each with a dimension of 64, for a total hidden dimension of 128. The feedforward sublayer has an input dimension of 128, intermediate hidden dimensions of 512, uses GELU activation, and has an output dimension of 128. Each sublayer is followed by a residual connection and layer normalization, which involves adding the sublayer's input and output and then normalizing the result.

[0049] After processing by a two-layer Transformer encoder, the 128-dimensional vector corresponding to the position of the classification marker ([CLS]) in the output sequence is fed into the linear classification head. The linear classification head is a fully connected layer with an input dimension of 128 and an output dimension of 2, corresponding to the two categories of "semantic incompleteness" and "semantic completeness" respectively. The output of the fully connected layer is normalized by the Softmax function to obtain the probability values ​​of each category, and the category with the larger probability value is taken as the classification result.

[0050] The model's input data is a sequence of token identifiers from accumulated text processed by a tokenizer. The data format is a two-dimensional integer tensor with dimension 1. ,in `32` represents the batch size, and `32` represents the maximum sequence length. The input is generated by concatenating all accumulated text fragments in the fragment buffer in chronological order, followed by word segmentation. The model's output data is a two-dimensional floating-point tensor with dimension `32`. , where represents the probability value of each input corresponding to the two categories. Subsequently, the probability value of the component corresponding to the "semantic integrity" category is taken as the criterion: if it is greater than 0.5, the cumulative text is determined to contain a complete predicate structure; otherwise, it is determined not to contain it.

[0051] The syntactic dependency relations formed by predicates and arguments in children's sentences determine whether the text expresses a complete instruction. After the input lexical identifier sequence is modeled in context by the Transformer encoder, the vector at the [CLS] position aggregates the syntactic information of the entire sequence. The linear classification head uses this to determine whether there is a complete predicate dependency structure in the sequence. There is a direct correspondence between the syntactic completeness of the input text and the output class probability.

[0052] In one example, a child says three speech fragments in sequence: "Let the little bear," "um," and "Sing a song." The VAD (Voice over Action) classifies each fragment and generates three text fragments. After the first fragment enters the fragment buffer, the accumulated text is "Let the little bear," which is then fed into the syntactic completeness classifier. The probability of the "semantic completeness" category is 0.15, which is below 0.5, so it is determined that it does not contain a complete predicate structure. After the second fragment arrives, the accumulated text changes to "Let the little bear um," with a probability of 0.12, and it is still determined that it does not contain a complete predicate structure. After the third fragment arrives, the accumulated text changes to "Let the little bear um sing a song," with a probability of 0.88, which is above 0.5, so it is determined that it contains a complete predicate structure.

[0053] According to some embodiments, the training process of the BERT-tiny model is as follows: the training data consists of positive samples and negative samples. Positive samples are children's sentences containing at least one complete predicate structure. The sources include transcripts of children's interactions with voice toys and publicly available children's speech corpora, such as transcripts of speakers under 18 years old in the publicly available children's speech corpus AISHELL-3. A total of 8,000 complete sentences containing at least one verb-object structure or subject-predicate structure are selected. 3,000 children's sentences marked as "complete instructions" are extracted from the interaction logs of the smart toy prototype. Negative samples consist of two parts: first, prefix segments obtained by randomly truncating positive sample sentences before the predicate, for example, randomly truncating before the predicate verb to obtain 4000 prefix segments; second, segments of hesitation-filling words commonly found in children's speech, such as "um," "that," and "that is to say," which form separate segments. The ratio of positive to negative samples is 1:2 to 1:3, and they are divided into a training set (13600 samples) and a validation set (3400 samples) at an 8:2 ratio. The validation set does not participate in gradient updates; it is only used to evaluate accuracy and save optimal model parameters at the end of each round. The loss function uses binary cross-entropy loss, and the loss value is calculated based on the model's predicted class probability value for each sample and the true label. The specific calculation method is as follows: ;in For batch size, For the first The true label of each sample, taking the value 0 or 1. The model predicts the probability that the sample belongs to the semantically complete category. The optimizer uses the Adam optimizer, with an initial learning rate set, for example, to [value missing]. The learning rate adjustment strategy is cosine annealing, meaning the learning rate gradually decays from the initial value to near zero according to a cosine curve. The number of training epochs is, for example, 20 to 50, and the batch size is, for example, 32 to 64. Dropout is applied after the outputs of the self-attention sublayer and feedforward sublayer of each Transformer encoder layer, with a dropout ratio of 0.1 to mitigate overfitting. During training, the final model parameters are selected based on the classification accuracy on the validation set at the end of each epoch. After training, the model parameters are fixed and deployed to the toy's edge processor; the parameters are not updated during the inference phase.

[0054] In the toy-side reasoning process, the single input tensor dimension is [1, 32], and the data type is int32 (word index sequence). The classifier's output tensor dimension is [1, 2], and the data type is float32. The component corresponding to index 1 (i.e., the probability of the "semantic integrity" category) is compared with 0.5 to obtain the judgment result.

[0055] According to embodiments of this disclosure, the BERT-tiny edge uses the TFLite Micro inference framework to perform dynamic range quantization (INT8 weight quantization, activation values ​​remain float32) on the trained BERT-tiny model. The quantized model file size is approximately 1.2MB, and the runtime memory usage (including input / output tensors and intermediate activation cache) is approximately 2.8MB. On an ARM Cortex-M7 core (400MHz), the measured single inference latency is 85~120 milliseconds (including word segmentation preprocessing), meeting the real-time requirements for interactive operation (<200 milliseconds).

[0056] In step S104, when both the intonation termination judgment and the predicate structure integrity judgment pass, the accumulated text is submitted to the large language model, which generates a response text; the response text is converted into speech and played, and action keywords are extracted from the response text to control the smart toy to perform the corresponding action.

[0057] According to the embodiments of this disclosure, when both the intonation termination determination and the predicate structure integrity determination pass, that is, the intonation determination of the latest speech segment is terminated and the syntactic integrity classifier determines that the accumulated text contains a complete predicate structure, the accumulated text is marked as ready to be submitted and enters the subsequent text cleaning and large language model processing stage; the segment buffer is then cleared, ready to receive the next round of speech segments. If either of the two determinations fails, the system continues to wait for the subsequent speech segments to arrive in the segment buffer.

[0058] According to some embodiments, to prevent the fragment buffer from waiting indefinitely when a child is distracted midway, a forced submission condition is set for the fragment buffer. When the time elapsed since the earliest fragment in the fragment buffer was buffered exceeds a preset timeout duration, or the number of accumulated text fragments in the fragment buffer exceeds a preset upper limit of fragment count, the accumulated text is directly submitted without waiting for the double integrity judgment result. The timeout duration can be configured as 8 seconds, for example, which is determined based on the fact that the average duration of a single continuous expression of intent by children aged 3 to 7 years usually does not exceed 6 seconds, plus a 2-second tolerance; the upper limit of fragment count can be configured as 5, for example, which is based on the fact that the number of fragments generated by pauses when children of this age group express a complete intent usually does not exceed 4, plus a 1-piece tolerance. Forcing submission is triggered when either of the two conditions is met. Optionally, the above two thresholds can be adjusted according to the statistical results of interaction logs after actual deployment.

[0059] Wherein, when submission is triggered by the forced submission condition, a piece of marker text is added to the front of the accumulated text to indicate that the current input may be incomplete. According to the preset instructions in the system prompt, the large language model preferentially tries to understand the existing content and give a preliminary response to the input with the marker text. The fragment buffer is cleared after the submission is completed.

[0060] Before the accumulated text is submitted to the large language model, text cleaning is required. The accumulated text is cleaned according to a preset hesitation filler vocabulary, text segments matching the entries in the hesitation filler vocabulary are retrieved and deleted in a string matching manner, and the cleaned text is used as the input submitted to the large language model. The hesitation filler vocabulary includes common hesitation filler entries in children's speech such as "um", "that", "just", "that is to say", "then", "ah", "uh", etc., and the vocabulary can be supplemented according to the occurrence frequency in the interaction logs after actual deployment.

[0061] The cleaned text and the dialogue history record together form the input of the large language model, and the large language model generates response text according to the role and content specification set in the system prompt. Wherein, optionally, the dialogue history record adopts a fixed window strategy, and only retains the latest 5 rounds of interactions in the current session (each round includes user input and LLM response). When the accumulated number of rounds exceeds 5, the earliest round is discarded. Each history record is spliced in the format of "User: {text}" and "Assistant: {text}", and the system prompt is added at the very front. If the total number of tokens after the history records are spliced exceeds 70% of the LLM context window (for example, 2048 tokens), the earliest history record is truncated preferentially until the length requirement is met.

[0062] The response text is sent to the TTS module for speech synthesis, and the synthesized audio is played through the toy's speaker. Simultaneously, action keywords, such as "nodding," "waving," and "dancing," are extracted from the response text. Matched keywords are mapped to preset action control sequences, controlling the smart toy to perform the corresponding actions. If no action keywords are matched in the response text, only the voice is played. The keyword-action mapping relationship is stored locally on the toy in the form of a configuration table, and developers or parents can optionally add or modify mapping entries in the backend management interface.

[0063] For example, the action mapping table is stored in the toy's local storage in JSON format, and each entry contains three fields: keyword (keyword string), action_id (action number), and control_sequence (servo / motor control byte sequence). Example entries: {"keyword":"nodding","action_id":1,"control_sequence": [0xA1,0x05,0x0A,0x00]} indicates controlling the neck servo to rotate from 0° to 15° and then back to 0° within 0.5 seconds; {"keyword":"waving","action_id":2,"control_sequence":[0xA2, 0x08, 0x1E, 0x0A]} indicates controlling the shoulder servo to complete a 30° reciprocating swing within 0.8 seconds; {"keyword":"dancing","action_id":3,"control_sequence":[0xA3,0x0F,0x14,0x0A,0xA1,0x05, 0x0F,0x0A]} indicates a combined action (head shaking + hand waving).

[0064] Optionally, if multiple action keywords are matched in the response text, they are executed sequentially according to the matching order, with a 0.3-second interval between each action and the next one; newly arriving voice commands during action execution are temporarily stored in the waiting queue.

[0065] By applying the above method, a dual integrity check step of fundamental frequency slope discrimination and syntactic integrity discrimination is added between speech endpoint detection and the large language model. Multiple incomplete speech segments generated by children's intermittent expression are temporarily stored and spliced ​​in a buffer. After both intonation and semantic dimensions are judged to be complete, they are submitted in a unified manner. Without prolonging the silence waiting threshold, the intermittent speech is restored to complete intention input, reducing the situation where the large language model generates follow-up questions or misunderstandings due to incomplete input. At the same time, the personalized intonation termination discrimination threshold is adapted to the fundamental frequency differences between different children, reducing the misjudgment rate of intonation discrimination.

[0066] According to one aspect of this disclosure, an LLM-based intelligent toy control device is also provided. Figure 4 A structural block diagram of an LLM-based intelligent toy control device according to an exemplary embodiment of the present disclosure is shown. Figure 4 As shown, the device specifically includes: a speech processing module 401, used to perform speech endpoint detection on the child's speech signal collected by the microphone, and to segment the speech stream into multiple speech segments; to perform speech recognition on each speech segment to obtain the corresponding text segment; a fundamental frequency analysis module 402, used to extract the fundamental frequency sampling value sequence within a specified time interval at the end of each speech segment, and to perform linear fitting on the fundamental frequency sampling value sequence to obtain the fundamental frequency slope; and an integrity discrimination module 403, used to store the text segment in a segment buffer, and to concatenate all the accumulated text segments in the segment buffer in chronological order to obtain the accumulated text segment. The text is processed by performing dual integrity judgment on the accumulated text: comparing the fundamental frequency slope corresponding to the latest speech segment with a pre-defined intonation termination judgment threshold to determine whether the intonation has terminated; feeding the accumulated text into a pre-trained syntactic integrity classifier to determine whether the accumulated text contains a complete predicate structure; and outputting a control module 404, which, when both intonation termination judgment and predicate structure integrity judgment pass, submits the accumulated text to a large language model, which generates a response text; converts the response text into speech playback, extracts action keywords from the response text, and controls the smart toy to perform corresponding actions.

[0067] It should be noted that the working process of each module in the LLM-based intelligent toy control device described in this embodiment can refer to the working process of the LLM-based intelligent toy control method described in the above embodiments, and the technical effect achieved is the same as that of the LLM-based intelligent toy control method described in the above embodiments, so it will not be repeated here.

[0068] See Figure 5 The following description serves as a structural block diagram of the electronic device described herein, which is an example of a hardware device applicable to various aspects of this disclosure. The electronic device can be different types of computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the disclosure described and / or claimed herein.

[0069] Figure 5 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. Figure 5As shown, the electronic device 500 may include at least one processor 501, working memory 502, I / O device 504, display device 505, storage device 506 and communication interface 507 that are capable of communicating with each other via system bus 503.

[0070] Processor 501 may be a single processing unit or multiple processing units, and all processing units may include single or multiple computing units or multiple cores. Processor 501 may be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuits, and / or any device that manipulates signals based on operating instructions. Processor 501 may be configured to acquire and execute computer-readable instructions stored in working memory 502, storage device 506, or other computer-readable media, such as program code of operating system 502a, program code of application program 502b, etc.

[0071] Working memory 502 and storage device 506 are examples of computer-readable storage media for storing instructions executed by processor 501 to perform the various functions described above. Working memory 502 may include both volatile and non-volatile memory (e.g., RAM, ROM, etc.). Furthermore, storage device 506 may include hard disk drives, solid-state drives, removable media including external and removable drives, memory cards, flash memory, floppy disks, optical disks (e.g., CDs, DVDs), storage arrays, network-attached storage, storage area networks, etc. Working memory 502 and storage device 506 may be collectively referred to herein as memory or computer-readable storage media, and may be non-transitory media capable of storing computer-readable, processor-executable program instructions as computer program code, which may be executed by processor 501 as a specific machine configured to perform the operations and functions described in the examples herein.

[0072] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0073] While embodiments or examples of this disclosure have been described with reference to the accompanying drawings, it should be understood that the methods, apparatus, and devices described above are merely exemplary embodiments or examples, and the scope of the invention is not limited by these embodiments or examples, but only by the granted claims and their equivalents. Various elements in the embodiments or examples may be omitted or replaced by their equivalents. Furthermore, the steps may be performed in a different order than that described in this disclosure. Further, various elements in the embodiments or examples may be combined in various ways. Importantly, as the technology evolves, many elements described herein can be replaced by equivalents that appear after this disclosure.

Claims

1. A method for controlling intelligent toys based on LLM, characterized in that, The process includes the following steps: performing speech endpoint detection on the child's speech signal collected by the microphone, dividing the speech stream into multiple speech segments; performing speech recognition on each speech segment to obtain the corresponding text fragment; extracting the fundamental frequency sampling value sequence within a specified time interval at the end of each speech segment, performing linear fitting on the fundamental frequency sampling value sequence to obtain the fundamental frequency slope; storing the text fragment in a fragment buffer, and concatenating all the accumulated text fragments in the fragment buffer in chronological order to obtain the accumulated text; The accumulated text undergoes dual integrity determination: the fundamental frequency slope corresponding to the latest speech segment is compared with a pre-defined intonation termination threshold to determine whether intonation has terminated; the accumulated text is fed into a pre-trained syntactic integrity classifier to determine whether the accumulated text contains a complete predicate structure; when both intonation termination determination and predicate structure integrity determination pass, the accumulated text is submitted to a large language model, which generates a response text; the response text is converted into speech for playback, and action keywords are extracted from the response text to control the smart toy to perform corresponding actions.

2. The method according to claim 1, characterized in that, The intonation termination detection threshold is determined as follows: multiple complete sentences spoken by the target child are collected, and the fundamental frequency sampling value sequence within the specified time interval at the end of each complete sentence is linearly fitted to obtain the fundamental frequency slope corresponding to each sentence, forming a set of slope samples; the mean and standard deviation of the set of slope samples are calculated, and the mean and standard deviation are added together to obtain the intonation termination detection threshold; when the fundamental frequency slope at the end of a certain speech segment is lower than the intonation termination detection threshold, it is determined that the intonation has terminated.

3. The method according to claim 1, characterized in that, The syntactic integrity classifier uses the BERT model, which includes a word embedding layer, a position embedding layer, a multi-layer Transformer encoder, and a linear classification head connected in sequence. The word embedding layer maps the input word identifier sequence into a vector representation of fixed dimension. The position embedding layer generates a position vector with the same dimension as the output of the word embedding layer, and adds it element by element to the word embedding vector before feeding it into the multi-layer Transformer encoder. After processing by the multi-layer Transformer encoder, the output vector corresponding to the position of the classification tag word is taken and sent to the linear classification head, which outputs the probability values ​​of the two categories. The category with the larger probability value is taken as the classification result.

4. The method according to claim 3, characterized in that, The BERT model adopts the BERT-tiny structure. The Transformer encoder has two layers with a hidden dimension of 128. Each Transformer encoder layer contains a multi-head self-attention sublayer and a feedforward network sublayer. The multi-head self-attention sublayer has two attention heads, each with a dimension of 64. The feedforward network sublayer has an intermediate hidden dimension of 512 and uses GELU as the activation function. Residual connections and layer normalization are set after each sublayer. The linear classification head has an input dimension of 128 and an output dimension of 2.

5. The method according to claim 3, characterized in that, The training data for the BERT model is constructed as follows: positive samples are children's sentences containing at least one complete predicate structure; negative samples include prefix segments obtained by truncating the positive samples at random positions before the predicate appears, as well as hesitant filler word segments in children's speech; the ratio of positive samples to negative samples is 1:2 to 1:

3.

6. The method according to claim 1, characterized in that, A forced submission condition is set for the fragment buffer: when the earliest fragment in the fragment buffer is buffered more than a preset timeout period from the current time, or when the number of accumulated text fragments in the fragment buffer exceeds a preset upper limit for the number of fragments, the accumulated text is directly submitted to the large language model; the fragment buffer is cleared after submission.

7. The method according to claim 6, characterized in that, When a submission is triggered by the forced submission condition, a marker text is appended to the beginning of the accumulated text, indicating that the current input may be incomplete; the large language model responds to the input with the marker text according to the preset instructions in the system prompts.

8. The method according to claim 1, characterized in that, Before submitting the accumulated text to the large language model, the accumulated text is cleaned according to a preset hesitant filler vocabulary. Text segments that match the entries in the hesitant filler vocabulary are deleted by string matching. The cleaned text is then used as input to the large language model.

9. A smart toy control device based on LLM, characterized in that, include: The speech processing module is used to perform speech endpoint detection on the child's speech signal collected by the microphone, and to segment the speech stream into multiple speech segments; to perform speech recognition on each speech segment to obtain the corresponding text fragment; the fundamental frequency analysis module is used to extract the fundamental frequency sampling value sequence within a specified time interval at the end of each speech segment, and to perform linear fitting on the fundamental frequency sampling value sequence to obtain the fundamental frequency slope; the integrity discrimination module is used to store the text fragments into a fragment buffer, and to concatenate all the accumulated text fragments in the fragment buffer in chronological order to obtain the accumulated text; Perform dual integrity judgment on the accumulated text: compare the fundamental frequency slope corresponding to the latest speech segment with the pre-calibrated intonation termination judgment threshold to determine whether the intonation has terminated; send the accumulated text into a pre-trained syntactic integrity classifier to determine whether the accumulated text contains a complete predicate structure; The output control module is used to submit the accumulated text to the large language model when both the intonation termination judgment and the predicate structure integrity judgment pass, so that the large language model generates the response text; convert the response text into speech playback, and extract action keywords from the response text to control the smart toy to perform the corresponding action.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-8.

Images