Toy interaction control method and device based on large language model, terminal and medium

Abstract

The invention provides a toy interaction control method and device based on a large language model, a terminal and a medium, and the method comprises the steps: obtaining user side information which comprises one or more of voice, text, image, limb movement or external environment information; performing feature extraction on the user side information to generate feature representation; inputting the feature representation into a large language model, and identifying a user intention; generating a corresponding control instruction according to the identified user intention; and converting the control instruction into a specific control signal, and sending the specific control signal to a control system of the toy so as to execute corresponding control. The large language model is utilized to perceive human language characters, voice emotions, limb actions and the like, human intentions are accurately recognized, then the feedback of the toy is controlled, and intelligent interaction is formed for users.

Description

Technical Field

[0001] The present invention relates to the intersection of artificial intelligence and intelligent device control technology, and in particular to a toy interactive control method, device, terminal and medium based on a large language model. Background Art

[0002] There are many structures and types of existing toys, such as toy dolls, etc., but most of them are relatively simple traditional toys, which usually require users to control them according to the preset operation methods of the toys. They lack intelligent interaction with users and cannot accurately understand the intentions expressed by users through language (text), voice (emotion) and body movements (intensity, frequency), etc., and the overall experience is not ideal.

[0003] At the same time, existing technologies have not yet achieved comprehensive processing of multimodal information (such as voice, text, images, body movements, and external environmental information), and lack methods to accurately identify user intentions and generate corresponding control instructions using deep learning models (such as large language models). In addition, existing technologies still have major deficiencies in terms of personalized user experience, real-time feedback, and security assurance, making it difficult to meet users' higher demands for intelligent and emotional interaction. Summary of the invention

[0004] The present invention aims to solve the problems of lack of intelligent interaction, inaccurate recognition of user intentions and insufficient personalized experience in the prior art toys, and provides a toy interactive control method based on a large language model and its related devices, terminals and computer-readable media. The present invention realizes intelligent interaction between toys and users through the combination of multimodal information processing and deep learning models, significantly improving user experience.

[0005] A first aspect of the present invention provides a toy interactive control method based on a large language model, comprising the following steps: Acquire user terminal information, where the user terminal information includes one or more of voice, text, image, body movement or external environment information; Extracting features from the user-side information to generate feature representation; Inputting the feature representation into a large language model to identify user intent; Generate corresponding control instructions according to the identified user intention; The control instructions are converted into specific control signals and sent to the control system of the toy to execute corresponding actions.

[0006] Optionally, the extracting features of the user terminal information to generate feature representation includes: When the user terminal information is one of voice, text, image, body movement or external environment information, directly generating a feature representation of a single modality; When the user-side information is any combination of two or more of voice, text, image, body movement or external environment information, features are extracted for each type of information respectively, and multimodal features are fused to generate a unified feature representation.

[0007] Optionally, extracting features from each type of information respectively and fusing multimodal features to generate a unified feature representation includes: Perform feature vectorization on each type of information to obtain the features of each modality; Use the attention mechanism to align the features of each modality to enhance the correlation between features; The aligned features are fused into a unified feature representation through the feature fusion network.

[0008] Optionally, the large language model includes one or more of the following functional units: A language understanding unit, used to parse and understand the text semantic features in the user-side information; A voice emotion recognition unit, used to analyze the emotional characteristics of the voice in the user-side information; A facial expression recognition unit, used to analyze the facial expression semantic features of the image in the user terminal information; An action recognition unit, used to understand the semantics of the body movements in the user terminal information; The context understanding unit is used to combine historical interaction information and current conversation content to build a semantic understanding framework; When including multiple functional units, the large language model also includes: The multimodal semantic fusion unit is used to fuse the semantic analysis results of the various functional units mentioned above to generate a unified semantic understanding.

[0009] Optionally, the user intention further includes a quantitative feature of an emotional state; correspondingly, the process of generating a control instruction includes: Based on the recognition of user intention, the behavioral goal of the user intention is extracted; Analyzing the quantitative characteristics of the emotional state to generate emotional parameters for adjusting device operation; A control instruction is generated by combining the behavioral goal of the user's intention, the emotional parameter and the external environment information.

[0010] Optionally, the control instruction includes one or more of the following: Motion control instructions for adjusting one or more of the toy's motion mode, intensity or frequency, where the parameter range of the motion mode is a set of preset motion modes; Voice control instructions for selecting a preset voice feedback template and adjusting voice parameters according to the quantitative characteristics of the emotional state; Atmosphere control commands are used to adjust one or more of the lighting brightness, color, flashing frequency, or ambient music to match the user's mood.

[0011] Optionally, the control method further includes: Determining the priority of each of the control instructions based on a preset weight of the user intention; Coordinate the execution order of multiple control instructions to ensure coordination and non-conflict between the control instructions; Collect real-time feedback data from users, including behavioral responses, and dynamically adjust control strategies to optimize user experience.

[0012] Optionally, the control method further includes a continuous learning module, which optimizes user intention recognition and control strategy in the following ways: Collect user interaction data and feedback information; Use deep learning algorithms and reinforcement learning algorithms to dynamically update the parameters of large language models; Adaptively adjust control logic to meet the user's personalized needs.

[0013] Optionally, the control method further includes a safety module, which is used to: Perform multi-factor authentication and authorization management on user identities to ensure that only authorized users can control the toy; End-to-end encryption of user data to prevent unauthorized access and data leakage; The execution process of the control instructions is monitored in real time, and abnormal instructions are detected and blocked to ensure safe use.

[0014] Optionally, the control method further includes: Remote update module, which downloads and installs the latest large language model parameters or control logic through a secure network connection or cloud platform. The update process uses a digital signature and verification mechanism to ensure the authenticity and integrity of the update content.

[0015] A second aspect of the present invention provides a toy interactive control device based on a large language model, comprising: Information acquisition module: acquires user-side information, wherein the user-side information includes one or more of voice, text, image, body movement or external environment information; Feature extraction module: extracts features from the user-side information and generates feature representation; Large language model module: inputs the feature representation into the large language model to identify the user's intention; Instruction generation module: generates corresponding control instructions according to the identified user intention; Control signal generation module: converts the control instruction into a specific control signal and sends it to the control system of the toy to execute the corresponding action.

[0016] According to a third aspect of the present invention, there is provided an electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the toy interactive control method based on a large language model are implemented.

[0017] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the toy interactive control method based on a large language model are implemented.

[0018] Compared with the prior art, the embodiments of the present invention have at least one of the following beneficial effects: The toy interactive control method based on the large language model in the embodiment of the present invention can realize intelligent interaction: using the large language model to perceive the user's multimodal information such as voice, text, image, body movement, etc., accurately identify the user's intention, and generate control instructions that meet the user's needs, thereby realizing intelligent interaction between the toy and the user.

[0019] The toy interactive control method based on the large language model in the embodiment of the present invention can realize personalized feedback: by dynamically adjusting the control parameters in combination with the quantitative characteristics of the user's emotional state and the external environmental information, it can provide personalized action, voice or atmosphere feedback according to the user's real-time needs, significantly improving the immersion and realism of the user experience.

[0020] The toy interaction control method based on a large language model in the embodiment of the present invention is technologically advanced: through the large language model and multimodal fusion technology, it overcomes the problems of the existing toy interaction mode being single and the response being unintelligent, and has higher interaction accuracy and adaptability.

[0021] The toy interactive control method based on the large language model in the embodiment of the present invention has system security: it provides security guarantees such as multi-factor authentication, data encryption and abnormal instruction detection mechanism, ensures the security of user data and device control process, and further enhances the user's sense of trust. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings: Figure 1 is a flow chart of a toy interactive control method based on a large language model in an exemplary embodiment of the present invention; Figure 2A flow chart is generated for the feature representation corresponding to the multimodal feature in an exemplary embodiment of the present invention; Figure 3 is a functional unit block diagram of a large language model in an exemplary embodiment of the present invention; Figure 4 FIG. 1 is a block diagram of an electronic device in an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0023] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several variations and improvements may be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0024] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the present invention are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0025] The toys described in the present invention may be various physical toys designed and produced to achieve interaction with users, including but not limited to children's toys (such as dolls, etc.), emotion comfort toys (such as chat robots and other elderly care toys that solve the loneliness problem of elderly people living alone), and other toys that can mobilize or affect the emotions of users. However, these toys currently lack interactivity and the user experience is not good. In order to enhance the interactive experience, the embodiment of the present invention provides a toy interactive control method based on a large language model, which increases intelligent interaction and provides users with a more immersive interactive method.

[0026] In the following embodiments, for children's toys (such as dolls), emotional comfort toys (such as toys for elderly care), etc., in order to achieve intelligent control, information collection components, such as sensors, induction sheets, etc., and some controllable components, such as action components and drive components of specific parts of the toy, etc., can be set on the toy itself or the user. Exemplarily, for a toy doll, a corresponding sensor can be set at each joint to sense the movement of the limbs, and a drive component, such as a motor, a servo motor, a pneumatic component or a hydraulic component, etc., can also be set, and the movement of specific joints or components can be controlled according to the control instructions to realize the movement of the body parts of the toy doll. Alternatively, the toy can also be connected to an APP, and the user can input information such as voice, text or images through the APP, and the APP can be installed on a smart device, such as a smart phone or a smart watch.

[0027] Reference Figure 1 As shown, in order to achieve the above purpose, an embodiment of the present invention provides a toy interactive control method based on a large language model, comprising the following steps S100-S500: S100, obtaining user terminal information, where the user terminal information includes one or more of voice, text, image, body movement or external environment information; Specifically, in this step, voice, text, image, body movement, and external environment information can be obtained through a variety of technologies and devices.

[0028] For example, voice information can be obtained through a microphone to obtain the user's voice, and can be further converted into text through voice recognition software, such as the technology used by smart assistants such as Siri, Google Assistant, and Alexa. Text information can be input through an input device, such as a smartphone, or a keyboard. Images can be images stored by the user themselves, or images taken by a camera. Body movements can be tracked by high-speed cameras, or detected by using related equipment such as inertial measurement units, including but not limited to wearable devices (such as smart watches). External environmental information can be obtained through corresponding sensors according to the type of information required. For example, light sensors can obtain the brightness of the environment, temperature sensors can obtain the ambient temperature, sound sensors can obtain the sounds in the environment (such as music), and so on.

[0029] In this embodiment, according to the above description, one or more of voice, text, image, body movement or external environment information can be obtained, and each type of information corresponds to a modality.

[0030] S200, extracting features from user-side information and generating feature representation; In this step, feature extraction is performed on one or more of speech, text, image, body movement or external environment information to generate corresponding modal features.

[0031] Specifically, when the user-side information is one of voice, text, image, body movement or external environment information, feature extraction is performed to directly generate a single modal feature representation. For example, feature extraction of voice information can generate a single modal feature representation corresponding to the voice, and similarly, feature extraction of text information can generate a single modal feature representation corresponding to the text.

[0032] When the user-side information is any combination of two or more of speech, text, image, body movement or external environment information, features are extracted for each type of information to obtain multiple modal features, i.e., multimodal features, and the multimodal features are fused to generate a unified feature representation. Exemplarily, when the user-side information is speech and body movement information, feature extraction is used to generate feature representations corresponding to speech and feature representations corresponding to body movement information, i.e., two modal features, and then the two modal features are fused to finally obtain a unified feature representation. For another example, when the user-side information is a combination of images, body movements, and external environment information, feature extraction is used to generate feature representations corresponding to images, body movements, and external environment information, i.e., three modal features, and then the three modal features are fused to finally obtain a unified feature representation.

[0033] Through the above method, this step obtains a single modal feature representation, or obtains a unified feature representation.

[0034] S300, inputting the feature representation into a large language model to identify the user's intention; In this step, a large language model (LLM) is used. The large language model can be an existing model as the infrastructure, such as the GPT (Generative Pre-trained Transformer) series, BERT (Bidirectional Encoder Representations from Transformers), T5 (Text-to-Text Transfer Transformer), etc. These large language models can be pre-trained and can identify user intent through feature representation.

[0035] In some embodiments, pre-training can also be performed using collected training data (one or more of speech, text, images, body movements, or external environment information) to obtain a large language model that can recognize user intent, and the feature representation obtained in S200 above can be input into the trained large language model to recognize user intent.

[0036] In this step, the user's intention can be identified based on one or more corresponding feature representations of voice, text, image, body movement or external environment information. The user's intention can be many, and it is set according to the possible work or use of the toy in use. The intention expressed by the human user's language, text, voice and body movement (intensity, frequency) is understood through the large language model.

[0037] For example, during the use of the toy, the user's voice can be directly obtained, and the user's intention can be directly identified through the semantics corresponding to the voice. The user's intention can also be identified through the text input by the user, or it can be identified by a combination of voice and text. Of course, it can also be identified by a combination of voice and body movements. Exemplarily, for a toy doll, when the collected user voice is "stop", it is recognized that the user intends to stop the current working state or use of the toy. When the collected user voice is "speed up", it is recognized that the user intends to speed up the current working frequency of the toy. For example, when the collected user voice is "intensity increase", it is recognized that the user intends to increase the action intensity of the current toy, such as increasing the action of the toy, or increasing the vibration intensity. Of course, for toys similar to toy dolls, other controls can also be added, such as smiling, hugging, voice chatting, etc. to imitate the possible expressions, actions, voices and other interactions of real people.

[0038] It should be noted that the above are only some examples. In actual applications, the type of user intention can be set according to the actual situation of the toy, and the user intention can be recognized through one or more of voice, text, image, body movement or external environment information. The toy in this embodiment can be any toy that can realize interaction, can be a toy doll of various shapes, or can be other intelligent toys.

[0039] S400, generating corresponding control instructions according to the identified user intention; After S300 recognizes the user's intention, a control instruction is generated according to the user's intention. The control instruction is an instruction to control the movable or controllable part of the toy, so as to form personalized intelligent interaction needs according to different users and improve the experience. For example, the working mode control, working frequency control, movement amplitude increase or decrease control, expression control of the toy doll, feedback control of the voice chat of the doll, control of the joint movement of the doll, etc. simulate the possible movable or controllable parts in the process of human interactive behavior.

[0040] S500, converting the control instruction into a specific control signal and sending it to the control system of the toy to execute corresponding control.

[0041] The control instruction corresponds to a specific user intention. After S400 receives the control instruction, it converts it into a control signal suitable for the target toy, and sends the control signal to the control system or actuator of the toy through an appropriate physical interface (wired, wireless module, etc.) to ensure that the signal can effectively control or drive the corresponding hardware of the toy.

[0042] In the above embodiment of the present invention, through steps S100-S500, according to the feature representation corresponding to the acquired user-side information, a large language model is used to identify the user's intention, and a corresponding control instruction is generated according to the identified user's intention to control the corresponding action or work of the toy. The embodiment of the present invention innovatively applies a large language model to the control of the toy, that is, through the large language model, human language, voice, emotion, body movement, etc. are perceived, the user's intention is accurately identified, and then the toy is controlled to give corresponding feedback according to the user's intention, forming a real-time intelligent interaction between the toy and the user during use, thereby improving the user's experience.

[0043] In practical applications, although the use of single-modal information can meet some interactive needs, it may also be instable due to noise. In order to better identify user intentions, in S200, the user-side information can be two or more of voice, text, image, body movement or external environment information. At this time, it is necessary to extract features from each type of information separately to obtain multimodal features, and then fuse the multimodal features to generate a unified feature representation.

[0044] Reference Figure 2 As shown, in a possible implementation of a toy interactive control method based on a large language model of the present invention, step S200 can be performed according to the following steps: S201, performing feature vectorization processing on each type of information to obtain each modal feature; In this step, the speech, text, image, body movement, and external environment information are feature vectorized to generate feature vectors that can be used in the large language model. For example, each modal information is organized into a vector form of fixed length and provided as input to the large language model.

[0045] When S200 uses information of a single modality, the information of each modality obtained in S201 can be used as a feature representation of the corresponding modality.

[0046] S202, using the attention mechanism to align the features of each modality to enhance the correlation between the features; The attention mechanism allows different weights to be automatically assigned to different parts of the features when processing them, so as to better focus on and utilize important information. In this step, modal feature alignment uses the attention mechanism to learn how to focus on the correlation and importance between different modal features, effectively capturing the relationship between different modalities, thereby improving overall performance and robustness. The attention mechanism can use a neural network to obtain a series of attention vectors, which represent the key features in the input features and are used to generate alignment results to ensure that the modal features from different data sources are semantically and spatially consistent.

[0047] In this step, semantic alignment is mainly used to identify user intent, which can be achieved by mapping different modal features into the same semantic space so that they can understand and interact with each other in this space. For example, in the multimodal alignment of speech, images, and text, the neural network needs to learn how to understand the objects, scenes, and actions in the image and match them with the semantics in the speech and text description.

[0048] In actual processing, the attention mechanism can be implemented using a Transformer model, which can map features of different modalities into a shared semantic space, so that features of different modalities can be compared and interacted in the same space. This interaction can be achieved through a cross-modal attention layer. Of course, in other embodiments, networks with other attention mechanisms can also be used, not limited to the above.

[0049] S203, fusion of the aligned features into a unified feature representation through a feature fusion network.

[0050] In some embodiments, in S203, in order to fuse the aligned features into a unified feature representation, the following may also be included: cascade splicing fusion, inter-modality relationship modeling, and deep feature extraction, as follows: Cascade concatenation and fusion: The standardized features of all modalities are directly concatenated into a high-dimensional feature vector to form a preliminary fused feature representation, where the high-dimensional feature vector contains complete information of all modalities; Modeling the relationship between modalities (attention mechanism): On the concatenated high-dimensional feature vector, the attention mechanism is used to dynamically calculate the weight of each modality, and then the concatenated features are weighted according to the modality weight to obtain the weighted fusion feature representation; Deep feature extraction: The weighted fused features are input into the deep learning network to further extract the nonlinear relationship between the modalities and generate the final unified feature representation.

[0051] In this embodiment, cascaded splicing fusion, inter-modal relationship modeling and deep feature extraction are used to fuse the aligned features into a unified feature representation. By introducing a combined strategy of cascaded splicing fusion and deep feature extraction, the information of different modalities is retained to the maximum extent, and the complex interactive relationship between modalities can be effectively captured.

[0052] In the above embodiments of the present invention, the fusion of multimodal features can improve the stability and generalization performance under different conditions, build a richer and more comprehensive semantic representation, and help understand and process complex tasks, thereby further improving the intelligence of the interaction between the toy and the user.

[0053] In a possible embodiment, when the type of user-side information is one of voice, text, image, body movement, and external environment information, the corresponding large language model in step S300 may include any one of a language understanding unit, a sound emotion recognition unit, a facial expression recognition unit, an action recognition unit, and a context understanding unit, and each single modal feature is obtained through these units.

[0054] Reference Figure 3 As shown, in order to better realize user intention recognition, in another possible embodiment, when based on the type of user-side information, i.e., multiple (two or more) of voice, text, image, body movement, and external environment information, the large language model correspondingly includes two or more of a language understanding unit, a sound emotion recognition unit, a facial expression recognition unit, an action recognition unit, and a context understanding unit, and also includes a multimodal semantic fusion unit, wherein the language understanding unit, the sound emotion recognition unit, the facial expression recognition unit, the action recognition unit, and the context understanding unit correspondingly obtain each single modal feature, and the multimodal semantic fusion unit semantically fuses multiple modal features to obtain a fusion of multimodal features.

[0055] Specifically, each functional unit implements the following functions: A language understanding unit, used to parse and understand the semantic features of text in user-side information; Voice emotion recognition unit, used to analyze the emotional characteristics of the voice in the user's information; A facial expression recognition unit, used to analyze the facial expression semantic features of the image in the user-side information; Action recognition unit, used to understand the semantics of body movements in user-side information; The context understanding unit is used to combine historical interaction information and current conversation content to build a semantic understanding framework; The multimodal semantic fusion unit, when including multiple functional units, is used to fuse the semantic analysis results of the multiple functional units to generate a unified semantic understanding.

[0056] In the above embodiments, for the language understanding unit, a variety of natural language processing (NLP) technologies can be used, such as a pre-trained deep learning model, etc., to parse and understand the semantic features of the text in the user-side information based on sentence structure, word meaning analysis, etc.

[0057] In the above embodiment, for the sound emotion recognition unit, sound-related features and speech-related features can be extracted. For example, the sound-related features can be fundamental frequency, energy, etc., wherein the fundamental frequency can reflect the frequency of the fundamental tone in the speech signal, which is related to the pitch of the speaker. Energy can reflect the intensity of the speech signal, which is related to the volume and emotional state of the speaker. Speech-related features can be speech rate, intonation, spectral features, etc., wherein the speech rate can reflect the speed of speaking, which is related to the emotional state; the intonation can reflect the pitch change of the speech signal, which is closely related to the emotional state; the spectral features can reflect the distribution characteristics of the speech signal on the spectrum, which is related to the pronunciation habits and emotional state. By extracting these features, further using machine learning or deep learning technology to train an emotion recognition model (neural network), the emotional characteristics of the speech in the user-side information can be analyzed.

[0058] In the above embodiments, for the facial expression recognition unit, the face part in the image is generally detected first, and then the key feature information of the face, such as the shape, size and position of the eyes, mouth, eyebrows and other parts, is extracted. Feature extraction methods include methods based on geometric features, methods based on deep learning (such as CNN), etc. Classification and recognition are performed based on the detected key feature information to achieve expression classification. Commonly used classifiers include SVM (support vector machine), decision tree, neural network and deep learning model. Of course, before classification, the model can be trained and tested, and the classifier can be trained using an image data set with known expression labels, and the accuracy and robustness of the classifier can be verified through a test set. It is also possible to use a pre-trained model, directly input the image to be tested, and obtain the expression classification result, thereby completing the analysis of the facial expression semantic features of the image in the user-side information.

[0059] In the above embodiment, the action recognition unit, similar to the above units, can obtain user images and implement them through pre-trained action recognition models, or use key point detection and other methods to implement action recognition and determine the specific state of the user's action. Action recognition can also be achieved through data or information obtained by sensing devices. For example, for users who use toys, wearable devices (bracelets, etc.) or other sensing devices (three-axis sensors, six-axis sensors, etc.) can be used to detect the speed, direction, and force of the user's limb movements. Based on the results of these action recognitions, the semantics of the limb movements in the user-side information can be understood.

[0060] In the above embodiment, for the context understanding unit, the historical interaction information can be interaction information such as voice, text, and image, and the current conversation content can also be conversation content such as voice, text, and image. According to the historical interaction information and the current conversation content, the semantic elements are determined, and the semantic relationship between words (such as synonyms, antonyms, and hyponyms) and the logical relationship between sentences (such as cause and effect, transition relationship, etc.) are analyzed, and the context information is introduced to construct a semantic understanding framework, so that the current conversation content can be understood more accurately and the semantics of context understanding can be obtained. Furthermore, according to the accumulation of historical interaction information, the semantic understanding framework can be continuously optimized and adjusted, so that the habits and hobbies of different users can be understood, and a better understanding framework can be provided for subsequent user intention recognition.

[0061] It can be seen that the context understanding unit can integrate historical interaction information and current conversation content, and generate a context semantic feature that can reflect the user's current intention and historical behavior by analyzing semantic relations and logical structures. These semantic features have stronger context relevance and dynamic adaptability.

[0062] Based on the above two or more units, a multimodal semantic fusion unit is used to fuse their semantic analysis results to generate a unified semantic understanding, that is, a multimodal semantic fusion unit is used to integrate the semantic analysis results from different sources or modes (such as speech, text, images, body movements, external environmental information, etc.) to obtain a richer and more comprehensive unified semantic understanding. For example, the multimodal semantic fusion unit can use a deep learning model to achieve feature-level fusion through shared layers or joint training, or each modality can make predictions independently, and then merge these prediction results in the decision stage. It is also possible to construct a multimodal neural network architecture by adding additional layers to be responsible for cross-modal feature fusion, and finally generate a unified semantic understanding.

[0063] On the basis of the above embodiments, in order to better respond to the user's intention, the user's intention also includes a quantitative feature of the emotional state, and the state and / or parameters of the toy during use can be accurately controlled or adjusted through the change of the quantitative feature of the emotional state. The quantitative feature of the emotional state here refers to a quantitative expression of the user's emotional state, which can truly reflect the emotional changes of the user during the use of the toy. The quantitative feature of the emotional state can be used to realize the quantitative classification of the emotional state by adding a classifier during the training of the neural network model. For example, after identifying that the user's emotion is like, the emotional state of "like" can be quantified, and a feature vector can be used to represent it, such as from 0-10, 0 means no emotion is recognized, 1-2 can represent a mild state, 3-5 can represent a normal state, 5-7 can represent a strong state, and 8-10 can represent an extreme state. For example, when the emotion of liking is detected, if the quantitative characteristic of the emotional state is 1-2, it can be considered as a slight liking, and the vibration intensity, frequency and other parameters of the toy can be controlled in a lower range. If the quantitative characteristic of the emotional state is 5-7, it means a strong liking, then the vibration intensity, frequency, movement amplitude and other parameters of the toy can be increased. Similarly, different modes, parameters, etc. of different toys can be controlled or adjusted according to different settings of the quantitative characteristics of the emotional state. For example, for a toy doll, in the above-mentioned 0-10 emotional state quantitative characteristics, different feedback can be made according to the different quantitative characteristics of the "like" emotional state. For example: when the emotional state quantitative characteristic is 0, the doll may be expressionless and have no movements; when it is 1-2, it is a slight like, and the expression can be controlled to be a smile; when it is 3-5, it is a normal state of "like", and the expression can be controlled to be an obvious smile (with a larger arc of the corners of the mouth), and it can be accompanied by a slight movement amplitude (nodding and / or coordinating with body movements); when it is 5-7, it is a strong state of "like", and the expression can be controlled to be a laugh, and it can be accompanied by a larger movement amplitude and body movements; correspondingly, when it is 8-10, while laughing, more expressions can be further added, and it can be accompanied by a larger movement amplitude and more body movements. Through this quantitative classification control method, accurate interaction with real people in terms of expressions and movements can be formed.

[0064] It should be noted that all controls or parameter adjustments of the toy are performed within the safe range of toy use. The above is only an example. In other embodiments, other classification methods and expression methods can also be set, depending on the usage and occasion of the toy.

[0065] In some preferred embodiments, when the user intention identified in S300 includes the above-mentioned emotional state quantitative characteristics, the corresponding process of generating a control instruction in S400 includes: S401, based on the recognition of the user's intention, extracting the behavior target of the user's intention; S402, analyzing the quantitative characteristics of the emotional state to generate emotional parameters for adjusting device operation; S403, generating a control instruction based on the user's intended behavior goal, emotional parameters and external environment information.

[0066] The above-mentioned embodiments of the present invention can simultaneously generate corresponding control instructions and parameters according to the identified user intentions, quantitative characteristics of emotional states and external environmental information, thereby optimizing the user experience and greatly improving the intelligence and accuracy of the interaction.

[0067] In order to better understand the implementation of the above embodiment, an exemplary description corresponding to the above step S200 from feature extraction to fusion of multimodal features into a unified feature representation is provided. For operations not described in detail in the following steps, refer to the above introduction or prior art. Specifically, this example includes the following steps: Step 1: Feature Extraction For the input user-side multimodal information, such as two or more of voice, image, text, body movement or external environment information, first extract each single modal feature to obtain the corresponding feature vector, as follows: (1) Speech modality features: Use the speech sentiment analysis module to extract the emotional features of speech, such as fundamental frequency, pitch, speaking speed, etc.

[0068] Example feature vector: F_voice = [0.5, 0.8, 0.3].

[0069] (2) Image modality features: Use the image processing module to extract visual features of user expressions or actions, such as facial key points, action postures, etc.

[0070] Example feature vector: F_image = [0.2, 0.6, 0.4].

[0071] (3) Text modality features: Use natural language processing modules (such as the pre-trained language model BERT) to extract semantic features.

[0072] Example feature vector: F_text = [0.9, 0.7, 0.1].

[0073] (4) Body movement modal characteristics: Use a motion capture module (such as an inertial measurement unit or a camera) to extract the user's motion trajectory and frequency characteristics.

[0074] Example feature vector: F_motion = [0.4, 0.5, 0.7].

[0075] (5) External environment modal characteristics: Use sensors to collect environmental data (such as brightness, temperature, sound, etc.) and extract features.

[0076] Example feature vector: F_env = [0.3, 0.2, 0.6].

[0077] Of course, in other embodiments, the external environment modal features may not be used.

[0078] Step 2: Feature Normalization The eigenvectors of each modality are normalized and all eigenvalues ​​are normalized to the same range (such as [0, 1]) to balance the impact of different modalities on subsequent fusion.

[0079] For example, for F_voice = [0.5, 0.8, 0.3], after normalization it may become: F_voice_normalized = [0.625, 1.0, 0.375] Step 3: Cascade splicing and fusion The standardized features of all modalities are directly concatenated into a high-dimensional feature vector to form a preliminary fused feature representation. The concatenated feature vector contains complete information of all modalities, as shown below: Assumptions: The voice modality feature is F_voice_normalized = [0.625, 1.0, 0.375]; The image modality feature is F_image_normalized = [0.2, 0.6, 0.4]; The text modality feature is F_text_normalized = [0.9, 0.7, 0.1]; The limb motion modality feature is F_motion_normalized = [0.4, 0.5, 0.7].

[0080] After cascade splicing, we get: F_concat = [0.625, 1.0, 0.375, 0.2, 0.6, 0.4, 0.9, 0.7, 0.1, 0.4,0.5, 0.7] Step 4: Modeling inter-modal relationships (attention mechanism) On the concatenated high-dimensional feature vector, the attention mechanism is used to dynamically calculate the weight of each modality to highlight the most important modality features for the current scene. Preferably, the specific implementation is as follows: (1) Calculate attention weight: Use self-attention mechanism or multi-head attention mechanism to dynamically adjust modality weights based on contextual information (such as the user’s current emotional state).

[0081] Example weights: Voice modality weight: α_voice = 0.4; Image modality weight: α_image = 0.3; Text modality weight: α_text = 0.2; Limb motion modality weight: α_motion = 0.1.

[0082] (2) Weighted feature fusion: The concatenated features are weighted according to the modal weights to obtain the weighted fused feature representation:

[0083] Step 5: Deep Feature Extraction The weighted fused features are input into a deep learning network (such as a multi-layer perceptron (MLP) or Transformer) to further extract the nonlinear relationship between the modalities and generate the final unified feature representation.

[0084] (1) Deep learning network structure: Input layer: accepts the fused feature vector F_weighted = [0.53, 0.77, 0.36].

[0085] Hidden layer: Extract the complex interactions between modalities through multi-layer perceptron (MLP) or Transformer.

[0086] Activation function: Use ReLU or GELU activation to enhance the nonlinear expression ability of features.

[0087] Output layer: Generates the final unified feature representation.

[0088] (2) Final feature representation: Example output vector: F_final = [0.6, 0.8, 0.5, 0.4, 0.7] This vector contains comprehensive semantic features of multimodal information and can be directly input into a large language model (such as GPT) for user intent recognition.

[0089] It should be understood that the above description of the features and their vectors is only an example for the convenience of understanding by technical personnel. In other embodiments, other vector expressions may be used and are not limited to the situations listed in the above examples.

[0090] Furthermore, as mentioned in the above embodiments, the emotional state quantitative characteristics are indicators that quantitatively express the user's emotions, and can truly reflect the changes in the user's emotions during the use of the toy. Through the emotional state quantitative characteristics, it is possible to achieve precise control or dynamic adjustment of the toy's operating parameters (such as vibration intensity, frequency, movement mode, movement amplitude, etc.). Based on this, in order to facilitate understanding of the specific implementation of the emotional state quantitative characteristics in the control process, an exemplary description is provided below. For operations that are not described in detail below, refer to the above description. Specifically, this example includes the following contents: 1. Scope and definition of quantitative characteristics of emotional states In this example, the numerical range of the emotional state quantitative characteristics is 0-10, and its specific classification and meaning are as follows: 0: no emotion detected (noise or invalid data); 1-2: Mild state (such as slight liking or low excitement); 3-5: normal state (such as moderate liking or moderate excitement); 6-7: Intense states (such as intense liking or high excitement); 8-10: Extreme state (such as extreme liking or great excitement).

[0091] 2. Example of setting the quantitative features of emotional state in application In the control of toys, the quantitative characteristics of emotional states can be directly used to adjust the operating parameters of the toys. For example, for the emotional characteristic of "like", the corresponding vibration intensity and frequency can be set according to the numerical value of the quantitative characteristics: i1. Mild condition (1-2): Vibration intensity: Set to the lowest range (eg 20%-40% power).

[0092] Vibration frequency: Keep it in a low range (e.g. 1-3 Hz).

[0093] i2. Normal state (3-5): Vibration intensity: Set to a medium range (e.g. 50%-70% power).

[0094] Vibration frequency: Keep it in a medium range (e.g. 4-6 Hz).

[0095] i3. Intense state (6-7): Vibration Intensity: Set to a higher range (eg 80%-90% power).

[0096] Vibration frequency: Increase to a higher range (e.g. 7-9 Hz).

[0097] i4. Extreme state (8-10): Vibration intensity: Set to the maximum safe range (eg 95%-100% power).

[0098] Vibration frequency: Reach the highest frequency allowed by the equipment (e.g. 10-12 Hz).

[0099] It should be noted that the above parameter ranges are all within the safe design range of the toy and can be preset or adjusted according to user preferences. In other embodiments, there may be more or fewer state settings for emotional features and corresponding quantitative features. Generally speaking, the same emotional state setting contains at least two or more quantitative classifications. The more state settings there are, the higher the accuracy requirement for user intention recognition, and accordingly, the finer the control of the toy. The same emotional state of the user is classified into multiple levels through the quantitative features of the emotional state, so as to better meet the user's intelligent interaction needs with the toy during use, and the emotional interaction with the user is more accurate.

[0100] 3. Generation of quantitative features of emotional states In this example, the quantitative features of emotional state are generated through multimodal emotion recognition technology. The specific steps are as follows: (1) Data collection Collect emotion-related data through multimodal information, including but not limited to voice data, image data, and external environment data, where: Voice data: collect user voice through microphone and extract voice features, such as pitch, volume, speaking speed, etc. Image data: Collect the user's facial expressions or body movements through the camera and extract visual features, such as facial key points, movement trajectories, etc. External environmental data: Environmental information is collected through sensors, such as background music, light intensity, etc., to assist in emotion recognition.

[0101] In other examples, one or more of the above data may be collected, such as omitting external environment data.

[0102] (2) Feature extraction Perform feature extraction on the collected data to generate multimodal emotion features: i1) Voice emotion features, including: Extract the fundamental frequency, volume, speaking speed, intonation and other features of the audio signal.

[0103] Generate sentiment vectors using a pre-trained sentiment recognition model.

[0104] i2)Facial expression features: Detect facial key points (such as eyebrows, mouth corners, and eye positions) and extract expression features.

[0105] Predict emotion categories using facial expression recognition models.

[0106] i3) Body movement characteristics: Analyze the user's movement trajectory, frequency and strength, such as through an inertial measurement unit (IMU).

[0107] Identifying emotion-related motion features using a motion pattern classifier.

[0108] (3) Modal Fusion The multimodal features are fused into a unified emotion feature vector, which includes the following operations: Concatenation and fusion: directly concatenate the features of each modality into a high-dimensional feature vector.

[0109] Weighted sum fusion: Dynamically adjust modal weights through the attention mechanism to highlight key modalities.

[0110] Deep fusion: Use a fusion network (such as Transformer or Multi-layer Perceptron MLP) to extract the nonlinear relationship between modalities and generate a unified sentiment vector.

[0111] 4. Quantification of emotional state Add an emotion classifier to the neural network model to quantify the emotional state. The specific operations are as follows: (1) Classifier design: Use a fully connected layer with a Softmax activation function to classify emotion categories.

[0112] Example output categories: like, dislike, excited, calm, etc.

[0113] (2) Quantitative feature generation: Based on the predicted scores of the emotion categories, quantitative feature values ​​(ranging from 0 to 10) are generated by combining contextual information.

[0114] Example: The sentiment category is "like", the prediction score is 0.85, and the quantitative value generated according to the mapping rule is 7.5.

[0115] Of course, the mapping rule can be adjusted based on experimental data, and the above is just an example.

[0116] 5. Real-time updates and feedback During the process of users using toys, the operating status of the toys can be dynamically adjusted through real-time emotion recognition and quantification. For example: The system updates the quantitative characteristics of the emotional state every set time (such as 5 seconds), and then adjusts the parameters (such as movement speed, vibration intensity, and frequency) according to the latest quantitative characteristic values.

[0117] The above example process can clearly understand the implementation technology of emotional state quantification in the embodiment of the present invention. Of course, in other embodiments, other ways of setting emotional state quantification can also be adopted.

[0118] The above preferred embodiment further refines the user's intention and the degree of his / her emotional state through the quantitative characteristics of the emotional state, so that in the interaction between the toy and the user, the state and / or parameters of the toy during use can be accurately controlled or adjusted based on the quantitative characteristics of the emotional state, providing the user with emotional feedback with a better experience, including voice interaction feedback, adjustment of the toy's working mode and working parameters, human-like working feedback of the toy doll, etc. This is smarter and more accurate than interacting based solely on an emotion recognition, and is more in line with user needs.

[0119] In the above embodiment of the present invention, after the above user intention (including the quantitative characteristics of the emotional state) is accurately identified using a large language model, a corresponding control instruction is generated according to the identified user intention. In some embodiments, the control instruction includes one or more of the following action control instructions, voice control instructions, and atmosphere control instructions.

[0120] Specifically, the motion control instructions can be used to adjust one or more of the toy's motion mode, intensity or frequency, and the voice control instructions can be used to select a preset voice feedback template and adjust voice parameters (such as voice tone or content) according to the quantitative characteristics of the emotional state. The parameter range of the motion mode is a preset set of motion patterns, which can be determined according to the type of toy and its own structure. The intensity or frequency can be the intensity or frequency of the toy's vibration, movement, etc., or the intensity or frequency of the toy doll's motion feedback / voice response. For example, the feedback expression can be "like" or "strong like". The toy doll has different motion and voice interactions with the user under different emotional characteristics. Whether it is from the voice tone or the action (such as imitating the movements of human joints, etc.), the whole process will simulate the reaction of real people to set up, so as to achieve a scene that fits real people's interaction as much as possible.

[0121] In addition, by obtaining external environment information, atmosphere control instructions can be added to the user end to optimize the environment when the user uses the toy. The atmosphere control instructions can be used to adjust one or more of the light brightness, color, flashing frequency or ambient music to match the user's mood and environmental needs.

[0122] The above-mentioned embodiments of the present invention can accurately realize intelligent interaction by identifying the user's intention and emotions and then generating corresponding control instructions. At the same time, in order to provide more choices and better user experience, in some embodiments, the toy interactive control method of the present invention also includes scheduling control instructions. Specifically, the priority of each control instruction is determined based on the preset weight of the user's intention, and then the execution order of multiple control instructions is coordinated to ensure coordination and non-conflict between the control instructions. In addition, the user's real-time feedback data, including behavioral responses, etc., can also be collected, combined with environmental information and emotional state quantitative characteristics, and the control strategy can be further dynamically adjusted to optimize the user experience.

[0123] In the above embodiments, the user intention and the control instruction generation content are described. For ease of understanding, in some embodiments, the control instruction is generated according to the identified user intention and the quantitative characteristics of the emotional state. The generation of the specific control instruction may include the following steps: M1. Determine the target control type, such as motion, voice or atmosphere control; M2. Match the preset control template according to the specific content of the user's intention (such as target behavior, emotional state, etc.) and / or external environment information (such as light brightness, background music, temperature, etc.); M3. Dynamically adjust control parameters, such as exercise intensity, frequency, voice volume and tone, light brightness and color, etc., to ensure that control instructions are consistent with the user's emotional state and / or external environment state; M4. Generate the final comprehensive control instructions and further optimize the instruction parameters through real-time feedback mechanisms (such as user behavior response).

[0124] It should be noted that, in some embodiments, the generation of the control instruction may also include: (1) Based on the quantitative characteristics of emotional state, determine the user's emotional level (such as relaxed, excited, nervous) and quantify it into a corresponding numerical range; (2) Analyze external environmental information (such as light brightness, background music, temperature, etc.) and match the preset optimization strategy based on the quantitative characteristics of emotional state; (3) Dynamically adjust the parameters of the control instructions (such as vibration intensity range, light color and brightness, volume, etc.) to ensure that the control instructions are consistent with the user's emotions and environmental conditions; (4) Generate the final comprehensive control instructions and further optimize the instruction parameters through real-time feedback mechanisms (such as user behavior response).

[0125] On the basis of the above-mentioned embodiments of the present invention, in order to continuously improve the control of toys in use, in some embodiments, the toy interactive control method may also include a continuous learning module to optimize user intention recognition and control strategies. Specifically, the following learning optimization methods may be included: collecting user interaction data and feedback information, and then using deep learning algorithms and reinforcement learning algorithms to dynamically update the parameters of the large language model, and finally adaptively adjusting the control logic to meet the personalized needs of users and improve the intelligence level of the system.

[0126] In order to ensure the security of user data and protect user privacy, based on the above embodiments of the present invention, the toy interactive control method may further include a security module, through which multi-factor authentication and authorization management of user identity can be performed to ensure that only authorized users can control the toy, and end-to-end encryption of user data can be performed to prevent unauthorized access and data leakage. In addition, the execution process of the control instructions can be monitored in real time, and abnormal instructions can be detected and blocked to ensure safety in use.

[0127] Based on the above embodiments of the present invention, the toy interactive control method may further include a remote update module, which downloads and installs the latest large language model parameters or control logic through a secure network connection or a cloud platform. The update process adopts a digital signature and verification mechanism to ensure the authenticity, integrity and security of the update content.

[0128] Based on the same technical concept as the above-mentioned toy interactive control method based on a large language model, in another embodiment of the present invention, a toy interactive control device based on a large language model is also provided, comprising: Information acquisition module: acquires user-side information, which includes one or more of voice, text, image, body movement or external environment information; Feature extraction module: extracts features from user-side information and generates feature representation; Large language model module: inputs feature representation into the large language model to identify user intent; Instruction generation module: generates corresponding control instructions according to the identified user intention; Control signal generation module: converts control instructions into specific control signals and sends them to the toy's control system to perform corresponding control.

[0129] In some embodiments, the information acquisition module acquires user-side information, which includes one or more of voice, text, image, body movement or external environment information. The information acquisition module can acquire the above-mentioned various information through one or more interfaces, and can also pre-process some of the acquired information (such as preliminary data formatting, noise reduction, etc.) before passing it to the feature extraction module. In addition, voice, text, image, body movement, and external environment information can be acquired through corresponding sensors or devices (such as microphones, cameras, tactile feedback devices, etc.), and can be obtained by wired or wireless means.

[0130] In some embodiments, the feature extraction module extracts features from user-side information and generates feature representations, wherein when the user-side information is one of speech, text, image, body movement, or external environment information, a single-modal feature representation is directly generated; when the user-side information is any combination of two or more of speech, text, image, body movement, or external environment information, features are extracted from each type of information, and multi-modal features are fused to generate a unified feature representation. Specifically, the feature extraction module extracts feature vectors (such as speech emotion features, text semantic features, facial expression features, etc.) from the collected single-modal or multi-modal data, and generates a unified feature representation through a feature fusion algorithm (such as an attention mechanism or a feature alignment network).

[0131] Specifically, in some embodiments, the feature extraction module extracts features from each type of information separately, and fuses multimodal features to generate a unified feature representation, including: performing feature vectorization processing on each type of information to obtain features of each modality; using an attention mechanism to align features of each modality to enhance the correlation between features; and fusing the aligned features into a unified feature representation through a feature fusion network.

[0132] In some preferred embodiments, in order to fuse the aligned features into a unified feature representation, the feature extraction module may further include: a cascade splicing fusion unit, an inter-modality relationship modeling unit, and a deep feature extraction unit, wherein: Cascade concatenation and fusion unit: directly concatenates the standardized features of all modalities into a high-dimensional feature vector to form a preliminary fused feature representation, where the high-dimensional feature vector contains complete information of all modalities; Inter-modal relationship modeling unit (attention mechanism): On the concatenated high-dimensional feature vector, the attention mechanism is used to dynamically calculate the weight of each modality, and then the concatenated features are weighted according to the modality weight to obtain the weighted fusion feature representation; Deep feature extraction unit: The weighted fused features are input into the deep learning network to further extract the nonlinear relationship between the modalities and generate the final unified feature representation.

[0133] In this embodiment, cascaded splicing fusion, inter-modal relationship modeling and deep feature extraction are used to fuse the aligned features into a unified feature representation. By introducing a combined strategy of cascaded splicing fusion and deep feature extraction, the information of different modalities is retained to the maximum extent, and the complex interactive relationship between modalities can be effectively captured.

[0134] The biggest innovation of the present invention is to apply a large language model to toy interactive control. In some embodiments, the large language model includes one or more of the following functional units: Language understanding unit, used to parse and understand the text semantic features in user-side information; Voice emotion recognition unit, used to analyze the emotional characteristics of the voice in the user's information; A facial expression recognition unit, used to analyze the facial expression semantic features of the image in the user-side information; Action recognition unit, used to understand the semantics of body movements in user-side information; The context understanding unit is used to combine historical interaction information and current conversation content to build a semantic understanding framework; When including multiple functional units, the large language model also includes: The multimodal semantic fusion unit is used to fuse the semantic analysis results of the various functional units mentioned above to generate a unified semantic understanding.

[0135] Therefore, the embodiments of the present invention build a semantic understanding framework of user intention based on a large language model (such as GPT or similar architecture), combined with contextual semantics, emotional state and user historical data, to accurately identify the user's behavioral goals and emotional needs.

[0136] In some embodiments of the present invention, the instruction generation module generates corresponding control instructions according to the identified user intention, wherein the control instructions include one or more of the following: Motion control instructions for adjusting one or more of the toy's motion mode, intensity or frequency, where the parameter range of the motion mode is a set of preset motion modes; Voice control instructions, used to select a preset voice feedback template and adjust voice parameters based on the quantitative characteristics of emotional state; Atmosphere control commands are used to adjust one or more of the lighting brightness, color, flashing frequency, or ambient music to match the user's mood.

[0137] The instruction generation module in the above embodiment can dynamically adjust the control parameters in combination with the user's intention through preset control logic (such as action template, voice template or atmosphere template, etc.) to generate control instructions that match the user's needs.

[0138] In other embodiments, when the user intention identified by the large language model module also includes a quantitative feature of the emotional state, the instruction generation module is used to: Based on the recognition of user intention, the behavioral goal of the user intention is extracted; Analyze the quantitative characteristics of emotional states and generate emotional parameters for adjusting device operations; Combine the user's intended behavioral goals, emotional parameters and external environment information to generate control instructions.

[0139] By generating control instructions including the fusion of quantitative features of emotional states, the interaction of toys can be accurately controlled according to the emotional characteristics of the user, thereby optimizing the user experience.

[0140] In some embodiments of the present invention, the control signal generation module converts the generated control instructions into control signals that can be recognized by the control system of the corresponding toy, thereby realizing the control of the toy. The specific control can be any one or more actions, control or adjustment of working parameters during the use of the toy, which can be set accordingly according to different types of toys.

[0141] The control signal generating module in the above embodiment can transmit the control instruction to the device hardware module through a communication interface (such as Bluetooth, Wi-Fi or a dedicated protocol) to trigger feedback behaviors such as motion, voice or atmosphere control in real time.

[0142] In the above embodiment of the toy interactive control system based on the large language model, the specific implementation of each module of the toy interactive control system based on the large language model can refer to the technology in the steps of any of the above-mentioned toy interactive control methods based on the large language model, and will not be repeated here.

[0143] Corresponding to the same technical concept of the above-mentioned toy interactive control method based on a large language model, in another embodiment of the present invention, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, and the steps of the toy interactive control method based on a large language model are implemented when the processor executes the program.

[0144] Reference Figure 4 As shown, Figure 4 FIG. 1 is a block diagram of an electronic device according to an exemplary embodiment. Figure 4As shown, the electronic device may include: a processor 401, a memory 402. The electronic device may also include a multimedia component 403, an input / output interface 404, and one or more of a communication component 405 and a power supply component 406. The processor 401 obtains user-side information and extracts feature representations, identifies user intent based on a large language model, generates control instructions according to the user intent and converts them into control signals, and sends the control signals to the control system of the toy through the communication component 405 to execute corresponding actions.

[0145] Among them, the processor 401 is used to control the overall operation of the electronic device to complete all or part of the steps in the toy interactive control method based on the large language model of the first aspect. The memory 402 is used to store various types of data to support the operation of the electronic device, which may include instructions for any application or method used to operate on the electronic device, as well as application-related data, such as user-side information data, messages sent and received, pictures, audio, video, etc. The multimedia component 403 may include a camera and an audio component 410, etc., and may also include a screen as required. The screen may be a touch screen, for example, and the camera may be used to obtain an image. The audio component 410 is used to output and / or input an audio signal. For example, the audio component 410 may include a microphone for receiving an external audio signal. The received audio signal may be further stored in the memory 402 or sent through the communication component 405. The audio component 410 also includes at least one speaker for outputting an audio signal. The input / output interface 404 provides an interface between the processor 401 and other interface modules. The above-mentioned other interface modules may be a keyboard, a mouse, buttons, etc., and these buttons may be virtual buttons or physical buttons. The communication component 405 is used for wired or wireless communication between the electronic device and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, 4G, NB-IOT, eMTC or other 5G, etc., or a combination of one or more of them, is not limited here. Therefore, the corresponding communication component 405 may include: Wi-Fi module, Bluetooth module, NFC module, etc. The electronic device in this embodiment can be installed on the toy or set independently. For example, it can be made into a very small control box and installed in the brain position of the toy doll, similar to the existence of the human brain. According to the acquired user-end information and user intention, control instructions are formed to control the use of the toy doll. It can also be made into a small component and set in the toy.

[0146] Optionally, the memory is used to store the program; the memory may include a volatile memory (English: volatile memory), such as a random access memory (English: random-access memory, abbreviated: RAM), such as a static random access memory (English: static random-access memory, abbreviated: SRAM), a double data rate synchronous dynamic random access memory (English: Double Data Rate Synchronous Dynamic Random Access Memory, abbreviated: DDR SDRAM), etc.; the memory may also include a non-volatile memory (English: non-volatile memory), such as a flash memory (English: flash memory). The memory is used to store computer programs (such as applications, functional modules, etc. that implement the above method), computer instructions, etc., and the above computer programs, computer instructions, etc. can be partitioned and stored in one or more memories. And the above computer programs, computer instructions, data, etc. can be called by the processor. The above computer programs, computer instructions, etc. can be partitioned and stored in one or more memories. And the above computer programs, computer instructions, data, etc. can be called by the processor.

[0147] The processor is used to execute the computer program stored in the memory to implement the various steps in the method involved in the above embodiment. For details, please refer to the relevant description in the above method embodiment.

[0148] The processor and the memory may be independent structures or integrated structures. When the processor and the memory are independent structures, the memory and the processor may be coupled and connected via a bus.

[0149] In addition, an embodiment of the present invention further provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer program is executed by a processor, the steps of the toy interactive control method based on a large language model are implemented.

[0150] Among them, computer-readable media include computer storage media and communication media, wherein the communication media include any media that facilitates the transmission of computer programs from one place to another. The storage medium can be any available medium that can be accessed by a general or special-purpose computer. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium can be located in an ASIC. In addition, the ASIC can be located in a user device. Of course, the processor and the storage medium can also be present in a communication device as discrete components.

[0151] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0152] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention. The above preferred features can be used in any combination without conflict.

Claims

1. A toy interactive control method based on a large language model, characterized in that: The following steps are involved: Acquire user terminal information, where the user terminal information includes one or more of voice, text, image, body movement or external environment information; Extracting features from the user-side information to generate feature representation; Inputting the feature representation into a large language model to identify user intent; Generate corresponding control instructions according to the identified user intention; The control instructions are converted into specific control signals and sent to the control system of the toy to execute corresponding control.

2. The control method according to claim 1, characterized in that: The feature extraction of the user terminal information to generate feature representation includes: When the user terminal information is one of voice, text, image, body movement or external environment information, directly generating a feature representation of a single modality; When the user-side information is any combination of two or more of voice, text, image, body movement, and external environment information, features are extracted for each type of information respectively, and multimodal features are fused to generate a unified feature representation.

3. The control method according to claim 2, characterized in that: The features of each type of information are extracted respectively, and the multimodal features are fused to generate a unified feature representation, including: Perform feature vectorization on each type of information to obtain the features of each modality; Use the attention mechanism to align the features of each modality to enhance the correlation between features; The aligned features are fused into a unified feature representation through the feature fusion network.

4. The control method according to claim 1, characterized in that: The large language model includes one or more of the following functional units: A language understanding unit, used to parse and understand the text semantic features in the user-side information; A voice emotion recognition unit, used to analyze the emotional characteristics of the voice in the user-side information; A facial expression recognition unit, used to analyze the facial expression semantic features of the image in the user terminal information; An action recognition unit, used to understand the semantics of the body movements in the user terminal information; The context understanding unit is used to combine historical interaction information and current conversation content to build a semantic understanding framework; When including multiple functional units, the large language model also includes: The multimodal semantic fusion unit is used to fuse the semantic analysis results of the various functional units mentioned above to generate a unified semantic understanding.

5. The control method according to claim 1, characterized in that: The user intention also includes a quantitative feature of emotional state; Correspondingly, the process of generating control instructions includes: Based on the recognition of user intention, the behavioral goal of the user intention is extracted; Analyzing the quantitative characteristics of the emotional state to generate emotional parameters for adjusting the toy's actions; A control instruction is generated by combining the behavioral goal of the user's intention, the emotional parameter and the external environment information.

6. The control method according to claim 5, characterized in that: The control instructions include one or more of the following: Motion control instructions for adjusting one or more of the toy's motion mode, intensity or frequency, where the parameter range of the motion mode is a set of preset motion modes; Voice control instructions for selecting a preset voice feedback template and adjusting voice parameters according to the quantitative characteristics of the emotional state; Atmosphere control commands are used to adjust one or more of the lighting brightness, color, flashing frequency, or ambient music to match the user's mood.

7. The control method according to claim 6, characterized in that: Also includes: Determining the priority of each of the control instructions based on a preset weight of the user intention; Coordinate the execution order of multiple control instructions to ensure coordination and non-conflict between the control instructions; Collect real-time feedback data from users, including behavioral responses, and dynamically adjust control strategies to optimize user experience.

8. The control method according to claim 1, characterized in that: It also includes a continuous learning module that optimizes user intent recognition and control strategies by: Collect user interaction data and feedback information; Use deep learning algorithms and reinforcement learning algorithms to dynamically update the parameters of large language models; Adaptively adjust control logic to meet the user's personalized needs.

9. The control method according to claim 1, characterized in that: Also included are security modules for: Perform multi-factor authentication and authorization management on user identities to ensure that only authorized users can control the toy; End-to-end encryption of user data to prevent unauthorized access and data leakage; The execution process of the control instructions is monitored in real time, and abnormal instructions are detected and blocked to ensure safe use.

10. The control method according to claim 1, characterized in that: Also includes: Remote update module, which downloads and installs the latest large language model parameters or control logic through a secure network connection or cloud platform. The update process uses a digital signature and verification mechanism to ensure the authenticity and integrity of the update content.

11. A toy interactive control device based on a large language model, characterized in that: include: Information acquisition module: acquires user-side information, wherein the user-side information includes one or more of voice, text, image, body movement or external environment information; Feature extraction module: extracts features from the user-side information and generates feature representation; Large language model module: inputs the feature representation into the large language model to identify the user's intention; Instruction generation module: generates corresponding control instructions according to the identified user intention; Control signal generation module: converts the control instruction into a specific control signal and sends it to the control system of the toy to execute the corresponding action.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that: When the processor executes the program, the steps of the toy interactive control method based on a large language model as described in any one of claims 1 to 10 are implemented.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the toy interactive control method based on a large language model described in any one of claims 1 to 10 are implemented.

Images