Toy interaction method combining voice, vision and RFID with multi-modal model

Abstract

According to the toy interaction method, a main toy unit and a plurality of auxiliary toy units are included, a controller is arranged in the main toy unit and connected with an RFID reader-writer, a microphone, a camera, an RFID tag and a loudspeaker, and each auxiliary toy unit is provided with an RFID tag; when the main toy unit is started, a user can connect and set the mode of the main toy unit through a mobile phone, the camera collects visual information, the microphone collects voice information and sends a signal to the multi-mode model, and when the auxiliary toy unit is close to the main toy unit, the RFID tag is activated and sends a signal to the multi-mode model; after analyzing the signal, the model generates model output and returns the model output to the main toy unit, and the main toy unit completes interactive response immediately; the main toy unit dynamically adjusts an interaction mode and output content according to a user setting mode, a connection state of each unit of the toy, RFID tag information, visual input and voice input. The embodiment of the invention can improve the interaction experience.

Description

Technical Field

[0001] This document relates to the field of artificial intelligence, specifically to a toy interaction method that combines voice, vision, and RFID with a multimodal model. Background Art

[0002] Currently, consumers' expectations for the playability, interactivity and educational value of toys are constantly increasing, prompting toy manufacturers to launch a variety of innovative toys to meet this growing demand.

[0003] However, despite the wide variety of toys on the market, many toys are still limited in providing interactive experiences and educational potential. These toys usually rely on preset programs and basic sensor technology, such as touch or sound sensors, to respond to user operations. Although this design can achieve basic interaction, its fixed response mode cannot adapt to the diversity of user behavior or changes in the environment, limiting the adaptability and educational potential of the toy. In addition, these toys often lack the ability to communicate deeply with users and cannot provide a richer and more personalized interactive experience, thus failing to fully realize their potential in promoting children's cognitive development and social skills. Summary of the invention

[0004] The embodiment of the present invention provides a toy interaction method that combines voice, vision, and RFID with a multimodal model. The toy interaction method provided by the embodiment of the present invention can improve the lack of deep communication capabilities and personalized interaction capabilities of current toys, and further enhance the educational and entertainment value of toys.

[0005] In a first aspect, an embodiment of the present invention provides a toy interaction method combining voice, vision, and RFID with a multimodal model, comprising:

[0006] The main toy unit and multiple auxiliary toy units, the main toy unit is equipped with a controller, connected to an RFID reader, a microphone, a camera, an RFID tag and a speaker, and each auxiliary toy unit is equipped with an RFID tag; when the main toy unit is started, the user can connect and set the main toy unit mode through a mobile phone, the camera collects visual information, the microphone collects voice information and sends a signal to the multimodal model, and when the auxiliary toy unit is close to the main toy unit, the RFID tag is activated and sends a signal to the multimodal model; after the model parses the signal, it generates a model output and returns it to the main toy unit, and the main toy unit immediately completes the interactive response; the main toy unit dynamically adjusts the interactive mode and output content according to the user setting mode, the connection status of each toy unit, the RFID tag information, the visual input, and the voice input.

[0007] In a second aspect, an embodiment of the present invention provides a method for integrating multimodal recognition and interactive response, including:

[0008] The controller of the main toy unit has the function of identifying the RFID tag information of the auxiliary toy unit to determine the identity of the auxiliary toy unit; the connection status recognition mechanism between the main toy unit and the auxiliary toy unit includes approach, contact and separation, and each connection status corresponds to a different interaction mode; the voice information collected by the microphone includes voice recognition and text transcription, and the recognized voice information is sent to the multimodal model, and the multimodal model generates a model output and returns it to the main toy unit; the multimodal model can parse the visual information collected by the camera, and combine the voice information and RFID tag information to generate a more accurate model output.

[0009] In a third aspect, an embodiment of the present invention provides a real-time personalized interaction method for a multimodal model toy, comprising:

[0010] The main toy unit can dynamically adjust the interaction mode and output content, including but not limited to voice, music, light and action, according to the RFID tag information and visual and voice input of the auxiliary toy unit; the controller of the main toy unit has a learning function, and can optimize the interactive response and output content in real time according to the user's interaction history and preferences; the connection between the main toy unit and the cloud server is wireless, and the wireless connection uses the Wi-Fi communication protocol;

[0011] The main toy unit has multi-language processing capabilities and can provide corresponding language interactions based on the user's geographic location and language preference; the voice information collected by the microphone also includes voiceprint recognition, which can identify the voice characteristics of a specific user to provide a personalized interactive response; in the method, the user can wirelessly connect to the main toy unit through a mobile phone and set the interaction mode of the main toy unit to achieve a personalized interactive experience. The information collected by the camera also includes face recognition, which can identify the facial features of a specific user to provide a personalized interactive response

[0012] In a fourth aspect, an embodiment of the present invention provides a multi-round dialogue and personalized interaction method, including:

[0013] The controller of the main toy unit has a multi-round dialogue function, which can maintain contextual consistency based on the user's input and historical interaction records to achieve smooth multi-round dialogue interaction; the main toy unit can record the user's interaction data and, with the user's consent, use the data for further training and optimization of the model.

[0014] The toy interaction method of the present embodiment combined with voice, vision, RFID and multimodal model: the method includes a main toy unit and multiple auxiliary toy units, wherein the main toy unit is provided with a controller, connected with an RFID reader, a microphone, a camera, an RFID tag and a speaker, and each auxiliary toy unit is equipped with an RFID tag; when the main toy unit is started, the user can connect and set the main toy unit mode through a mobile phone, the camera collects visual information, the microphone collects voice information and sends a signal to the multimodal model, and when the auxiliary toy unit is close to the main toy unit, the RFID tag is activated and sends a signal to the multimodal model; after the model parses the signal, the model output is generated and returned to the main toy unit, and the main toy unit immediately completes the interactive response; the main toy unit dynamically adjusts the interactive mode and output content according to the user setting mode, the connection status of each toy unit, the RFID tag information, the visual input, and the voice input. This embodiment can enhance the interactive experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0016] Figure 1 A flowchart of a toy interaction method combining voice, vision, and RFID with a multimodal model provided by an embodiment of the present invention;

[0017] Figure 2 for Figure 1 A flowchart of a specific implementation method of step 101 in the illustrated embodiment;

[0018] Figure 3 for Figure 1 A flowchart of a specific implementation method of step 102 in the illustrated embodiment;

[0019] Figure 4 for Figure 1 A flowchart of a specific implementation method of step 103 in the illustrated embodiment;

[0020] Figure 5 A schematic diagram of a process of a user setting a main toy unit mode provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the present invention are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0023] The embodiment of the present invention provides a toy interaction method combining voice, vision, RFID and multimodal model, the method includes a main toy unit and multiple auxiliary toy units, the main toy unit is provided with a controller, connected to an RFID reader, a microphone, a camera, an RFID tag and a speaker, and each auxiliary toy unit is equipped with an RFID tag; when the main toy unit is started, the user can connect and set the main toy unit mode through a mobile phone, the camera collects visual information, the microphone collects voice information and sends a signal to the multimodal model, when the auxiliary toy unit is close to the main toy unit, the RFID tag is activated and sends a signal to the multimodal model; after the model parses the signal, the model output is generated and returned to the main toy unit, and the main toy unit immediately completes the interactive response; the main toy unit dynamically adjusts the interactive mode and output content according to the user setting mode, the connection status of each toy unit, the RFID tag information, the visual input, and the voice input. This embodiment can enhance the interactive experience.

[0024] refer to Figure 1 , Figure 1 : is a flowchart of a toy interaction method combining voice, vision, RFID and a multimodal model provided by an embodiment of the present invention. The interaction method includes:

[0025] Step 101: When the main toy unit is started, the user can connect and set the main toy unit mode through the mobile phone, the camera collects visual information, the microphone collects voice information and sends a signal to the multimodal model, and when the auxiliary toy unit is close to the main toy unit, the RFID tag is activated and sends a signal to the multimodal model.

[0026] For example, reference Figure 5In the mobile phone interactive interface shown, the user can enter "child's age", "toy type", "knowledge type selection", "voiceprint" and "face" as the basic settings of the main toy unit. After that, the camera starts to collect visual information, such as when the user is building a scene, and the microphone starts to collect the user's voice information. For example, if the user says "I want to build a car", this visual information and voice information will be recorded and sent to the server via MQTT Broker; at the same time, when the user brings the main toy unit and the auxiliary toy unit close to each other, the RFID reader of the main toy unit reads the RFID tag information of the auxiliary toy unit and sends it to the server via MQTT Broker. The server sends the visual information, voice information, user setting information and RFID information to the multimodal model. Figure 2 As shown, the specific implementation of step 101 includes:

[0027] Step 1011: The user inputs user information, including age, toy type, knowledge type, voiceprint, and face. After input, the toy is started. The startup method is voice wake-up and button. After startup, the user begins to interact with the toy by voice, assembly, and vision.

[0028] Exemplarily, the user enters personal information through the mobile phone interactive interface: the specific content is the child's age, toy type, knowledge type, voiceprint, and face. For example, the user selects "8 years old" as the child's age, "car" as the toy type, "English" as the knowledge type in the application, and records voiceprints and facial samples. The user can start the toy by voice activation or physical button activation. When voice activated, the user starts the toy by saying a preset wake-up word, such as "Hey, toy buddy, I'm ready." When the physical button is activated, the user starts the interaction by pressing the physical button on the main toy unit. This action is recognized by the controller of the main toy unit and triggers the start sequence. After that, the camera starts to collect visual information, such as the shape and color of the auxiliary toy unit held by the user when interacting with the toy, and the microphone starts to collect the user's voice information. For example, if the user says "I want to build a car", this voice information will be recorded and converted into a digital signal. This visual signal and voice signal are transmitted to the server through the MQTT Broker; at the same time, the RFID reader of the main toy unit reads the RFID tag information attached to the auxiliary toy unit when the user brings the main toy unit and the auxiliary toy unit close to each other. After reading, this RFID information will also be transmitted to the server through the MQTT Broker.

[0029] Step 1012: The main toy unit obtains visual information through the camera, obtains voice information through the microphone, obtains RFID tag information through the RFID reader, and sends the information to the server through the MQTT Broker.

[0030] For example, when a user interacts with a car toy, he places an auxiliary toy unit representing a car engine on the car model as the main toy unit, and the RFID reader in the main toy unit immediately reads the RFID tag information attached to the auxiliary toy unit. The RFID tag contains characteristic information about the car engine. At the same time, the user can ask specific questions through the microphone, such as "How does the engine work?" to seek information or suggestions related to the car engine; the camera will capture the current assembly status of the auxiliary toy unit engine model at this time, and the main toy unit will send the visual information, voice information and RFID tag information to the server through the MQTT Broker.

[0031] Step 1013: The server sends the visual information, voice information, user information, and RFID tag information to the multimodal model.

[0032] Exemplarily, after receiving the information through the MQTT Broker, the server starts to process the user's current interaction request. The server will query its internal user information knowledge base, which stores the user's personal information, preference settings, and historical interaction records. If the user has shown interest in the working principle of the car in the previous interaction, the server will retrieve this preference information from the knowledge base. At the same time, the server will also combine the RFID tag information of the auxiliary toy unit, which is read by the RFID reader when the user brings the auxiliary toy unit close to the main toy unit. If the user has just brought an auxiliary toy unit representing a car engine close to the main toy unit, the server will recognize this action and use it as context information for the current interaction. Combining the user's voice request, visual information, personal preferences, and RFID tag information, the server constructs a comprehensive data packet. This data packet contains the user's request to inquire about the working principle of the engine, the current assembly status of the auxiliary toy unit engine model, the user's interest preference in the working principle of the car, and the RFID tag information of the car engine that the user is interacting with. Finally, the server sends this data packet to the multimodal model.

[0033] Step 102: After the model analyzes the signal, it generates a model output and returns it to the main toy unit, and the main toy unit then completes the interactive response.

[0034] For example, the multimodal model analyzes and generates corresponding text information and voice information based on the received data packet, which contains the user's request to inquire about the working principle of the engine, the current assembly status of the auxiliary toy unit engine model, the user's interest preference in the working principle of the car, and the RFID tag information of the car engine that the user is interacting with. For example, if the user asks "how does the engine work", the multimodal model will generate detailed text information and voice information about the working principle of the car engine, and may also include educational content such as the changes of the engine under different working conditions, mechanical characteristics and its role in the car.

[0035] The voice information generated by the multimodal model will be stored on the server, and the generated text information will then be converted into voice information by the server through the TTS service. This conversion process involves converting text data into a natural voice stream so that the user can hear it through the speaker of the main toy unit. The converted voice information is stored on the server, and after successful storage, a notification of successful storage will be sent to the main toy unit through the MQTT Broker. After receiving the notification, the main toy unit downloads the stored voice information through the network and plays it to the user using its built-in speaker. The user hears a detailed explanation of the working principle of the car engine, thereby completing an interactive response. Figure 3 As shown, the specific implementation of step 102 includes:

[0036] Step 1021: The multimodal model outputs corresponding text and voice information based on the received visual information, voice information, user information, and RFID tag information.

[0037] For example, during the interaction with the car toy, the user asks a question through the microphone of the main toy unit, such as "How does the engine work?" The question is captured by the microphone and converted into a digital signal. The camera of the main toy unit captures the assembly status of the current engine model of the secondary toy unit. These signals are transmitted to the server through the MQTT Broker. After receiving the current voice and visual information, the server combines it with the user's personal information, historical interaction records, and RFID tag information for in-depth analysis by the multimodal model. Based on these comprehensive data, the multimodal model generates detailed text and voice information about the working principle of the car engine. This process includes an in-depth understanding of user preferences, historical interaction records, and current interaction scenarios to ensure that the generated text information is both accurate and educational for users.

[0038] Step 1022: The multimodal model sends the output text and voice information to the server. The server directly stores the voice information generated by the multimodal model and simultaneously calls the TTS service to convert the text information into voice information and store it.

[0039] Exemplarily, once the multimodal model generates text information and voice information containing the working principle and mechanical characteristics based on the user's query "How does the engine work?", the server will perform the following operations: The server receives the text information and voice information output by the multimodal model and processes them in turn, and will use the built-in text-to-speech (TTS) service to convert the text information into natural and fluent voice information. This conversion process ensures that the text information can be understood and accepted by the user in the form of voice. During the conversion process, the server will select voice characteristics suitable for the user's age and interests, such as the speed, tone and style of the voice, based on the user's personal information and preference settings. For example, for an 8-year-old child user, the server will select a voice characteristic with a slower speed and lively tone to increase the attractiveness and comprehensibility of the information. After the conversion is completed, the server stores the voice information generated by TTS and the voice information directly generated by the multimodal model in an accessible storage medium so that the main toy unit can download and play it.

[0040] Step 1023: The server sends a notification of successful voice information storage to the main toy unit through the MQTT Broker. After receiving the notification, the main toy unit downloads the stored voice information and plays it, completing the interactive response.

[0041] Exemplarily, after the server successfully stores the voice information converted from the text information generated by the multimodal model and the voice information directly generated by the multimodal model together, it sends a confirmation notification to the main toy unit through the MQTT Broker. The notification contains the storage location of the voice information, the corresponding identifier, and the synchronization information to ensure that the main toy unit can accurately identify and download the corresponding voice content. After receiving the notification forwarded by the MQTT Broker, the main toy unit establishes a connection with the server through the network and downloads the stored voice file according to the information in the notification. After the voice file is downloaded, the speaker of the main toy unit will play the stored voice information. For example, the speaker clearly plays "the car engine generates power by burning fuel, pushing the piston to move, thereby driving the wheels to rotate", "if you want to build an engine yourself, the best way is to become a mechanical engineer", providing users with real-time interactive guidance and educational knowledge.

[0042] Step 103: The main toy unit dynamically adjusts the interaction mode and output content according to the user setting mode, the connection status of each toy unit, the RFID tag information, the visual input, and the voice input.

[0043] For example, reference Figure 5In the mobile phone interactive interface shown, users can enter "child age", "toy type", "knowledge type selection", "voice print" and "face" as the basic settings of the main toy unit. The user's selection is recorded by the system and used to guide and optimize the subsequent interactive content to ensure that the interactive experience matches the user's selection. Furthermore, the main toy unit collects the user's voice instructions through the built-in microphone and transmits these instructions to the server for further processing. At the same time, when the user brings the main toy unit and the auxiliary toy unit close together, the RFID reader of the main toy unit reads the RFID tag information of the auxiliary toy unit and sends it to the server through the MQTT Broker. In addition, the main toy unit monitors the user's interactive scene through the camera to ensure that the system can respond to the user's visual input in real time. When the user's visual interaction with the toy changes, the camera captures the corresponding visual information and notifies the server of these state changes in real time. After receiving the user's voice instructions, RFID tag information and visual input, the server will perform a series of dynamic adjustment operations to ensure the personalization and educational nature of the interactive content. The server will access a comprehensive user information knowledge base, which not only contains the user's basic information such as age, preferences and historical interaction records, but also integrates educational content and historical data related to toy interaction. When the user adds a car engine auxiliary toy unit during the interaction with the car toy, the server will recognize this action and retrieve information related to the engine from the knowledge base. This information may include the working principle of the engine, historical background, etc. The server will dynamically generate interactive content suitable for the user's age and interests based on the user's personal information and the current interaction context. For example, if the user is an 8-year-old child who is interested in how cars work, the server may provide a simplified explanation of how car engines work. At the same time, the server will dynamically adjust the content of the knowledge base based on the user's interactive feedback and newly collected information after receiving the data generated by the multimodal model to improve the accuracy and relevance of future interactions. The server sends the adjusted interactive content to the main toy unit, and the main toy unit generates personalized interactive responses based on this information. It will include playing educational voice information related to the car engine and providing interactive suggestions. At the same time, the server will monitor the user's interaction process and collect new data points, such as user feedback on specific information or new points of interest. These data will be used to further enrich the user information knowledge base, so that the system can evolve over time and better meet the personalized needs of users. Figure 4 As shown, the specific implementation of step 103 includes:

[0044] Step 1031: The user inputs user information, including age, toy type, knowledge type, voiceprint, and face, and begins to interact with the toy through voice, assembly, and vision.

[0045] For example, reference Figure 5 In the mobile phone interaction interface shown, the user can enter "child age", "toy type", "knowledge type selection", "voice print" and "face" as the basic settings of the main toy unit. For example, the user can select "8 years old" as the child's age, select "car" as the toy type, select "English" as the knowledge type, record a voice print sample for subsequent voice recognition, and enter a face sample for subsequent face recognition. The user can start the toy by voice activation or physical button activation. When voice activation, the user starts the toy by saying a preset wake-up word, such as "Hey, toy buddy, I'm ready." When physical button activation, the user starts the interaction by pressing the physical button on the main toy unit. This action is recognized by the controller of the main toy unit and triggers the start sequence. After that, the camera starts to collect visual information, such as the shape and color of the auxiliary toy unit held by the user when interacting with the toy, and the microphone starts to collect the user's voice information. If the user says "I want to know about this toy", this voice information will be recorded and converted into a digital signal. These signals are transmitted to the server through MQTT Broker; at the same time, when the user brings the main toy unit and the auxiliary toy unit close to each other, the RFID reader of the main toy unit reads the RFID tag information of the auxiliary toy unit. After reading, this RFID information will also be transmitted to the server through MQTT Broker.

[0046] Step 1032: After the user information, visual information, voice information, and RFID tag information are sent to the server, they are stored in the user information knowledge base.

[0047] For example, when a user is interacting with a car toy, he places an auxiliary toy unit representing a car engine on the main toy unit, and the RFID reader in the main toy unit reads the RFID tag information of the auxiliary toy unit, while the camera of the main toy unit captures the visual scene of the user interacting with the toy. The RFID tag information contains the specific identification of the auxiliary toy unit, and the visual information captured by the camera may include the user's expression and the appearance of the toy. In addition, the user's voice command "I want to know about the engine" is captured by the microphone of the main toy unit and converted into a digital signal. These signals are transmitted to the server through the MQTT Broker. After receiving the RFID tag information, visual information and voice information, the server combines this information with the existing data in the user information knowledge base. The user information knowledge base stores the user's personal information, including but not limited to the child's age, toy type, knowledge type selection, voiceprint data, facial features and past interaction records. The server updates the user information knowledge base by comparing and integrating this information to ensure that it contains the user's latest preferences, interaction records and visual interaction features.

[0048] Step 1033: The server sends the latest user information knowledge base information to the multimodal model, and the multimodal model dynamically adjusts the output content based on the latest knowledge base information.

[0049] Exemplarily, after updating the user information knowledge base, the server sends the latest user information, including the user's interest in car engines, the user's current visual scene of the building, the newly added auxiliary toy unit information, and the user's personal information, such as the child's age and knowledge type selection, to the multimodal model. This information forms the basis for the multimodal model to analyze and generate interactive content. After receiving the information provided by the server, the multimodal model conducts in-depth analysis to understand the user's specific interests and needs. Based on these analysis results, the multimodal model generates personalized interactive content. For example, if the user shows interest in the working principle of the car engine, the multimodal model may generate content consisting of two parts: one part is text information, which is used to introduce the working principle and mechanical characteristics of the car engine in detail; the other part is voice information, which is used to provide a more vivid and direct explanation. The text information is then converted into voice information by the server through the text-to-speech (TTS) service. This conversion process involves converting text data into a natural voice stream so that the user can hear it through the speaker of the main toy unit. The converted voice information is stored in the specified storage address of the server. At the same time, the voice information part directly generated by the multimodal model does not require TTS conversion and can be directly stored and sent. After storing the voice information, the server sends a storage completion notification to the main toy unit through MQTTBroker. The notification contains the storage address of the voice information and the necessary synchronization information to ensure that the main toy unit can accurately identify and download the corresponding voice content. After receiving the storage completion notification, the main toy unit establishes a connection with the server through the network and downloads the voice file according to the storage address in the notification. After the download is complete, the control unit of the main toy unit activates the built-in speaker to play the downloaded voice information to complete the interactive response with the user. In addition, during the user's interaction with the toy, the server will continue to collect user feedback information and new interaction data, such as the user's interest in specific car information or interactive behavior with other parts of the car. These data are used to further enrich the user information knowledge base, so that the system can evolve over time and better meet the personalized needs of users. In this way, the multimodal model can provide a richer and more personalized interactive experience and improve the educational and entertainment value of the toy.

[0050] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for using speech, vision, and RFID as inputs of a multimodal model, the method comprising: A main toy unit and multiple auxiliary toy units, the main toy unit is equipped with a controller connected to an RFID reader, a microphone, a camera, an RFID tag and a speaker, and each auxiliary toy unit is equipped with an RFID tag; When the main toy unit is started, the user can connect and set the main toy unit mode through the mobile phone. The camera collects visual information, the microphone collects voice information and sends a signal to the multimodal model. When the auxiliary toy unit is close to the main toy unit, the RFID tag is activated and sends a signal to the multimodal model. After the model analyzes the signal, it generates the model output and returns it to the main toy unit, which then completes the interactive response; The main toy unit dynamically adjusts the interaction mode and output content according to the user setting mode, the connection status of each toy unit, RFID tag information, visual input, and voice input.

2. The method according to claim 1, characterized in that: The controller of the main toy unit has the function of identifying the RFID tag information of the auxiliary toy unit to determine the identity of the auxiliary toy unit.

3. The method according to claim 1, characterized in that: The voice information collected by the microphone includes voice recognition and text transcription. The recognized voice information is sent to the multimodal model, and the multimodal model generates a model output and returns it to the main toy unit.

4. The method according to claim 1, characterized in that: The multimodal model can parse the visual information collected by the camera and combine it with the voice information and RFID tag information to generate a more accurate model output.

5. The method according to claim 1, characterized in that: The main toy unit can dynamically adjust the interaction mode and output content, including but not limited to voice, music, light and action, according to the RFID tag information and visual and voice input of the auxiliary toy unit.

6. The method according to claim 1, characterized in that: The controller of the main toy unit has a learning function and can optimize the interactive response and output content in real time according to the user's interaction history and preferences.

7. The method according to claim 1, characterized in that: The connection between the main toy unit and the cloud server is wireless, and the wireless connection uses the Wi-Fi communication protocol.

8. The method according to claim 1, characterized in that: The main toy unit has multi-language processing capabilities and can provide corresponding language interactions based on the user's geographical location and language preference.

9. The method according to claim 1, characterized in that: The main toy unit is capable of recording the user's interaction data and, with the user's consent, using the data for further training and optimization of the model.

10. The method according to claim 1, characterized in that: The controller of the main toy unit has a multi-round dialogue function, which can maintain contextual consistency based on the user's input and historical interaction records to achieve smooth multi-round dialogue interaction.

11. The method according to claim 1, characterized in that: The voice information collected by the microphone also includes voiceprint recognition, which can identify the voice characteristics of a specific user to provide a personalized interactive response.

12. The method according to claim 1, characterized in that: Users can wirelessly connect to the main toy unit through their mobile phones and set the interaction mode of the main toy unit to achieve a personalized interactive experience.

13. The method according to claim 1, characterized in that: The information collected by the camera also includes facial recognition, which can identify the facial features of a specific user to provide personalized interactive responses.

14. The method according to claim 1, characterized in that: The connection state identification mechanism between the main toy unit and the auxiliary toy unit includes approach, contact and separation, and each connection state corresponds to a different interaction mode.

Images