Multi-mode NFC role switching AI tide play base device and interaction method

Abstract

The invention provides a multi-mode NFC role switching AI tide play base device and an interaction method, and aims to solve the problems that role switching is not dynamic, multi-mode alignment is difficult, the safety is low, the calculation requirement is high and the like in an existing NFC tide play technology. The base device comprises an NFC identification module, a multi-mode sensor layer, an edge computing unit, a cloud interface and a security module; the interaction method comprises the steps that a user places an IP hand board with an NFC label on a base, reads a label ID and then downloads and loads a role data packet, the multi-mode sensor collects input and fuses the input, the edge calculation unit conducts local reasoning to generate a response, the safety module adopts non-falling encryption to protect the IP copyright, and a lamplight synchronization algorithm shows the lamplight effect. According to the invention, flexible switching of IP roles, real-time multi-mode fusion response, data encryption protection and low-delay interaction are realized, and the competitiveness of the AI tidal play base in the IP derivative market is improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence knowledge base management and personalized dialogue system technology, and in particular to an AI trendy toy dock device and interaction method with multimodal NFC role switching. Background Technology

[0002] In the field of AI knowledge base management and interaction technology, existing technologies have made certain progress. For example, technologies such as Amazon Bedrock Knowledge Bases and Rezolve.ai support the integration of external knowledge into large language models (LLMs) through RAG (Retrieval Augmentation) technology, providing basic support for knowledge management. For multi-role needs, Multi-tenant RAG systems have implemented personalized knowledge access functions. Meanwhile, AI agent systems based on LLM, such as Moveworks' AI Knowledge Base and Atlassian Confluence's Atlassian Intelligence, can dynamically generate content and adapt to user roles, and generate responses by fine-tuning the knowledge base, becoming important technical references in this field.

[0003] Meanwhile, in the field of interactive toys and smart hardware, Near Field Communication (NFC) or Radio Frequency Identification (RFID) technologies have been used to achieve interaction between physical and virtual environments. For example, US Patent 20120258802A1 describes an interactive toy system that creates characters by scanning cards or tags, enabling characters to switch between different media and connect physical and virtual environments. However, it does not integrate multimodal sensor data (such as voice, vision, and light) and relies on cloud computing, resulting in high response latency. US Patent 8475275B2 focuses on the connection between physical toys and virtual games, activating character interaction functions through tags. It does not integrate the ability to generate personalized parameters of large AI models, nor does it have a real-time emotion perception mechanism. US Patent 7878905B2 emphasizes multi-layered interactive gameplay, but such technologies generally lack AI-driven multimodal fusion capabilities. Among related domestic technologies, Huawei's AIoT edge smart hardware supports multimodal perception, but it is mainly applied to audio devices and does not implement NFC character switching for trendy toy bases. Tencent Smart Retail's trendy toy ecosystem involves IP content operation, but lacks the ability to dynamically load characters onto hardware bases.

[0004] Overall, existing technologies still have many shortcomings that need to be addressed: (1) Multimodal inputs are not effectively aligned, resulting in fragmented voice, visual, and lighting interaction dimensions and a poor user experience; (2) The computation process relies excessively on the cloud, with weak edge-side inference capabilities, and cannot support offline modes, limiting its application in scenarios such as child and elderly companionship. Furthermore, cloud reliance increases computational costs, which is not conducive to the promotion of portable trendy toys; (3) Raw voice data needs to be uploaded to the cloud for storage, posing a high risk of privacy leakage and making it unsuitable for low-power companionship devices; (4) Role switching lacks dynamism, only supporting preset roles and unable to hot-load new IP data packets, resulting in poor scalability; (5) There is a lack of effective data encryption mechanisms, making IP copyrights easy to be stolen. These problems collectively limit the application depth and market competitiveness of existing technologies in the fields of AI trendy toys, IP derivatives, and smart home companionship devices. Therefore, it is urgent to develop a technical solution that can solve the above-mentioned shortcomings. Summary of the Invention

[0005] To address the aforementioned technical issues, this application provides a multimodal NFC role-switching AI trendy toy dock device and interaction method.

[0006] According to one aspect of this application, an interaction method for a multimodal NFC role-switching AI trendy toy dock is proposed, comprising: S1: Scan the NFC tag of the figure IP to obtain the IP figure tag ID, and encapsulate the IP figure tag ID into an ID frame.

[0007] The IP figurine tag ID is a unique digital identifier embedded in the NFC tag of the IP figurine. It is a string that conforms to the ISO / IEC 14443 A standard and is pre-written into the tag by the IP copyright holder. It is the core basis for distinguishing different IP roles. S2: Parse the IP figurine tag ID from the ID frame, and call the pre-stored ID index table to determine whether to download the "soul pack" of the character corresponding to the IP figurine tag ID by comparing the IP or version. If download is required, initiate a download request for the "soul pack".

[0008] The ID index table is an encrypted index table that records the cached IP figurine tag IDs and corresponding "soul pack" metadata. It is maintained regularly and stored using AES-256 encryption to prevent tampering. The “soul package” includes a voice library, interactive scripts, a fine-tuned LLM model, and personality parameters; wherein the voice library supports few-shot cloning, the interactive scripts define dialogue logic, the fine-tuned LLM model is based on Llama or a similar lightweight model, and the personality parameters include preset emotion thresholds and a knowledge base subset. Furthermore, the voice library of the Soul Pack adopts TTS parameterized storage, containing basic voice features and few-shot cloning parameters, and supports the generation of voices with corresponding character timbres within 100ms after inputting text, achieving high-precision cloning with few samples without the need for training with a large number of samples; and the voice library has built-in basic voice features and intonation fluctuation coefficients, supporting the generation of natural character timbres within 100ms after inputting text, avoiding timbretic distortion and robotic quality caused by cloud transmission.

[0009] S3: After receiving the download request for the "Soul Pack", obtain the latest version of the "Soul Pack" through encrypted streaming communication.

[0010] Download the "soul packet" obtained from encrypted streaming communication and load the "soul packet" into memory, then process it using a non-disk encryption mechanism; S4: Load the “Soul Pack” into memory and build a character knowledge base based on the contents of the “Soul Pack”.

[0011] The character knowledge base is a structured set of data specific to IP characters, formed by integrating fine-tuned LLM model weights, speech synthesis parameters, interactive script key-value pairs, and personality parameter configuration files after loading the "soul package". It is used to provide support for IP characters' exclusive semantic understanding, dialogue generation, voice style, and emotional response logic. Furthermore, when loading the "Soul Package", an "authentication request" is first sent to the cloud platform, carrying the device's unique code and a temporary token generated by the security module based on AES-256 encryption. After the cloud platform verifies the request, it returns a "download token" instruction. The "download token" is then used to initiate a streaming request. The cloud platform splits the "Soul Package" into data packets containing sequence numbers and transmits them through a combined HTTPS and AES-256 encrypted channel. The cloud is a remote server network cluster based on the Internet. Specifically, it is a collection of data centers operated by cloud service providers and distributed globally or in specific regions, which centrally store the latest version of the "Soul Pack". Furthermore, if the latest version of the "Soul Package" is already cached locally and is complete, the "Soul Package" file can be read directly from the cache area of ​​the storage module and loaded into memory through memory mapping, avoiding the time-consuming copying process of reading files and writing them to memory in the traditional way.

[0012] S5: Collect user input data in real time, extract features from the collected user input data, perform multimodal input fusion to generate emotional states, and finally generate personalized responses based on the emotional states and the constructed role knowledge base.

[0013] The user input data includes voice data and facial image data; The personalized response includes voice response and light response.

[0014] Furthermore, during the process of collecting user input data, voice signals are collected and features are extracted using the MFCC algorithm. Then, the similarity between the MFCC features of each frame and the "pronunciation feature template" of the currently loaded character in the "Soul Package" personality parameters is calculated, using the following formula: in, The attention weights for the i-th frame (0≤ ≤1), For the k-th dimension MFCC feature of the i-th frame, , These are the mean and variance of the k-th dimension feature of the character name template, respectively. The MFCC feature of each frame is multiplied by the attention weight of the corresponding frame to obtain the weighted MFCC feature matrix. Then, the fully connected layer of the CNN network structure is used to infer whether the character name in the voice command is consistent with the currently loaded character.

[0015] Furthermore, when collecting user input data, the camera captures three consecutive frames of facial images and performs the following image processing: aligning the face to obtain the aligned three frames of facial images, which are then recorded as follows: , , ;Will , , Input each frame into the MobileNetV2 network, extract the high-level feature maps for each frame, and output the feature vectors, denoted as... , , ; Will , , The feature sequence is constructed in chronological order and input into a bidirectional LSTM network for bidirectional feature learning as follows: Forward LSTM from arrive Positive changes in learning emotions; Backward LSTM from arrive The inverse relationship characteristics of learning emotions; The outputs of the bidirectional LSTM are concatenated to obtain the temporal fusion feature vector. (t=1,2,3); finally, a 1×1 convolutional layer is applied to... Dimensionality reduction yields feature vectors Collect facial images labeled with various emotions, and perform the same image processing steps described above to obtain prototype vectors for each emotion. , represents the prototype vector of the m-th emotion; Calculate separately The cosine similarity between the vector and the prototype vector of each category is used, and the emotion corresponding to the prototype vector with the highest similarity is selected as the preliminary emotion result. This similarity value is also used as the initial confidence score, denoted as follows: ; right Calculate the cosine similarity between each pair of adjacent frames, and set the initial confidence level for each frame as follows: The formula for assigning dynamic weights is as follows: ,in The preset allocation coefficient; Finally, weighted scores were calculated for each of the five emotion categories. The emotion with the highest score is selected as the final result. The normalization result is The confidence level of the final emotion; If the confidence level of the final emotion is greater than or equal to the preset emotion threshold, it is determined to be a valid emotion and is used as the emotional state; if the confidence level is less than the preset emotion threshold, the camera is triggered to fill in the light, and 3 more frames of images are re-captured to repeat the above process; if the confidence level of the final emotion after 3 re-captures is still less than the preset emotion threshold, the cloud is requested to verify and the "soul package" is resent.

[0016] Preferably, the preset emotion threshold is set to 80%.

[0017] Finally, the generated emotional states will be inferred locally, and personalized responses will be generated by combining them with the constructed character knowledge base. Finally, the corresponding audio and lighting effects will be output based on the personalized response content to complete the first interactive response.

[0018] According to another aspect of this application, an AI-powered trendy toy dock device with multimodal NFC role switching is proposed, including an NFC identification module, a multimodal sensor layer, an edge computing unit, a cloud interface, and a security module; The NFC identification module is used to identify and acquire the NFC tag of the collectible figure IP, parse and process it to obtain the IP collectible figure tag ID and encapsulate it into an ID frame; The edge computing unit is used to receive ID frames to trigger the download operation of the "Soul Pack". Specifically, it determines the download or loading requirement of the "Soul Pack" by comparing IP or version. If download is required, it initiates a download request for the "Soul Pack" to the cloud interface. In addition, it is also used to load the "Soul Pack" and build a character knowledge base based on the content of the "Soul Pack", and generate personalized responses based on emotional state and the character knowledge base. Furthermore, the "soul package" includes a voice library, an interactive script, a fine-tuned LLM model, and personality parameters; wherein the voice library supports few-shot cloning, the interactive script defines the dialogue logic, the fine-tuned LLM model is based on Llama or a similar lightweight model, and the personality parameters include a preset emotion threshold and a knowledge base subset. The cloud interface is used to receive download requests for the "soul package" from the edge computing unit, obtain the latest version of the "soul package" from the cloud through encrypted streaming communication, and transmit the obtained "soul package" to the edge computing unit. The multimodal sensor layer includes a microphone, a camera, and a light control unit, used to collect user input data and generate emotional states based on the collected user input data; Preferably, the microphone in the multimodal sensor layer is a high-sensitivity microphone, the camera is a high-definition camera, and the lighting control unit is an RGB LED light ring; The edge computing unit uses a low-power AI chip based on the ARM architecture and runs a lightweight large model (multimodal Transformer model) for local inference, enabling offline or online interaction, and is also suitable for scenarios such as accompanying children and the elderly. The security module adopts a non-disk encryption mechanism, which only opens a temporary encryption area in the memory of the edge computing unit and does not call the persistent flash memory of the storage module to persist data, thus protecting IP data copyright. In the aforementioned AI-powered trendy toy base device with multimodal NFC character switching, the core area is the NFC recognition area (antenna). This area contains an NFC recognition module that supports the ISO / IEC 14443 standard to read the tag ID of the figure's IP. Users can place the figure's tag in this area to trigger the recognition and loading of the corresponding IP character's "soul pack".

[0019] Furthermore, in the aforementioned multimodal NFC role-switching AI trendy toy dock device, the edge computing unit integrates a light synchronization algorithm, which maps the emotional state during interaction to a preset light parameter matrix to obtain specific light mapping parameters; then, the light mapping parameters are encapsulated into standardized control commands, which are used as the content of the light response to ensure that the light control unit can accurately parse them; then, the edge computing unit transmits the generated light response to the light control unit through the SPI interface; after receiving the light response, the light control unit completes the status update of all LED beads within 1 second, synchronously presenting a light effect that matches the emotional state.

[0020] Furthermore, the aforementioned AI-powered trendy toy dock device with multimodal NFC role switching also includes an audio output module, which converts the voice response generated by the edge computing unit into a sound wave signal and uses a full-range micro speaker to achieve clear sound quality output.

[0021] Furthermore, the aforementioned AI trendy toy dock device with multimodal NFC role switching also includes a storage module for temporarily caching the "soul pack" and interaction logs. It uses an encrypted flash memory chip to temporarily store the data, and the non-disk storage mechanism of the security module ensures that the data is not stored on the ground.

[0022] Furthermore, the aforementioned multimodal NFC role-switching AI trendy toy dock also includes a power module, which provides stable power support for all components of the device, supports a power interface and a built-in lithium battery power mode, and meets the needs of both fixed desktop use and portable mobile scenarios.

[0023] The beneficial effects of the multimodal NFC role-switching AI trendy toy base device and interaction method of the present invention are as follows: (1) The fusion processing of multimodal sensors and the lighting synchronization algorithm enable voice and visual lighting interaction to form a synergy, avoiding fragmented experience; (2) The low-power AI chip based on the edge computing unit ARM architecture runs a lightweight large model, and the soul package is downloaded and cached locally, without the need for an external network, which is suitable for offline companionship scenarios; and local inference reduces the cost of cloud computing power rental and reduces equipment deployment and operation expenses; (3) Feature extraction, emotion analysis and response generation are completed on the edge side through the edge computing unit, and the original data does not need to be uploaded to the cloud. Combined with the non-disk encryption of the security module and the automatic clearing mechanism of the interaction log of the storage module, privacy is protected; (4) Through the dynamic hot loading mechanism triggered by the NFC tag, the NFC identification module reads the IP figurine tag ID to trigger the soul package download, and the cloud interface adopts encrypted streaming communication. (5) Security mechanisms such as non-disk encryption and temporary log caching not only protect IP copyright but also further reduce the risk of privacy data leakage; (6) Realize real-time emotional fusion and personalized heartwarming response. Relying on the multimodal sensor layer to collect user face and voice data, the lightweight large model is used to generate a comprehensive emotional state. Combined with the soul package, the edge computing unit calls the role knowledge base to generate a response that matches the emotion, solving the problem of insufficient emotional fusion and inability to match the user's emotions; (7) Through the design of few-shot cloning parameters of the voice library and TTS parameterized storage, combined with the localized speech synthesis of the edge computing unit, the distortion of tone caused by a large number of sample dependencies and cloud transmission is avoided, solving the problem of low cloning accuracy and robotic voice in the scenario with few samples. Attached Figure Description

[0024] Figure 1 This is an interaction flowchart of the interaction method of the AI ​​trendy toy dock with multimodal NFC role switching according to the present invention; Figure 2 This is a schematic diagram of the hardware structure of the AI ​​trendy toy dock device with multimodal NFC role switching according to the present invention. Detailed Implementation

[0025] To provide a further understanding of the purpose, structure, features, and functions of this invention, Example embodiments according to this application will be described in detail with reference to the accompanying drawings.

[0026] Flowcharts are used in this application to illustrate the operations performed according to embodiments of this application. Hereinafter, exemplary embodiments according to this application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein.

[0027] Example 1: As Figure 1 As shown, an interaction method for a multimodal NFC-enabled AI-powered toy dock with character switching includes the following steps:

[0028] S1: Scan the NFC tag of the figure IP to obtain the IP figure tag ID, and encapsulate the IP figure tag ID into an ID frame.

[0029] When a user places an IP figurine with an NFC tag conforming to the ISO / IEC 14443 A standard on the NFC recognition area at the top center of the base device, the NFC recognition module in the NFC recognition area is activated. Simultaneously, a "scan command" is sent to read the IP figurine's NFC tag, parse and process it to obtain the IP figurine tag ID, and encapsulate the IP figurine tag ID into an ID frame.

[0030] Furthermore, if multiple NFC tags exist within the NFC recognition area, the NFC module will only read the closest NFC tag.

[0031] S2: Parse the IP figurine tag ID from the ID frame, and call the pre-stored ID index table to determine whether to download the "soul pack" of the character corresponding to the IP figurine tag ID by comparing the IP or version. If download is required, initiate a download request for the "soul pack".

[0032] Upon receiving the ID frame, the IP figurine tag ID is first parsed out, and then the pre-stored ID index table shown in Table 1 is called.

[0033] Table 1: ID Index Table; IP figurine tag ID "Soul Pack" version number "Soul Pack" Size (MB) Verification code NFC001-IP001-00010 V1.2.0 12.3 a1b2c3d4e5f6... NFC001-IP002-01001 V1.4.2 15.9 f6e5d4c3b2a1... NFC001-IP006-10010 V3.1.0 23.4 3d4e5f6a7b8c... The ID index table is maintained regularly and automatically synchronized each time the "Soul Package" is loaded or updated. It is stored using AES-256 encryption and the key is dynamically generated.

[0034] The following criteria are used to determine whether to download the character "soul pack" for the IP figure tag ID: First, the ID index table is traversed. If the current IP figure tag ID is not found, it is determined to be a "new IP character". The download process is triggered and the current IP figure tag ID and the corresponding "soul pack" metadata are added to the ID index table as a new record in a complete and synchronized manner. If the tag ID of the current figure IP is found, the version number of "Soul Pack" in the ID index table is extracted, a version query request is sent, and the latest version number is obtained. If the local version is lower than the latest version, it is determined that an incremental update is required, and the download process is triggered.

[0035] Furthermore, if the latest version is already cached locally and is complete, the local cached "soul pack" can be used directly without downloading it again.

[0036] S3: After receiving the download request for the "Soul Pack", obtain the latest version of the "Soul Pack" through encrypted streaming communication; The "Soul Pack" is the core data carrier of the IP character. After processing with the LZ4 compression algorithm, the size of the "Soul Pack" is controlled within 50MB, and it supports incremental updates. Specifically, it contains four types of structured data, and the format and function of each data are as follows: (1) Fine-tuning of the LLM model: Based on the INT4 quantization version of Llama-2-7B-Chat, the model weights are stored as binary files (.bin format), containing 12 layers of Transformer encoders, and the context window length is adapted to the NPU computing power of the edge computing unit; (2) Voice library: It adopts TTS (text-to-speech) parameterized storage, which includes basic speech features and few-shot cloning parameters. The format is .protobuf. It supports the generation of the corresponding character's voice within 100ms after inputting text. It does not require a large number of samples for training, realizes high-precision cloning with few samples, and improves the accuracy of few-sample cloning. In addition, the voice library has built-in basic speech features and intonation fluctuation coefficients, which supports the generation of natural character voices within 100ms after inputting text, avoiding voice distortion and robotic texture caused by cloud transmission.

[0037] Furthermore, the basic speech features are specifically: fundamental frequency 50-500Hz, speech rate 120-180 words / minute; the few-shot cloning parameters are based on the voiceprint features extracted from 3-5 sample speech sentences and stored as a voiceprint feature vector.

[0038] (3) Interactive script: JSON format file (.json), which defines the dialogue logic of the character and includes the mapping between trigger words and response sentences; (4) Personality parameters: XML format file (.xml) containing emotion threshold and knowledge base subset.

[0039] The knowledge base subset consists of exclusive knowledge based on IP characters, including "classic lines" of anime characters, and is stored as a compressed index of the RAG retrieval library.

[0040] S4: Load the “Soul Pack” into memory and build a character knowledge base based on the contents of the “Soul Pack”.

[0041] After obtaining the latest version of the "Soul Pack", load the obtained "Soul Pack" into memory;

[0042] When loading the "Soul Package", an "authentication request" is first sent to the cloud platform, carrying the device's unique code and a temporary token generated by the security module based on AES-256 encryption. After the cloud platform verifies the request, it returns a "download token" instruction. The "download token" is then used to initiate a streaming request. The cloud platform splits the "Soul Package" into data packets containing sequence numbers and transmits them through a combined HTTPS and AES-256 encrypted channel. Specifically, if the download is interrupted, the block sequence number of the downloaded data packet is recorded. Once the network is restored, a "resume request" is sent to the cloud to download only the incomplete blocks, avoiding duplicate transmissions.

[0043] Furthermore, during the process of loading the "Soul Package" into memory, the security module uses the AES-256 algorithm to decrypt the file in real time, and the key is dynamically generated and only temporarily stored in memory to ensure that the "Soul Package" is not stored on the ground, i.e., not written to persistent storage media, thus preventing IP copyright leakage.

[0044] After loading the "Soul Pack" into memory, start the initialization process and complete the character knowledge base construction corresponding to the IP figurine tag ID by following steps (Ⅰ)-(Ⅲ): (I) Loading the LLM model: First, load the INT4 quantized Llama model weights into the NPU computing power pool and initialize the model's input layer, attention layer and output layer; The NPU computing power pool is a collection of computing resources of the built-in neural network processing unit (NPU), used to centrally schedule the computing units of the NPU and provide dedicated computing power support; (II) Voice library loading: Extract the TTS parameter file and the .protobuf file of the voiceprint feature vector from the "soul package" in memory; TTS parameters are parsed into recognizable floating-point parameters and mapped to the parameter register of the built-in speech synthesis engine; The voiceprint feature vector is converted into the basic feature matrix of the audio synthesis algorithm, which is used to calibrate the timbre of the synthesized speech; The speech synthesis engine reads the parsed parameters, initializes the core components and completes the TTS parameter configuration. At this time, the speech library is loaded. (III) Loading of Interactive Scripts and Personality Parameters: The interactive scripts in JSON format are parsed into key-value pairs of "trigger-response" and stored in an in-memory hash table; the personality parameters in XML format are converted into binary configuration files so that subsequent emotional responses and knowledge retrieval conform to the role settings.

[0045] The memory hash table is a key-value data structure built in random access memory, specifically designed for quickly storing and querying the mapping relationship between trigger conditions and response content in interactive scripts.

[0046] After constructing the role knowledge base corresponding to the IP figurine tag ID through the above steps (Ⅰ)-(Ⅲ), the interaction rules for adapting to the new IP role are completed, and the entire process of switching to the new IP role is thus completed.

[0047] S5: Collect user input data in real time, extract features from the collected user input data, perform multimodal input fusion to generate emotional states, and finally generate personalized responses based on the emotional states and the constructed role knowledge base.

[0048] Once the character's "soul pack" is loaded, the microphone captures ambient speech at a 16kHz sampling rate, and the camera captures the user's facial image at a 30fps frame rate. When the user issues their first interaction command, the microphone converts the speech signal into a digital audio stream; simultaneously, the camera captures the user's facial image.

[0049] Feature extraction is performed on the collected speech and facial image data, and the extracted speech and facial image features are used as multimodal inputs: First, MFCC (Mel-frequency cepstral coefficients) features are extracted from the speech. Then, the character name in the instruction is identified by combining the CNN (convolutional neural network) algorithm. On the basis of traditional MFCC+CNN, a "character name-specific attention layer" is added to highlight the key features of the character name pronunciation, reduce environmental noise and irrelevant word interference, and confirm the match with the currently loaded character. The specific processing steps are as follows (5.1)-(5.3): (5.1) Speech signal preprocessing and MFCC feature extraction; The voice signal captured by the microphone is first passed through a pre-emphasis filter to compensate for the attenuation of high-frequency signals and enhance the high-frequency characteristics of the character's name pronunciation; The pre-emphasized speech signal is divided into frames according to a preset frame length and frame shift. Each frame of data is processed through a Hamming window to reduce inter-frame spectral leakage. The MFCC feature calculation process is as follows: First, perform an FFT transformation on each frame of signal with 512 points to obtain the power spectrum; The power spectrum is then filtered using 26 Mel filter banks, and the output energy of each filter is calculated. Then, the logarithm of the energy value is taken, and the 13-dimensional MFCC coefficients are extracted by DCT (Discrete Cosine Transform). The MFCC coefficients are the core of the MFCC features and reflect the static spectral features of the speech signal. The role of DCT is to remove the correlation of the output energy of the Mel filter bank. Generally, the first 13 coefficients can cover the main speech features. The subsequent higher-order coefficients contribute little to speech recognition, so they are discarded to reduce the dimensionality. Next, the first-order difference (13-dimensional) and second-order difference (13-dimensional) of the MFCC coefficients are calculated. The first-order difference coefficients reflect the dynamic rate of change of speech features, and the calculation formula is as follows: ,in, Represents the basic MFCC coefficients of the i-th frame. The first-order difference coefficients are for the t-th frame; the second-order difference coefficients are the second difference of the first-order difference coefficients, reflecting the dynamically changing acceleration, and are calculated using the following formula: ,in These are the second-order difference coefficients; Finally, the MFCC coefficients, first-order difference coefficients, and second-order difference coefficients are fused into a 39-dimensional / frame MFCC feature matrix; the MFCC feature matrix ensures the complete capture of static and dynamic speech features.

[0050] (5.2) Calculation of attention weights; Based on the pronunciation feature template of the currently loaded character (stored in the personality parameters of the "Soul Pack", containing the mean and variance of the MFCC features of each character's name), the similarity between the MFCC features of each frame and the template is calculated, using the following formula: in, The attention weights for the i-th frame are 0 ≤ 0 ≤ 1. ≤1, For the k-th dimension MFCC feature of the i-th frame, , These are the mean and variance of the k-th feature of the character name template, respectively. The MFCC features of each frame are multiplied by the corresponding frame attention weights to obtain a weighted MFCC feature matrix (dimension 100×39). Frames with higher weights have a higher proportion in subsequent recognition, while the weights of irrelevant frames are suppressed.

[0051] (5.3) Classification and Decision Making; The CNN network structure employs a lightweight architecture to reduce the computational power consumption of edge computing units. The output of the fully connected layer calculates the probability using the Softmax function. If the "match" probability is ≥ 92%, the character name in the instruction is determined to match the currently loaded character, triggering an interactive response; otherwise, the instruction is ignored to avoid misidentification. The specific process for calculating the "match" probability of the CNN fully connected layer output is as follows: The fully connected layer of a CNN is designed for binary classification (output dimension is 2), corresponding to two neurons: Neuron 1 outputs the raw score for the "match" category and records it as . ; Neuron 2: Outputs the raw score for the "mismatch" category and denoted as . .

[0052] The two scores mentioned above are unnormalized values ​​calculated by the network through forward propagation, which directly reflect the network's prediction tendency for the two classes. The higher the score, the more obvious the tendency.

[0053] To convert the raw scores into probabilities, the Softmax function is used: ; in, That is, the "match" probability. Let be the probability of "mismatch".

[0054] During the network training phase, parameters are optimized using labeled data, enabling the network to learn the unique features of character name pronunciation: when the input speech contains the current character's name, Significantly greater than , It approaches 1; otherwise it does not. Approaching 0.

[0055] The labeled data includes "voices containing the current character name" and "voices not containing the current character name".

[0056] Ultimately, take As a basis for judgment, if If the threshold is ≥92%, it is determined that the character name in the voice command is consistent with the currently loaded character; otherwise, it means that the character name in the command is inconsistent with the currently loaded character, and a cloud verification is requested and the "soul pack" is resent.

[0057] The aforementioned MFCC calculation and CNN inference process are executed in the NPU computing power pool; the character's pronunciation feature template is loaded from the "soul pack" and is automatically updated each time a character is switched to ensure that the recognition is adapted to the current character.

[0058] Traditional speech recognition is susceptible to environmental noise and irrelevant words, leading to incorrect character name recognition and requiring users to repeat the prompt. By combining MFCC and CNN algorithms, the accuracy of character name recognition is improved, enabling reliable recognition even in noisy environments. This makes user operation more convenient, especially suitable for home scenarios with multiple devices and high noise levels, and improves the efficiency of character switching.

[0059] Subsequently, facial emotion recognition is performed on the images based on the OpenCV library and the MobileNetV2 model. The specific processing steps include the following (5.3.1)-(5.3.4): (5.3.1) Image acquisition and preprocessing; The camera captures three consecutive frames of facial images at a frame rate of 30fps. Preprocessing is performed using the OpenCV library, which provides low-level image manipulation functions. Specifically, the raw images captured by the camera are converted to RGB format using OpenCV, and the `cv2.resize()` function is called to compress the image from its original resolution to 224×224 to fit the input size of MobileNetV2. Next, adaptive histogram equalization (CLAHE algorithm, clipLimit=2.0, grid size 8×8) is performed on each frame to eliminate uneven brightness caused by backlighting and low light. Simultaneously, based on OpenCV's channel splitting (`cv2.split()`) and mean calculation functions, the image color temperature is unified to 6500K to ensure stable skin tone features.

[0060] Then, an "occlusion detection branch" is added to the MTCNN algorithm to recognize faces wearing masks and glasses. It detects facial regions in each frame, outputs bounding boxes, and then performs an affine transformation on the face based on 68 facial landmarks to align the face to a standard pose, resulting in three aligned facial images, which are then denoted as... , , This reduces the impact of posture deviations on emotion recognition.

[0061] (5.3.2) MobileNetV2 basic feature extraction; Will , , The data is input into the MobileNetV2 network, and high-level feature maps for each frame are extracted. Specifically, an "emotional feature channel attention" module (SE module) is added to the "bottleneck layer" of MobileNetV2. The attention weights of each channel are used to enhance the weights of emotion-related feature channels and suppress irrelevant channels (background, hair, etc.). The calculation process of the SE module is as follows: First, global average pooling is performed on the H×W×C feature map output by the bottleneck layer to obtain 1×1×C channel features. Specifically: Calculate the average value of the H×W feature map for each channel using the following formula: in It is the feature value of the i-th row and j-th column of the c-th channel, where c = 1, 2, ..., C. It is the global average feature of the c-th channel; finally, a 1×1×C channel feature vector is obtained, which compresses spatial information and preserves the global features of the channel; Then, attention weights for each channel are generated using two fully connected layers and a sigmoid function. And 0≤ ≤1; Specifically, the first fully connected layer is used to reduce the dimensionality of the 1×1×C features, thereby reducing the computational cost, while simultaneously learning the correlation between channels; and the ReLU function is used to introduce non-linearity to enhance feature representation; Furthermore, the first fully connected layer has an input dimension of C and an output dimension of C / 16, and the second fully connected layer has an input dimension of C / 16 and an output dimension of C.

[0062] Next, a second fully connected layer is used to restore the features to the original number of channels; and the Sigmoid function is used to map the output to [0,1] to obtain the weight of each channel. ; Finally, each channel of the original feature map is associated with its corresponding weight. Multiplication, the formula is in These are the enhanced feature values, resulting in an enhanced feature map with dimensions H×W×C. After each aligned frame is processed by MobileNetV2, a 1×1×1280 feature vector is output and denoted as... , , It includes emotion-related features of the face; where 1280 is the fixed number of channels in the MobileNetV2 pre-trained model, which is adapted to the computing power of edge computing units, so as to retain sufficient emotion features and avoid computational delay due to excessive dimensionality.

[0063] (5.3.3) Temporal feature fusion; Will , , The feature sequences, arranged in chronological order, have a dimension of 3×1280 and are then input into a bidirectional LSTM network. Then, the following bidirectional feature learning is performed: Forward LSTM from arrive Positive changes in learning emotions; Backward LSTM from arrive The inverse relationship characteristics of learning emotions; The outputs of the bidirectional LSTM (each with 256 dimensions) are concatenated to obtain a 512-dimensional temporal fusion feature vector. (t=1,2,3); Finally, a 1×1 convolutional layer is used to... Dimensionality reduction yields a 256-dimensional feature vector. This reduces subsequent computational load while preserving temporal correlation information.

[0064] By analyzing the feature vectors of 3 frames , , Perform temporal correlation to capture the continuous trend of emotional changes and avoid misjudgment in a single frame.

[0065] (5.3.4) Emotion Classification and Decision Making; For five emotion categories—joy, sadness, anger, doubt, and calm—face images labeled with each emotion category were first collected. For each image, the above steps (5.3.1) to (5.3.3)—"preprocessing → MobileNetV2 feature extraction → bidirectional LSTM fusion → 1×1 convolution dimensionality reduction"—were performed to obtain a 256-dimensional feature vector. Then, the mean of all vectors for the same emotion category was calculated to obtain the "prototype vector" for that emotion category. Finally, the prototype vectors for the five emotion categories were obtained. The personality parameters are stored in the "Soul Package," among which... It is the prototype vector of the m-th emotion, where m = 1, 2, 3, 4, 5.

[0066] The 256-dimensional temporal feature vectors of the three frames obtained by dimensionality reduction Calculate the cosine similarity with each of the five prototype vectors, using the following formula: Where m=1,2,3,4,5 corresponds to 5 types of emotions, and sim ranges from [-1,1]. The closer the value is to 1, the more similar the feature is to the prototype.

[0067] The emotion corresponding to the prototype vector with the highest similarity is selected as the preliminary emotion result, and this similarity value is used as the initial confidence score, denoted as follows: (If the highest similarity of the first frame is 0.92, then the corresponding initial confidence level is...) (92%).

[0068] The 256-dimensional temporal feature vectors of the three frames obtained by dimensionality reduction Calculate the cosine similarity between adjacent frames. To measure the continuity of emotional characteristics; The initial confidence level for each frame is... The formula for assigning dynamic weights is as follows: And t=1,2,3, where The allocation coefficient is preset to 0.7; when the inter-frame similarity is high, it indicates that the emotion is stable, and the weight is tilted towards the high confidence frame; when a frame has low similarity with the frames before and after it, the weight is suppressed. Finally, weighted scores were calculated for each of the five emotion categories. The emotion with the highest score is selected as the final result. The normalization result is The confidence level of the final emotion.

[0069] If the confidence level of the final emotion is ≥80%, it is determined to be a valid emotion and is used as an emotional state; if the confidence level is <80%, the camera is triggered to fill in the light (the infrared fill light is turned on, and the brightness is 150 lux), and the process is restarted from (5.3.1) image acquisition and preprocessing to ensure recognition accuracy; if the confidence level of the final emotion after 3 re-acquisitions is still <80%, the cloud is requested to verify and the "soul package" is resent.

[0070] Furthermore, the NPU computing power pool is used to perform MobileNetV2 inference and temporal LSTM computation; emotional feature prototypes are stored in the personality parameters of the "soul package", and different roles can be adapted to different prototype thresholds to improve interaction sensitivity.

[0071] Traditional single-frame image recognition is easily affected by changes in lighting and facial expressions, resulting in a high rate of false emotion recognition. By using the MobileNetV2 algorithm with temporal augmentation, when a user's facial expression jitters or ambient lighting changes, the algorithm acquires three consecutive frames of images. After extracting features using MobileNetV2, it uses a bidirectional LSTM to learn the temporal trend of facial expression changes, and then combines this with emotion prototype matching to accurately identify emotions, significantly improving the accuracy of emotion recognition. Furthermore, it can still stably identify emotions even when lighting and facial expressions are dynamically changing.

[0072] Finally, the generated emotional states will be used for local reasoning, combined with the constructed character knowledge base to generate personalized responses. Based on the personalized response content, corresponding audio and lighting effects will be output to complete the initial interactive response. The specific process includes the following steps (A)-(C): (A) The generated emotional state is processed by the light synchronization algorithm, the emotional state in the interaction is mapped to the preset light parameter matrix to obtain the light mapping parameters, and the light mapping parameters are encapsulated to generate the light response; The generated emotional states are processed as follows: Based on the emotional confidence score, a preset light parameter matrix is ​​mapped, and the preset parameter mapping shown in Table 2 is invoked. The light mapping parameters are dynamically adjusted according to the emotional type and emotional confidence score. Table 2: Preset Emotion Confidence - Parameters - Dynamic Pattern Mapping Table; mood RGB parameters Brightness and dynamic mode with confidence level ≥90% Brightness and dynamic mode at 85% ≤ confidence level < 90% Brightness and dynamic mode at 80% ≤ confidence level < 85% joy （50,255,50） Brightness 90, breathing cycle 1.5 seconds Brightness 80, breathing cycle 2 seconds Brightness 70, breathing cycle 2.5 seconds sad （50,255,50） Brightness 60, dimming time 2 seconds Brightness 50, dimming time 3 seconds Brightness 40, dimming time 4 seconds angry （255,50,50） Brightness 90, flicker frequency 1Hz Brightness 80, flicker frequency 0.8Hz Brightness 70, flicker frequency 0.6Hz doubt （255,255,50） Brightness 80, pulse frequency 1.2 seconds Brightness 70, pulse frequency 1.5 seconds Brightness 60, pulse frequency 1.8 seconds calm (100,100,100) Brightness 50, constant light mode Brightness 40, constant light mode Brightness 30, constant light mode The RGB parameters, brightness, and dynamic mode are used as light mapping parameters and encapsulated into SPI instructions, which are then used as the content of the light response.

[0073] Specifically, if the generated emotional state is not processed, a "verification request" is sent to the cloud platform, carrying the device's unique code and a temporary token generated by the security module; After the cloud platform verification is passed, request the latest version information of the "Soul Package" for the corresponding IP role; if an update is required, push the incremental package through the HTTPS+AES-256 encrypted channel; Upon receiving an incremental packet, it is directly loaded into memory through the security module's "non-disk encryption mechanism." The decryption key is temporarily stored in memory and destroyed immediately after use, ensuring that the "soul packet" is not stored on the ground and preventing IP copyright leakage. (B) Adjust the output of personalized lighting effects based on the lighting response, and simultaneously generate and output voice responses based on local inference; Upon receiving a light response, the status of all LED beads is updated within 1 second. This allows the light to synchronize with emotions in real time, providing vivid and three-dimensional visual feedback, making the IP character more lifelike, and enhancing the collectible and interactive value of the toy.

[0074] Synchronously, based on the emotional state generated by multimodal input fusion and the constructed role knowledge base, a voice response is generated; specifically, the voice response is generated by the edge computing unit by combining the emotional state and the user's voice semantic input into a lightweight LLM model, combined with the interactive script of the "soul package" to generate personalized interactive text. (C) Monitor user input in real time to determine whether to initiate a real-time interactive loop; Start a multimodal monitoring timer to monitor the user's voice and visual input in real time. Voice input comes from valid voice detected by the microphone; visual input comes from facial emotion changes detected by the camera. The initial value of the multimodal monitoring timer is set to 5 seconds; valid speech refers to speech with a signal strength ≥ -50dBm.

[0075] If user voice or visual input is detected within 5 seconds, the timer is reset to 5 seconds, and the process returns to step (A) to continue generating responses in a loop. If no user's voice or visual input is detected within 5 seconds, the interaction is deemed "terminated" and temporary memory is released.

[0076] The above process enables collaborative interaction between voice, visuals, and lighting, solving the problems of fragmented multimodal experiences and insufficient emotional interaction in existing technologies.

[0077] The above steps enable dynamic switching of IP characters, solving the problems of fixed and poorly scalable traditional trendy toy characters. Localized inference eliminates cloud dependence, allowing continuous companionship even in offline environments. Furthermore, localized inference reduces cloud computing power rental costs, lowering equipment deployment and operational expenses. The MTCNN algorithm and MobileNetV2 model address the impact of lighting and instantaneous changes in facial expressions on single-frame images, improving emotion recognition accuracy. This ensures high-precision recognition with low power consumption. Simultaneously, through parameter linkage with the "Soul Pack," it adapts to the personalized interaction needs of multiple IP characters, providing core technological support for the commercialization of the AI-powered trendy toy base.

[0078] Example 2: As Figure 2 As shown, this invention proposes an AI-powered trendy toy dock device with multimodal NFC role switching, including an NFC identification module, a multimodal sensor layer, an edge computing unit, a cloud interface, and a security module; The NFC identification module is used to identify and acquire the NFC tag of the collectible figure IP, parse and process it to obtain the IP collectible figure tag ID and encapsulate it into an ID frame; The edge computing unit is used to receive ID frames to trigger the download operation of the "Soul Pack". Specifically, it determines the download or loading requirement of the "Soul Pack" by comparing IP or version. If download is required, it initiates a download request for the "Soul Pack" to the cloud interface. In addition, it is also used to load the "Soul Pack" and build a character knowledge base based on the content of the "Soul Pack", and generate personalized responses based on emotional state and the character knowledge base. The cloud interface is used to receive download requests for the "soul package" from the edge computing unit, obtain the latest version of the "soul package" from the cloud through encrypted streaming communication, and transmit the obtained "soul package" to the edge computing unit. The multimodal sensor layer includes a high-sensitivity microphone, a high-definition camera, and a light control unit with an RGB LED light ring, used to collect user input data and generate emotional states based on the collected user input data; The AI-powered toy base features a circular design, with a main body diameter of approximately 15cm and an overall height of approximately 3cm. The outer shell is made of environmentally friendly ABS material with a frosted finish to enhance grip. Four anti-slip silicone pads are arranged in a square pattern on the bottom to ensure stability. The base's core functional modules work together to form the interactive hardware architecture: The NFC identification area (antenna) is located in the central area of ​​the base device. The central area has a circular recessed design. The upper recess is used to place the IP figure for positioning; the lower recess is used to accommodate the PCB board of the NFC identification module. The PCB board is fixed to the top interactive layer shell by four M2 threaded posts, and its surface is flush with the bottom surface of the upper recessed layer. The NFC identification area has a built-in NFC identification module that conforms to the ISO / IEC 14443 standard, and the NFC identification module is connected to the edge computing unit; The antenna coil is embedded in the PCB board at the bottom of the recess, and the center of the antenna coil coincides with the geometric center of the base device, which can accurately read the IP figure tag ID and ensure the symmetry and stability of tag recognition; Users can trigger a character switch by placing the NFC-tagged figurine in the recessed area. The recessed structure also serves to locate the figurine. Children can independently place the figurine and switch characters, significantly improving the device's ease of use.

[0079] In the multimodal sensor layer, a high-sensitivity microphone is used to collect user voice input to realize voice interaction and command recognition; a high-definition camera is used to collect user visual input and analyze facial emotional state, and the lens is embedded in the housing's preset mounting hole, with the lens surface flush with the housing surface. A rubber sealing ring is set around the mounting hole, and an adapter PCB board is soldered below the camera, which is connected to the main control chip of the motherboard through the MIPI-CSI interface. An infrared fill light is integrated on the adapter PCB board for facial image acquisition in low-light environments. The lighting control unit uses an RGB LED light ring that surrounds the NFC identification area and is fixed to the housing by four positioning posts. The RGB LED light ring is connected to the motherboard's driver chip via an SPI interface to receive control commands generated by a lighting synchronization algorithm. It adjusts the RGB parameters, brightness, and dynamic mode of the RGB LED light ring in real time, so that the lighting effect is precisely matched with the user's mood and the style of the IP character.

[0080] The edge computing unit uses a low-power AI chip based on the ARM architecture to run a multimodal Transformer model for local inference, enabling offline or online interaction; The low-power AI chip is soldered to the center of the motherboard, and a graphene heat-conducting sheet is covered on top of the low-power AI chip to quickly dissipate heat. The low-power AI chip also integrates a storage module and a security module, and is directly connected to the edge computing unit through an onboard bus. The base device has a physical interface area on its side, including the power supply interface of the power module and the full-range miniature speaker of the audio output module.

[0081] The security module is connected to the edge computing unit and soldered onto the motherboard; the storage module is also soldered onto the motherboard via BGA packaging. The security module employs a non-disk encryption mechanism, only allocating a temporary encryption area in the memory of the edge computing unit, and does not use the persistent flash memory of the storage module for data persistence, thus protecting IP data copyright. The power module's power interface supports 5V / 2A input and has a built-in overvoltage and overcurrent protection circuit. The power module is connected to all the above hardware modules through a power management circuit to provide power support to each module.

[0082] In actual use, users can connect a 5V / 2A charger to power the device through the power interface, which is compatible with various scenarios such as household sockets and outdoor power banks. When encountering abnormal situations such as fluctuations in mains power or inferior chargers, the overvoltage and overcurrent protection circuit built into the interface will automatically cut off the power supply to prevent core components such as edge computing units and multimodal sensors from being damaged due to abnormal power supply.

[0083] The audio output module uses a full-range miniature speaker, which is installed at the bottom of the base device and fixed in the preset mounting position of the housing by a buckle. The surface of the mounting position has honeycomb-shaped sound outlet holes to ensure that the sound propagation is unobstructed. The full-range miniature speaker is connected to the audio decoding chip on the motherboard through the audio interface to achieve clear voice output.

[0084] The present invention has been described in the above-described embodiments; however, these embodiments are merely examples for implementing the present invention. It must be noted that the disclosed embodiments do not limit the scope of the present invention. Conversely, any modifications and refinements made without departing from the spirit and scope of the present invention are within the scope of patent protection of the present invention.

Claims

1. An interactive method for an AI-powered trendy toy dock with multimodal NFC character switching, characterized in that, Includes the following steps: S1: Scan the NFC tag of the figure IP to obtain the IP figure tag ID, and encapsulate the IP figure tag ID into an ID frame; The IP figurine tag ID is a unique digital identifier embedded in the NFC tag of the IP figurine. It is a string that conforms to the ISO / IEC 14443 A standard and is pre-written into the tag by the IP copyright holder. It is the core basis for distinguishing different IP roles. S2: Parse the IP figurine tag ID from the ID frame, and call the pre-stored ID index table to determine whether to download the "soul pack" of the character corresponding to the IP figurine tag ID by comparing the IP or version. If download is required, initiate a download request for the "soul pack". The ID index table is an encrypted index table that records the cached IP figurine tag IDs and corresponding "soul pack" metadata. It is maintained and synchronized regularly and uses AES-256 encryption for storage to prevent tampering. The "Soul Package" includes a voice library, interactive scripts, a fine-tuned LLM model, and personality parameters; wherein the voice library supports few-shot cloning, the interactive scripts define dialogue logic, the fine-tuned LLM model is based on Llama or a similar lightweight model, and the personality parameters include preset emotion thresholds and a knowledge base subset. S3: After receiving the download request for the "Soul Pack", obtain the latest version of the "Soul Pack" through encrypted streaming communication; Download the "soul packet" obtained from encrypted streaming communication and load the "soul packet" into memory, then process it using a non-disk encryption mechanism; S4: Load the "Soul Package" into memory and build a character knowledge base based on the contents of the "Soul Package"; The character knowledge base is a structured set of IP character-specific data formed by integrating fine-tuned LLM model weights, speech synthesis parameters, interactive script key-value pairs, and personality parameter configuration files after loading the "soul package". It is used to provide IP character-specific semantic understanding, dialogue generation, voice style, and emotional response logic support. S5: Collect user input data in real time, extract features from the collected user input data, perform multimodal input fusion to generate emotional states, and finally generate personalized responses based on the emotional states and the constructed role knowledge base. The user input data includes voice data and facial image data; The personalized response includes voice response and light response, and finally outputs corresponding audio and light effects based on the personalized response content.

2. The interaction method of the AI ​​trendy toy dock with multimodal NFC character switching according to claim 1, wherein the voice library of the "soul pack" adopts TTS parameterized storage, includes basic voice features and few-shot cloning parameters, and supports the generation of voice corresponding to the character's timbre within 100ms after inputting text; and the voice library has built-in basic voice features and intonation fluctuation coefficient, and supports the generation of natural character timbre within 100ms after inputting text.

3. The interaction method of the multimodal NFC character switching AI trendy toy dock according to claim 1, characterized in that, When loading the "Soul Package", an "authentication request" is first sent to the cloud platform, carrying the device's unique code and a temporary token generated by the security module based on AES-256 encryption. After the cloud platform verifies the request, it returns a "download token" instruction. The "download token" is then used to initiate a streaming request. The cloud splits the "Soul Package" into data packets containing sequence numbers and transmits them through a combined HTTPS and AES-256 encrypted channel. If the latest version of the "Soul Package" is already cached locally and is complete, the "Soul Package" file is read directly from the cache area of ​​the storage module and loaded into memory through memory mapping.

4. The interaction method of the multimodal NFC character switching AI trendy toy dock according to claim 1, characterized in that, During the process of collecting voice data from user input data, the voice signal is acquired and features are extracted using the MFCC algorithm. Then, the similarity between the MFCC features of each frame and the "pronunciation feature template" of the currently loaded character in the "Soul Package" personality parameters is calculated. The formula is as follows: in, The attention weights for the i-th frame (0≤ ≤1), For the k-th dimension MFCC feature of the i-th frame, , These are the mean and variance of the k-th dimension feature of the character name template, respectively. The MFCC feature of each frame is multiplied by the attention weight of the corresponding frame to obtain the weighted MFCC feature matrix. Then, the CNN fully connected layer of the CNN network structure is used to infer whether the character name in the voice command is consistent with the currently loaded character.

5. The interaction method of the multimodal NFC character switching AI trendy toy dock according to claim 1, characterized in that, When collecting facial image data from user input, the camera captures three consecutive frames of facial images and performs the following image processing: aligning the face to obtain the aligned three frames of facial images, which are then recorded as follows: , , ;Will , , Input each frame into the MobileNetV2 network, extract the high-level feature maps for each frame, and output the feature vectors, denoted as... , , ; Will , , The feature sequence is constructed in chronological order and input into a bidirectional LSTM network for bidirectional feature learning as follows: Forward LSTM from arrive Positive changes in learning emotions; Backward LSTM from arrive The inverse relationship characteristics of learning emotions; The outputs of the bidirectional LSTM are concatenated to obtain the temporal fusion feature vector. And t=1,2,3; finally, a 1×1 convolutional layer is used to... Dimensionality reduction yields feature vectors ; Collect facial images labeled with various emotions, and perform the same image processing steps as described above to obtain prototype vectors for each emotion. , represents the prototype vector of the m-th emotion; Calculate separately The cosine similarity between the vector and the prototype vector of each category is used, and the emotion corresponding to the prototype vector with the highest similarity is selected as the preliminary emotion result. This similarity value is also used as the initial confidence score, denoted as follows: ; right Calculate the cosine similarity between each pair of adjacent frames, and set the initial confidence level for each frame as follows: The formula for assigning dynamic weights is as follows: ,in The preset allocation coefficient; Finally, weighted scores were calculated for each of the five emotion categories. The emotion with the highest score is selected as the final result. The normalization result is The confidence level of the final emotion; If the confidence level of the final emotion is greater than or equal to the preset emotion threshold, it is determined to be a valid emotion and is used as an emotional state. If the confidence level is less than the preset emotion threshold, the camera will be illuminated and three more frames will be captured, repeating the above process. If the confidence level of the final emotion after three recaptures is still less than the preset emotion threshold, the cloud will be requested to verify and the "soul package" will be resent.

6. A multimodal NFC character-switching AI trendy toy dock device, characterized in that, Includes an NFC identification module, a multimodal sensor layer, an edge computing unit, a cloud interface, and a security module; The NFC identification module is used to acquire the NFC tag of the figure IP, parse and process it to obtain the IP figure tag ID and encapsulate it into an ID frame; The edge computing unit is used to receive ID frames to trigger the download operation of the "Soul Pack". Specifically, it determines the download or loading requirement of the "Soul Pack" by comparing IP or version. If download is required, it initiates a download request for the "Soul Pack" to the cloud interface. In addition, it is also used to load the "Soul Pack" and build a character knowledge base based on the content of the "Soul Pack", as well as to perform local reasoning on emotional states and generate personalized responses in combination with the built character knowledge base. The cloud interface is used to receive download requests for the "Soul Package" from the edge computing unit, obtain the latest version of the "Soul Package" from the cloud through encrypted streaming communication, and transmit the obtained "Soul Package" to the edge computing unit. The multimodal sensor layer is used to collect user input data and generate emotional states based on the collected user input data; The security module uses a non-disk encryption mechanism, which only creates a temporary encryption area in the memory of the edge computing unit and does not call the persistent flash memory of the storage module to persist data.

7. The multimodal NFC character switching AI trendy toy dock device according to claim 6, characterized in that, It also includes an audio output module, which converts the speech response generated by the edge computing unit into a sound wave signal, and uses a full-range micro speaker to achieve clear sound quality output, matching the timbre characteristics of the voice library in the "Soul Pack".

8. The multimodal NFC character switching AI trendy toy dock device according to claim 6, characterized in that, The edge computing unit integrates a lighting synchronization algorithm for generating lighting responses and presenting lighting effects. Specifically, the lighting synchronization algorithm maps the emotional state in the interaction to a preset lighting parameter matrix to obtain specific lighting mapping parameters; then, it encapsulates the lighting mapping parameters into standardized control instructions, which are used as the content of the lighting response; and the edge computing unit then transmits the generated lighting response to the lighting control unit; after receiving the lighting response, the lighting control unit completes the status update of all LED beads and synchronously presents lighting effects that match the emotional state.

9. The multimodal NFC character switching AI trendy toy dock device according to claim 6, characterized in that, It also includes a storage module for temporarily caching the "soul package" and interaction logs. It uses encrypted flash memory chips to temporarily store data, and works with the security module's non-disk storage mechanism to ensure that the data is not stored on the ground.

Images