Electronic pet multi-mode interaction system based on artificial intelligence

Abstract

The invention relates to an electronic pet multi-modal interaction system based on artificial intelligence. The system comprises a data acquisition module, a multi-modal emotion analysis module, an emotion state modeling module and a dynamic interaction module. The data acquisition module is used for acquiring physiological electric signals, voice data and behavior track information of a user. The multi-modal emotion analysis module adopts a fusion model to carry out correlation analysis on time domain features of physiological signals, semantic features of voice and behavior space features, and generates multi-dimensional emotion feature vectors. The emotional state modeling module performs weighted fusion on the physiological indexes, the voice semantics and the behavior frequency in the multi-dimensional emotional feature vectors based on the LSTM network to generate emotional quantization parameters. And the dynamic interaction module generates an emotion interaction action instruction based on the user interaction intention in combination with the emotion quantization parameter and a preset dynamic interaction mechanism.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and human-computer interaction, and in particular relates to an electronic pet multimodal interaction system based on artificial intelligence. Background Art

[0002] With the rapid development of artificial intelligence (AI) technology, electronic pets have garnered widespread attention as an emerging form of interactive entertainment. Existing electronic pet systems typically rely on single-modal interaction methods, such as voice or touch, and lack the ability to comprehensively process multimodal user information. This single-modal interaction approach limits the depth and authenticity of the emotional connection between the electronic pet and the user. Furthermore, existing systems often fail to effectively simulate the interactions between objects in real physical environments when handling complex interaction scenarios, resulting in a lack of immersion and realism in the interactive experience. Summary of the Invention

[0003] Based on this, it is necessary to provide an artificial intelligence-based electronic pet multimodal interaction system that can solve the above technical problems.

[0004] In a first aspect, the present application provides an artificial intelligence-based electronic pet multimodal interaction system, comprising:

[0005] Data acquisition module, used to obtain the user's physiological electrical signals, voice data and behavioral trajectory information;

[0006] Multimodal emotion analysis module, which uses a fusion model to correlate and analyze the time domain features of physiological signals, the semantic features of speech, and the spatial features of behavior to generate a multidimensional emotion feature vector;

[0007] The emotional state modeling module is used to perform weighted fusion processing on physiological indicators, voice fundamental frequency, and behavior frequency in the multi-dimensional emotional feature vector based on the LSTM network to generate emotional quantitative parameters;

[0008] The dynamic interaction module is used to generate emotional interaction action instructions based on the user's interaction intention, combined with emotional quantification parameters and preset dynamic interaction mechanisms; the emotional interaction action instructions are used to control the behavior of the electronic pet to achieve dynamic interaction.

[0009] In one embodiment, the dynamic interaction module is further configured to:

[0010] Acquire the stereoscopic spatial data of real scenes in real time and construct 3D point cloud features based on the stereoscopic spatial data;

[0011] Based on 3D point cloud features, a spatial topological relationship matrix of physical objects is established through a spatiotemporal graph convolutional network, and an interactive utility function is generated by combining it with a reinforcement learning strategy network.

[0012] Based on the interaction utility function, the inverse kinematics Jacobian matrix is used to generate physical interaction action instructions.

[0013] In one embodiment, the dynamic interaction module is further configured to construct a spatial topological relationship matrix of entities using the following formula:

[0014]

[0015] φ(p i ,v i )=p i ·e jωt +v i ×ablaB(p i )

[0016]

[0017] f dyn (Δt)=e -λ||Δt|| ·(1+tanh(β||v rel ||))

[0018] Among them, T ij represents the spatial topological relationship matrix between items, φ(p i ,v i ) represents the dynamic position encoding parameter, ψ(p j ,v j ) represents the spatiotemporal attention kernel parameter, f dyn (Δt) represents the dynamic attenuation factor parameter, represents the three-dimensional coordinates of item i, represents the velocity vector of item i, ω represents the spatial frequency parameter, represents the magnetic field gradient field tensor, represents the three-dimensional convolution kernel, represents the attention weight matrix, Represents the pet motion trajectory tensor, λ represents the time attenuation coefficient, Δt represents the data timestamp difference, β represents the speed sensitivity coefficient, Represents the relative velocity vector.

[0019] In one embodiment, the dynamic interaction module is further configured to generate a physical interaction action instruction using the following formula:

[0020] π θ (a|s)=argmax a Q π (s, a)

[0021]

[0022] Among them, θ represents the policy network parameters, a represents the action of the electronic pet, s represents the position, speed and environmental semantic information of the electronic pet, Q π (s, a) represents the state-action value function, represents the collision penalty term, η represents the collision penalty intensity coefficient, P pet Indicates the center of mass position of the electronic pet, P obj represents the center of mass of the obstacle, n obj represents the obstacle surface normal vector, represents the motion optimization term, ω motion optimization weight, represents the motion trajectory tensor predicted at step k, J ik represents the inverse kinematics Jacobian matrix.

[0023] In one embodiment, the system further includes a personalized appearance module for:

[0024] Map the emotion quantization parameters to the HSV color space and generate a dynamic color mapping table;

[0025] Combined with the particle swarm algorithm to simulate the physical deformation of touch response and generate dynamic texture maps based on the behavior trajectory information;

[0026] The dynamic color map is superimposed on the dynamic texture map, and the corresponding colors and textures are rendered in real time using the GPU shader to obtain the dynamic appearance of the electronic pet.

[0027] In one embodiment, the system further includes a random action module for:

[0028] Use quantum random numbers to generate random sequences and map them to the action parameters of electronic pets;

[0029] Adjust the node weights of the preset reinforcement learning decision model in real time based on action parameters and generate behavioral intention information;

[0030] Generate corresponding action instructions based on behavioral intention information.

[0031] In one embodiment, the system further includes a smart home control module for:

[0032] Obtain user behavior feature data and build a dynamic behavior pattern library;

[0033] Based on the behavior pattern library and smart home control instruction set, matching and generating the mapping relationship of environmental adjustment parameters;

[0034] Combined with the real-time behavioral feature data and the environmental adjustment parameter mapping relationship, the corresponding smart home control instructions are output.

[0035] In a second aspect, the present application also provides an artificial intelligence-based multimodal interaction method for electronic pets, comprising:

[0036] Obtain the user's physiological electrical signals, voice data, and behavioral trajectory information;

[0037] A fusion model is used to correlate and analyze the time domain features of physiological signals, the semantic features of speech, and the spatial features of behavior to generate a multi-dimensional emotion feature vector.

[0038] Based on the LSTM network, the physiological indicators, voice fundamental frequency and behavior frequency in the multi-dimensional emotion feature vector are weighted and integrated to generate emotion quantification parameters;

[0039] Based on the user's interaction intention, combined with the emotion quantification parameters and the preset dynamic interaction mechanism, the emotion interaction action instructions are generated; the emotion interaction action instructions are used to control the behavior of the electronic pet and realize dynamic interaction.

[0040] In a third aspect, the present application also provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above-mentioned artificial intelligence-based electronic pet multimodal interaction system function when executing the computer program.

[0041] In a fourth aspect, the present application also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of implementing the functions of the above-mentioned artificial intelligence-based electronic pet multimodal interaction system are implemented.

[0042] The above-mentioned artificial intelligence-based electronic pet multimodal interaction system obtains the user's physiological electrical signals, voice data and behavioral trajectory information through the data acquisition module, and the multimodal emotion analysis module uses a fusion model to correlate and analyze the time domain features, semantic features and behavioral space features of these data to generate a multidimensional emotion feature vector. The emotion state modeling module performs weighted fusion processing on physiological indicators, voice fundamental frequency and behavioral frequency based on the LSTM network to generate emotion quantification parameters. The dynamic interaction module combines the emotion quantification parameters and the preset dynamic interaction mechanism to generate emotion interaction action instructions for controlling the behavior of the electronic pet and realizing dynamic interaction. Through the multimodal interaction method, the present invention can comprehensively process the user's physiological, voice and behavioral information, enhance the depth and authenticity of emotional communication, and effectively simulate the interaction of objects in a real physical environment through the dynamic interaction mechanism, enhance the immersion and authenticity of the interactive experience, thereby solving the problem of insufficient single-modal interaction and complex scene interaction experience in the existing technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0044] Figure 1 This is a schematic diagram of the structure of an artificial intelligence-based electronic pet multimodal interaction system of the present invention;

[0045] Figure 2 This is a flow chart of an artificial intelligence-based multimodal interaction method for electronic pets according to the present invention. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0047] The present invention provides an artificial intelligence-based electronic pet multimodal interaction system, which uses terminal devices or electronic pet devices to collect the user's physiological electrical signals, voice data and behavioral trajectory information through sensors, and uploads the data to an AI server. The AI server runs a multimodal emotion analysis module and an emotion state modeling module to generate interactive action instructions, and sends them to the electronic pet device through an API interface to achieve interaction.

[0048] In one embodiment, Figure 1 As shown, a multimodal interactive system for electronic pets based on artificial intelligence is provided. This embodiment uses the system deployed on a terminal as an example. It is understandable that the system can also be deployed on a server, or on a collaborative architecture including a terminal and a server, and implemented through the interaction between the terminal and the server. In this embodiment, the system includes the following modules:

[0049] The data acquisition module 101 is used to: S01, obtain the user's physiological electrical signals, voice data and behavior trajectory information.

[0050] Wearable devices such as smart bracelets and smartwatches, or specialized sensors such as electroencephalogram (EEG) caps can be used to collect physiological electrical signals, including heart rate, brain waves, and galvanic skin response. These signals can reflect a user's emotional state, such as excitement, calmness, and tension, providing a physiological basis for sentiment analysis. Microphones or voice sensors can be used to collect user voice data for analyzing emotions such as joy, sadness, and anger, and combining semantics to understand user intent. Sensors or cameras can be used to collect user behavior trajectory information, such as using a camera to capture gestures and posture; or using sensors such as accelerometers and gyroscopes to record user movement trajectories. This behavior trajectory information can include changes in the user's position and movement path. This information can reflect user behavior patterns, such as approach, distance, and touch, providing a reference for generating interactive actions.

[0051] The multimodal emotion analysis module 102 is used to: S02, use a fusion model to correlate and analyze the time domain features of physiological signals, the semantic features of speech, and the behavioral space features to generate a multidimensional emotion feature vector.

[0052] For time-domain features of physiological signals, such as amplitude, frequency, and rate of change, can be extracted from the user's physiological electrical signals, such as heart rate and galvanic skin response. For semantic features of speech, such as phonetic meaning, emotional intonation, speaking rate, and volume, can be extracted from the user's speech data. For spatial features of behavior, such as movement paths, position changes, and posture, can be extracted from the user's behavioral trajectory information. A deep fusion model can be used to jointly model features from different modalities through a deep learning network, extracting higher-level feature representations and generating a multidimensional emotional feature vector. This vector comprehensively represents the user's emotional state and can include the following dimensions: physiological dimensions, reflecting emotional features of the user's physiological state, such as excitement and calmness; speech dimensions, reflecting emotional features of the user's speech, such as joy and sadness; and behavioral dimensions, reflecting behavioral patterns, such as approaching and withdrawing. This can comprehensively describe the user's emotional state and provide input for subsequent modules.

[0053] The emotional state modeling module 103 is used to: S03, perform weighted fusion processing on the physiological indicators, voice fundamental frequency and behavior frequency in the multi-dimensional emotional feature vector based on the LSTM network to generate emotional quantification parameters.

[0054] Among them, the LSTM (Long Short-Term Memory) network can effectively process and predict time series data. By introducing gating mechanisms such as input gates, forget gates, and output gates, it controls the flow of information and avoids the vanishing gradient problem, making it suitable for processing long time series data. Physiological indicators, speech semantics, and behavior frequency can be extracted from the multidimensional emotion feature vector, normalized, and weighted according to feature importance. For example, physiological indicators may have a greater impact on emotional state, so they are assigned a higher weight to generate emotion quantification parameters. Emotion quantification parameters are numerical values or vectors that describe the user's emotional state. They can include the following information: emotion intensity, reflecting the intensity of the user's emotion; emotion type, reflecting the user's emotion category such as joy, sadness, and anger; emotion change trend, reflecting the user's emotional changes over time; and speech semantics, reflecting the user's most direct interaction intentions. Based on these emotional state representations, the electronic pet can generate corresponding interactive actions based on the user's emotional state.

[0055] The dynamic interaction module 104 is used to: S04, generate emotional interaction action instructions based on the user's interaction intention, combined with the emotional quantification parameters and the preset dynamic interaction mechanism; the emotional interaction action instructions are used to control the behavior of the electronic pet to achieve dynamic interaction.

[0056] The dynamic interaction mechanism defines how the pet responds based on the user's interaction intent, emotional state, and environmental information. This can be done directly based on user interaction intent, such as gestures, voice text, and other information. For example, users can play with the pet through gestures like patting; interact with the pet through voice interaction phrases like greeting, stretching, looking up, and crawling down; and intelligently interact with the pet through long voice text. For example, if a user says, "Please record this voice and summarize it into a document," the corresponding interaction could be: the pet receives the corresponding voice data and sends it to the AI server, which then sends the feedback to the pet or other user terminals. Emotional quantification parameters can be combined to generate corresponding interactive action commands. For example, in a scenario where the emotional quantification parameters show high emotional intensity and joy, indicating a happy user; a stable emotional trend indicates a consistently happy user; and the interactive intent of speech semantics is to find local food recommendations, corresponding joyful emotional interactive action commands are generated based on the dynamic interaction mechanism. For example, Action 1 drives the electronic pet to wag its tail to indicate happiness; Action 2 drives the electronic pet to approach the user, gently bump its head against the user's hand, and emit cheerful sounds to indicate affection; Action 3 drives the electronic pet to output voice navigation content tailored to its current emotional state based on AI-generated food recommendations. This approach enables natural and authentic interactive actions to be generated based on the user's emotional state and environmental information, enhancing the user's interactive experience.

[0057] The present application discloses an artificial intelligence-based multimodal interaction system for electronic pets. The data acquisition module acquires the user's physiological electrical signals, voice data, and behavioral trajectory information. The multimodal emotion analysis module uses a fusion model to correlate and analyze the time domain features, semantic features, and behavioral space features of these data to generate a multidimensional emotion feature vector. The emotion state modeling module performs weighted fusion processing on the physiological indicators, voice fundamental frequency, and behavioral frequency in the multidimensional emotion feature vector based on the LSTM network to generate emotion quantification parameters. The dynamic interaction module combines the emotion quantification parameters with a preset dynamic interaction mechanism to generate emotion interaction action instructions for controlling the behavior of the electronic pet and realizing dynamic interaction. Through the multimodal interaction method, the user's physiological, voice, and behavioral information can be comprehensively processed to enhance the depth and authenticity of emotional communication. At the same time, the dynamic interaction mechanism effectively simulates the interaction of objects in a real physical environment, enhancing the immersion and authenticity of the interactive experience, thereby solving the problem of insufficient single-modal interaction and complex scene interaction experience in the existing technology.

[0058] In one embodiment, the dynamic interaction module 104 is further configured to:

[0059] S11, real-time acquisition of stereoscopic spatial data of a real scene and construction of three-dimensional point cloud features based on the stereoscopic spatial data;

[0060] S12, based on 3D point cloud features, establishes the spatial topological relationship matrix of physical objects through the spatiotemporal graph convolutional network, and combines it with the reinforcement learning strategy network to generate the interaction utility function;

[0061] S13, based on the interaction utility function, uses the inverse kinematics Jacobian matrix to generate physical interaction action instructions.

[0062] Specifically, sensors such as binocular cameras and depth sensors can be used to acquire real-time 3D spatial data of real scenes, including information such as the 3D coordinates, position, and velocity of objects in the scene. This 3D spatial data can be converted into 3D point cloud features using deep learning algorithms or traditional computer vision techniques. 3D point cloud features represent the distribution of objects in 3D space and can describe their shape, position, and motion. Processing these 3D point cloud features using a spatiotemporal graph convolutional network captures the spatial and temporal relationships between objects, dynamically models their interactions, and generates a matrix describing their spatial topological relationships. The utility of different actions can be evaluated using the interaction utility function generated by a reinforcement learning policy network. The reinforcement learning policy network learns the optimal policy through trial and error, generating an interaction utility function that is used to evaluate the action choices of the electronic pet in the current environment. A higher utility value in the interaction utility function indicates that the action is more appropriate for the current environment and the user's emotional state. The action choices in the interaction utility function are converted into specific joint motion instructions using the inverse kinematics Jacobian matrix. The inverse kinematics Jacobian matrix converts target positions and postures into joint angles, ensuring smooth and natural movements of the electronic pet. Based on the interaction utility function and the inverse kinematics Jacobian matrix, specific physical interaction action commands are generated. These commands are used to control the joint movements of the electronic pet, enabling it to interact naturally with the user and the environment. By perceiving the 3D spatial data of the real scene in real time, the spatial topological relationships between objects are established. Natural and realistic interaction action commands are generated using a reinforcement learning policy network and the inverse kinematics Jacobian matrix. This process enables the electronic pet to respond appropriately to the user's emotional state and environmental information, enhancing the immersiveness and realism of the interactive experience.

[0063] In one embodiment, the dynamic interaction module 104 is further configured to: S21, construct a spatial topological relationship matrix of entities using the following formula:

[0064]

[0065] φ(p i ,v i )=p i ·e jωt+v i ×ablaB(p i )

[0066]

[0067] f dyn (Δt)=e -λ||Δt|| ·(1+tanh(β||v rel ||))

[0068] Among them, T ij represents the spatial topological relationship matrix between items, φ(p i ,v i ) represents the dynamic position encoding parameter, ψ(p j ,v j ) represents the spatiotemporal attention kernel parameter, f dyn (Δt) represents the dynamic attenuation factor parameter, represents the three-dimensional coordinates of item i, represents the velocity vector of item i, ω represents the spatial frequency parameter, represents the magnetic field gradient field tensor, represents the three-dimensional convolution kernel, represents the attention weight matrix, Represents the pet motion trajectory tensor, λ represents the time attenuation coefficient, Δt represents the data timestamp difference, β represents the speed sensitivity coefficient, Represents the relative velocity vector.

[0069] For example, the dynamic position encoding parameter φ(p i ,v i ) The three-dimensional coordinates of the object p i and speed v i Encoded as dynamic position feature, plural encoding e jωt The periodic spatial frequency ω is introduced to capture the dynamic position change of the object, and the magnetic field gradient term v i ×ablaB(p i ) can simulate the influence of the environmental magnetic field on the object's movement and enhance the physical reality. The three-dimensional convolution Conv3D is used to extract the local spatial features of the object j, and the attention mechanism Attention (v j , o pet ) Calculate the speed of object j and the trajectory of the electronic pet o pet Relevance, focusing on key interaction objects. Time decay -λ||Δt|| The speed sensitive term tanh(β||v rel ||) According to the relative speed v relAdjust the interaction intensity to have a more significant impact on fast-moving objects. By fusing dynamic position and spatiotemporal features, dynamically adjust the relationship weights and generate a probabilistic spatial relationship matrix T. ij , highlighting key interactive objects. This solves the problems of misjudgment of collisions and awkward interactions in traditional systems.

[0070] In one embodiment, the dynamic interaction module 104 is further configured to: S31, generate a physical interaction action instruction using the following formula:

[0071] π θ (a|s)=argmax a Q π (s, a)

[0072]

[0073] Among them, θ represents the policy network parameters, a represents the action of the electronic pet, s represents the position, speed and environmental semantic information of the electronic pet, Q π (s, a) represents the state-action value function, represents the collision penalty term, η represents the collision penalty intensity coefficient, P pet Indicates the center of mass position of the electronic pet, P obj represents the center of mass of the obstacle, n obj represents the obstacle surface normal vector, represents the motion optimization term, ω motion optimization weight, represents the motion trajectory tensor predicted at step k, J ik represents the inverse kinematics Jacobian matrix.

[0074] Specifically, the policy network parameter θ determines the neural network weights of the electronic pet's action selection. Through reinforcement learning training and optimization, it can directly affect the decision quality. The electronic pet's action a is the specific behavioral instruction executed by the electronic pet, such as movement direction and speed. The data type is a multidimensional vector, such as joint angle and motor speed. The position, speed and environmental semantic information s of the electronic pet include the position P pet And environmental semantic information. It is used to describe the current state of the electronic pet's environment and is the basic input for decision-making. The collision penalty intensity coefficient η is used to control the degree of activeness of obstacle avoidance behavior. The larger the η value, the sooner the electronic pet will stay away from obstacles. pet and P obj Used to calculate the real-time distance between the pet and the obstacle, and to avoid collision with the environment entities. For example, if the distance between the two is less than 0.3 meters, an emergency stop will be triggered. The obstacle surface normal vector n obj Used to define the direction of the collision penalty. Can be perpendicular to the obstacle surface to guide the pet in a safe direction. Movement optimization weight. Represents the k-th step predicted motion trajectory tensor, which is calculated in real time by the LSTM network to predict the position deviation of the next k steps. Using the inverse kinematics Jacobian matrix J ik , mapping the end motion to the joint space to avoid mechanical singularities. Dynamic adjustment The value is used to suppress the current action to prevent the joint from getting out of control, while taking into account the smoothness of the movement of the next multiple steps.

[0075] In one embodiment, the system further includes a personalized appearance module for:

[0076] S41, mapping the emotion quantization parameter to the HSV color space to generate a dynamic color mapping table;

[0077] S42, combines the particle swarm algorithm to simulate the physical deformation of the touch reaction and generates a dynamic texture map based on the behavior trajectory information;

[0078] S43, superimposing the dynamic color mapping table and the dynamic texture map, and using the GPU shader to render the corresponding colors and textures in real time to obtain the dynamic appearance of the electronic pet.

[0079] For example, the mapping rules for mapping emotional quantification parameters to the HSV color space can be: hue H determines the primary color of the emotional type, such as red 0° represents anger and blue 240° represents sadness; saturation S represents the intensity of the emotion, controlling color purity, such as S = 30% for calm and S = 100% for excitement; and value V represents the physiological impact of brightness, such as increasing the V value by 20% when an increased heart rate is detected. A particle swarm algorithm can be used to define each particle's position, velocity, and deformation influence, and update each particle's state according to a preset update rule. When a virtual pet collides, 50 new particles can be added at the point of contact, spreading outward at an initial velocity. The particle's state in the density field is used to generate a normal map, and dynamic texture maps are generated based on the behavior trajectory information. A GPU (Graphics Processing Unit) shader is a program specifically designed for graphics processing that can be executed in parallel on the GPU, enabling efficient graphics rendering. GPU shaders can be used to process and render dynamic colors and textures in real time, quickly generating a corresponding appearance image based on the virtual pet's current emotional state and behavior trajectory. Colors and textures can be overlaid using multiplication blending or alpha blending, while also taking into account spatial alignment, such as UV coordinate synchronization, ensuring the color table and texture map share the same UV mapping coordinate system; and displacement compensation, such as applying a 0.5px to 2px coordinate offset to the deformed area to achieve the dynamic appearance of the electronic pet. For example, in a certain scene, if the user is recognized as happy and the interactive action is stroking and patting, the rendering color is bright yellow HSV (60°, 90%, 85%), the texture deformation thickness is 0.2mm uniform raised particles, the rendering performance is normal mapping to enhance hair reflections, the color brightness fluctuates by ±8% with heart rate, and the deformation animation transition time is <16ms at a display frame rate of 60fps.

[0080] In one embodiment, the system further includes a random action module for:

[0081] S51, using quantum random numbers to generate random sequences and mapping them into action parameters of the electronic pet;

[0082] S52, adjusting the node weights of the preset reinforcement learning decision model in real time based on the action parameters, and generating behavior intention information;

[0083] S53: Generate corresponding action instructions according to the behavior intention information.

[0084] Specifically, quantum random numbers are generated based on the principles of quantum mechanics. A quantum random number generator can be a chip based on the single-photon quantum tunneling effect. It generates a series of random numbers, forming a random sequence, which is then mapped to the action parameters of the electronic pet. The action parameters of the electronic pet can include movement speed, movement direction, and movement amplitude. For example, different value ranges in the random sequence can correspond to different movement speeds (fast, medium, slow), or different movement directions. This allows the electronic pet's movements to have a certain degree of randomness, making them more realistic. Reinforcement learning is a machine learning method that uses the electronic pet's interaction with the environment to learn optimal behavior strategies based on reward signals from the environment. The preset reinforcement learning decision model is typically a neural network, and the node weights in the network determine the model's decision-making behavior. The node weights of the reinforcement learning decision model are adjusted in real time based on the action parameters. The action parameters can serve as external interference or guidance to change the model's learning direction. For example, if the action parameters indicate that the electronic pet is trying a new movement method, the model's node weights will be adjusted accordingly, making subsequent decisions more likely to choose actions related to this movement method. After the node weights are adjusted, the reinforcement learning decision model generates behavioral intention information based on the current environmental state and the adjusted weights. Behavioral intention information can be understood as what the electronic pet intends to do, such as exploring a new area or approaching an object. It provides an abstract description of the electronic pet's next move. This behavioral intention information is converted into specific action instructions. For example, if the behavioral intention information is to explore a new area, the corresponding action instruction might be to move forward at a certain speed and randomly change direction. Based on the electronic pet's current state, environmental information, and behavioral intention information, specific, executable action instructions are generated to control the electronic pet to perform the corresponding actions. The introduction of randomized actions increases the fun and realism of the electronic pet. Furthermore, by combining it with a reinforcement learning decision model, the electronic pet's random behaviors can interact and learn from the environment to a certain extent, further enriching its behavioral patterns.

[0085] In one embodiment, the system further includes a smart home control module for:

[0086] S61, obtaining user behavior feature data and building a dynamic behavior pattern library;

[0087] S62, based on the behavior pattern library and the smart home control instruction set, matching and generating the mapping relationship of the environment adjustment parameters;

[0088] S63: Combine the real-time behavior feature data and the environmental adjustment parameter mapping relationship to output corresponding smart home control instructions.

[0089] Specifically, various sensors can be used to obtain user behavioral data. For example, cameras can be used to identify a user's location, movements, and posture; infrared sensors can be used to detect whether a user is in a room; and smart door locks can be used to record the user's entry and exit times. This data can reflect the user's behavioral characteristics at different times and in different scenarios. The collected behavioral data is analyzed and organized to identify representative behavioral patterns. For example, after waking up in the morning, users typically open the curtains and turn on the lights; before going to bed at night, they turn off appliances and dim the lights. This behavioral pattern library is not static but dynamic, constantly updated and improved as user behavior changes. For example, if a user develops a new habit of taking a bath before bed, this new behavior pattern will be added to the library. The smart home control command set includes control commands for various home devices, such as controlling the temperature and fan speed of the air conditioner, adjusting the opening and closing of curtains, controlling the brightness and color of lights, and controlling the user's favorite songs on the speakers. Each command corresponds to a specific environmental adjustment parameter. The behavioral patterns in the behavior pattern library are matched with the smart home control command set. When a certain behavior pattern is detected, the system determines which control commands and corresponding environmental adjustment parameters need to be executed. For example, if the behavior pattern library records that a user prefers to set the room lights to 30% brightness when reading at night, then when the system detects that the user has entered this behavior pattern, it will match the control command to adjust the light brightness to 30%, thereby establishing a mapping relationship between the behavior pattern and the environmental adjustment parameters. The electronic pet continuously collects the user's current behavioral characteristic data to determine the user's current behavior pattern. Based on this real-time behavioral characteristic data, it finds a matching behavior pattern from the behavior pattern library and then outputs the corresponding smart home control command based on the environmental adjustment parameter mapping relationship. For example, if the system detects that the user is reading on the sofa in the living room at night, combined with the behavior pattern library and the mapping relationship, it will output a control command to adjust the living room light brightness to 30% and send it to the smart lighting for control. At the same time, the user's physiological electrical signals, such as abnormal heart rate fluctuations, voice data such as distress keywords, behavioral trajectories such as fall posture, or dynamic behavior patterns such as prolonged inactivity, are mapped to environmental adjustment parameters to generate alarm commands. For example, these can include playing exercise rehabilitation guidance audio through a smart speaker; vibrating a health bracelet to issue a health alert; a mobile app sending a location-based rescue message to a preset contact; a smartwatch automatically dialing an emergency number; and driving an electronic pet to perform soothing actions while simultaneously displaying the alarm status. By collecting, analyzing, and processing user behavioral feature data, the system automatically adjusts the home environment based on user behavior, transforming the electronic pet into not only an emotional companion device but also a smart interactive center for the home, providing users with a comprehensive intelligent experience ranging from emotional support to daily life services.

[0090] The present application discloses an artificial intelligence-based multimodal interaction system for electronic pets, which effectively solves the technical problems of monotonous interaction, distorted emotional feedback and scene fragmentation of traditional electronic pets through the deep integration of multimodal emotional computing and dynamic interaction mechanism. Specifically, the data acquisition module obtains the user's physiological electrical signals, voice data and behavioral trajectories, and generates a multi-dimensional emotional feature vector through fusion analysis by the multimodal emotional analysis module. The emotional quantification parameters are then weighted by the LSTM network, and finally the dynamic interaction module generates emotional interaction action instructions based on the user's intention. The system constructs three-dimensional point cloud features and spatial topological relationship matrices through a spatiotemporal graph convolutional network, generates interactive utility functions in combination with reinforcement learning strategies, and optimizes action instructions using inverse kinematics Jacobian matrices to achieve natural interaction in complex scenarios. The personalized appearance module uses HSV color mapping and particle swarm deformation simulation to render dynamic appearance in real time; the random action module introduces a quantum random number-driven reinforcement learning model to enhance behavioral diversity; the smart home control module realizes intelligent adjustment of environmental parameters based on a dynamic behavior pattern library. The above innovative design significantly enhances the emotional interaction depth, physical environment adaptability and home scene linkage efficiency of electronic pets, providing users with an immersive interactive experience that combines emotional companionship and intelligent service functions.

[0091] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0092] Based on the same inventive concept, the present application also provides a method for implementing the aforementioned artificial intelligence-based multimodal interaction method for electronic pets. The solution provided by this method is similar to the solution described in the aforementioned artificial intelligence-based multimodal interaction system for electronic pets. Therefore, the specific limitations of one or more embodiments of the artificial intelligence-based multimodal interaction method for electronic pets provided below can be found in the aforementioned limitations of the artificial intelligence-based multimodal interaction system for electronic pets, and will not be further elaborated here.

[0093] In an exemplary embodiment, Figure 2 As shown, a multimodal interaction method for electronic pets based on artificial intelligence is provided, comprising:

[0094] S1, obtains the user's physiological electrical signals, voice data, and behavior trajectory information;

[0095] S2 uses a fusion model to correlate and analyze the time domain features of physiological signals, the semantic features of speech, and the spatial features of behavior to generate a multidimensional emotion feature vector;

[0096] S3, based on the LSTM network, performs weighted fusion of physiological indicators, voice fundamental frequency, and behavior frequency in the multi-dimensional emotion feature vector to generate emotion quantification parameters;

[0097] S4, based on the user's interaction intention, combined with the emotion quantification parameters and the preset dynamic interaction mechanism, generates emotion interaction action instructions; the emotion interaction action instructions are used to control the behavior of the electronic pet and realize dynamic interaction.

[0098] In one embodiment, the method further comprises:

[0099] Acquire the stereoscopic spatial data of real scenes in real time and construct 3D point cloud features based on the stereoscopic spatial data;

[0100] Based on 3D point cloud features, a spatial topological relationship matrix of physical objects is established through a spatiotemporal graph convolutional network, and an interactive utility function is generated by combining it with a reinforcement learning strategy network.

[0101] Based on the interaction utility function, the inverse kinematics Jacobian matrix is used to generate physical interaction action instructions.

[0102] In one embodiment, the method further comprises constructing a spatial topological relationship matrix of the entity using the following formula:

[0103]

[0104] φ(p i ,v i )=p i ·e jωt +v i ×ablaB(p i )

[0105]

[0106] f dyn (Δt)=e -λ || Δt ||·(1+tanh(β||v rel ||))

[0107] Among them, T ij represents the spatial topological relationship matrix between items, φ(p i ,v i) represents the dynamic position encoding parameter, ψ(p j ,v j ) represents the spatiotemporal attention kernel parameter, f dyn (Δt) represents the dynamic attenuation factor parameter, represents the three-dimensional coordinates of item i, represents the velocity vector of item i, ω represents the spatial frequency parameter, represents the magnetic field gradient field tensor, represents the three-dimensional convolution kernel, represents the attention weight matrix, Represents the pet motion trajectory tensor, λ represents the time attenuation coefficient, Δt represents the data timestamp difference, β represents the speed sensitivity coefficient, Represents the relative velocity vector.

[0108] In one embodiment, the method further includes generating a physical interaction action instruction using the following formula:

[0109] π θ (a|s)=argmax a Q π (s, a)

[0110]

[0111] Among them, θ represents the policy network parameters, a represents the action of the electronic pet, s represents the position, speed and environmental semantic information of the electronic pet, Q π (s, a) represents the state-action value function, represents the collision penalty term, η represents the collision penalty intensity coefficient, P pet Indicates the center of mass position of the electronic pet, P obj represents the center of mass of the obstacle, n obj represents the obstacle surface normal vector, represents the motion optimization term, ω motion optimization weight, represents the motion trajectory tensor predicted at step k, J ik represents the inverse kinematics Jacobian matrix.

[0112] In one embodiment, the method further comprises:

[0113] Map the emotion quantization parameters to the HSV color space and generate a dynamic color mapping table;

[0114] Combined with the particle swarm algorithm to simulate the physical deformation of touch response and generate dynamic texture maps based on the behavior trajectory information;

[0115] The dynamic color map is superimposed on the dynamic texture map, and the corresponding colors and textures are rendered in real time using the GPU shader to obtain the dynamic appearance of the electronic pet.

[0116] In one embodiment, the method further comprises:

[0117] Use quantum random numbers to generate random sequences and map them to the action parameters of electronic pets;

[0118] Adjust the node weights of the preset reinforcement learning decision model in real time based on action parameters and generate behavioral intention information;

[0119] Generate corresponding action instructions based on behavioral intention information.

[0120] In one embodiment, the method further comprises:

[0121] Obtain user behavior feature data and build a dynamic behavior pattern library;

[0122] Based on the behavior pattern library and smart home control instruction set, matching and generating the mapping relationship of environmental adjustment parameters;

[0123] Combined with the real-time behavioral feature data and the environmental adjustment parameter mapping relationship, the corresponding smart home control instructions are output.

[0124] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the functions of the aforementioned artificial intelligence-based electronic pet multimodal interaction system are realized.

[0125] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0126] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the components described as separate parts may or may not be physically separated, and the parts displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the disclosed solution. A person of ordinary skill in the art can understand and implement it without expending creative work.

[0127] The above-described embodiments merely represent several implementation methods of the embodiments of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that a person skilled in the art may make various modifications and improvements without departing from the concept of the embodiments of the present application, and these modifications and improvements fall within the scope of protection of the embodiments of the present application.

Claims

1. An artificial intelligence-based electronic pet multimodal interaction system, characterized in that: The system comprises: Data acquisition module, used to obtain the user's physiological electrical signals, voice data and behavioral trajectory information; Multimodal emotion analysis module, which uses a fusion model to correlate and analyze the time domain features of physiological signals, the semantic features of speech, and the spatial features of behavior to generate a multidimensional emotion feature vector; An emotional state modeling module is used to perform weighted fusion processing on the physiological indicators, voice fundamental frequency and behavior frequency in the multidimensional emotional feature vector based on the LSTM network to generate emotional quantification parameters; The dynamic interaction module is used to generate emotional interaction action instructions based on the user's interaction intention, combined with the emotional quantification parameters and the preset dynamic interaction mechanism; the emotional interaction action instructions are used to control the behavior of the electronic pet to achieve dynamic interaction.

2. The system according to claim 1, wherein: The dynamic interaction module is also used for: Acquire stereoscopic spatial data of a real scene in real time and construct three-dimensional point cloud features based on the stereoscopic spatial data; Based on the three-dimensional point cloud features, a spatial topological relationship matrix of the physical object is established through a spatiotemporal graph convolutional network, and an interactive utility function is generated by combining it with a reinforcement learning strategy network; Based on the interaction utility function, the inverse kinematics Jacobian matrix is used to generate physical interaction action instructions.

3. The system according to claim 2, characterized in that The dynamic interaction module is further configured to construct a spatial topological relationship matrix of entities using the following formula: φ(p i ,v i )=p i ·and jωt +v i ×hereB( p i ) f dyn (Δt)=e -λ || Δt ||·(1+tanh(β||v rel ||)) Among them, T ij represents the spatial topological relationship matrix between items, φ(p i ,v i ) represents the dynamic position encoding parameter, ψ(p j ,v j ) represents the spatiotemporal attention kernel parameter, f dyn (Δt) represents the dynamic attenuation factor parameter, represents the three-dimensional coordinates of item i, represents the velocity vector of item i, ω represents the spatial frequency parameter, represents the magnetic field gradient field tensor, represents the three-dimensional convolution kernel, represents the attention weight matrix, Represents the pet motion trajectory tensor, λ represents the time attenuation coefficient, Δt represents the data timestamp difference, β represents the speed sensitivity coefficient, Represents the relative velocity vector.

4. The system according to claim 2, wherein: The dynamic interaction module is further configured to generate physical interaction action instructions using the following formula: π θ (a|s)=argmax a Q π (s,a) Among them, θ represents the policy network parameters, a represents the action of the electronic pet, s represents the position, speed and environmental semantic information of the electronic pet, Q π (s, a) represents the state-action value function, represents the collision penalty term, η represents the collision penalty intensity coefficient, P pet Indicates the center of mass position of the electronic pet, P obj represents the center of mass of the obstacle, n obj represents the obstacle surface normal vector, represents the motion optimization term, ω motion optimization weight, represents the motion trajectory tensor predicted at step k, J ik represents the inverse kinematics Jacobian matrix.

5. The system according to claim 1, wherein: The system further includes a personalized appearance module for: Mapping the emotion quantization parameter to the HSV color space to generate a dynamic color mapping table; Combined with the particle swarm algorithm to simulate the physical deformation of the touch reaction, and generate a dynamic texture map based on the behavior trajectory information; The dynamic color mapping table is superimposed on the dynamic texture map, and the corresponding colors and textures are rendered in real time using a GPU shader to obtain a dynamic appearance of the electronic pet.

6. The system according to claim 1, wherein: The system further includes a random action module for: Use quantum random numbers to generate random sequences and map them to the action parameters of electronic pets; Adjusting the node weights of a preset reinforcement learning decision model in real time based on the action parameters and generating behavioral intention information; Generate corresponding action instructions according to the behavior intention information.

7. The system according to claim 1, wherein: The system also includes a smart home control module for: Obtain user behavior feature data and build a dynamic behavior pattern library; Based on the behavior pattern library and the smart home control instruction set, matching and generating an environment adjustment parameter mapping relationship; Combined with the real-time mapping relationship between the behavioral feature data and the environmental adjustment parameters, the corresponding smart home control instructions are output.

8. An artificial intelligence-based multimodal interaction method for electronic pets, characterized in that: The method comprises: Obtain the user's physiological electrical signals, voice data, and behavioral trajectory information; A fusion model is used to correlate and analyze the time domain features of physiological signals, the semantic features of speech, and the spatial features of behavior to generate a multi-dimensional emotion feature vector. Based on the LSTM network, the physiological indicators, voice fundamental frequency and behavior frequency in the multidimensional emotion feature vector are weighted and integrated to generate emotion quantization parameters; Combining the emotion quantification parameters with a preset dynamic interaction mechanism, an emotion interaction action instruction is generated; the emotion interaction action instruction is used to control the behavior of the electronic pet to achieve dynamic interaction.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the functions of the system according to any one of claims 1 to 7 are realized.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the functions of the system according to any one of claims 1 to 7 are realized.

Images