VEM-robot craniofacial end-to-end emotion bionic system

Abstract

The VEM-robot craniofacial end-to-end emotion bionic system disclosed by the invention comprises hardware of bionic eyeballs, bionic skin, a bionic mouth, bionic ears and a bionic neck, and realizes real-time and anthropomorphic direct interaction between a robot and a real person under the management support of the end-to-end emotion bionic system, end-to-end emotional simulation is realized from the input ends of binaural hearing, binocular vision, face touch, lip touch, environmental perception and emergency capture to the output ends of binocular expression, skull flexibility, emotional sound, mandibular flexibility, face expression and mouth expression, and intervention of a third party such as a remote controller or background manual work is not needed. According to the system, methods of expression self-learning, interaction object long-time-sequence interaction, memory personalized emotion adaptation, hyperparallel comparison emotion calculation and the like are adopted, an entity emotion craniofacial capable of recognizing people, simulating human expressions, crossing interaction and personalized emotion interaction is achieved, and a multi-mode emotion bionic head is provided for an existing humanoid robot body.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, specifically to the subfield of emotion computing and emotion realization in artificial intelligence, and particularly to the subfield of multimodal craniofacial emotion bionic system integration for humanoid robots and embodied robots, aiming to provide a simulated robot head and face with expressions and emotions. Background Technology

[0002] With the development of the humanoid robot and android industry, the innovation of various motion components in robot limbs has enabled robots to gain widespread application and recognition. However, in terms of robot heads, especially humanoid heads, no suitable technology and manufacturing solutions have been introduced to date. In fact, this predicament not only hinders the development and application of anthropomorphic humanoid robots, but also provides a real opportunity for in-depth innovation in robots.

[0003] Based on the research conducted by the invention team, the most advanced technological solutions related to robotic craniofacial structures worldwide include: 1. Servo motor drive mode This is a robot head made using "Sophia" technology invented by Hanson Robotics. It uses 60 servo motors, giving it 60 degrees of freedom (DOF), arranged within the frame of the robot head, and is covered by a "Frubber" headgear—a specially made elastomer-like biomimetic rubber material. Facial expressions are generated by the servo motors, and some typical expressions can be preset. In terms of sensory interaction, expressions are primarily generated by a remote control operated by a person in the background, without any real emotional interaction.

[0004] 2. Pneumatic pump drive mode This is the "Erica" ​​technology invented by Hiroshi Ishiguro's laboratory at Osaka University. It uses 10 pneumatic actuators, also called "Erica," driven by pneumatic pumps to drive multiple preset facial expressions. The pneumatic pumps and actuators are arranged in the frame of the robot's head, which is covered by an "Erica" ​​headgear—a specially made elastomeric biomimetic rubber material. The facial expressions are generated using preset expressions. In terms of sensory interaction, the expressions are primarily generated by a remote control operated by a person in the background, without any real emotional interaction.

[0005] 3. The main limitations of Sophia and Erica The "performance" aspect of the expressions outweighs their "emotional" aspect: the expressions are pre-programmed or triggered, lacking the natural generation and transition based on the internal continuous emotional state. The changes in expressions sometimes appear abrupt or repetitive. What is even more unacceptable is that even during performances, the output of robot expressions is mostly generated by back-end operators using remote controls.

[0006] The disconnect between perception and expression: There is no deep, model-based emotional state mapping between the user's visually recognized emotions and their own generated facial expressions. It is more of a simple mapping of "recognition - recalling preset expressions".

[0007] Lack of tactile and physical interaction capabilities: The skin is a purely passive, flexible covering layer without any integrated tactile sensors, making it unable to sense touch, pressure, or temperature. The robot cannot respond to physical contact.

[0008] The system is closed and expensive: the highly customized mechanical and software system is difficult to replicate and use in other research, and the maintenance costs are extremely high.

[0009] 4. VEM-Token Model Technology This invention team has for the first time proposed a set of novel and innovative VEM (Vocal-Emotion-Multimodal) technologies, comprising a patent pool consisting of the following six already granted invention patents: 4.1 VEM-Token Vocal Emotion Multimodal Model This refers to the model in "VEM-Token Vocal Emotion Multimodal Tokenization Deep Learning Method for Singing and Accompaniment, CN120126506B", which specifically includes the following steps and methods 1-4: (1) Use one or more modalities to record emotions, label vocal emotion multimodalities as VEM, construct VEM classification, VEM coordinate system, VEM function and VEM library. Vocal emotions include one or a combination of joy, sadness, anger, fear, disgust, surprise, calmness, expectation, trust, love, hate, affection and enmity. Multimodalities include one or a combination of lyrics, singing, accompaniment, vocal style, music, emotional basis, accompanying instruments, video and image. The VEM coordinate system includes a coordinate axis system established based on independent emotions, opposite emotion pairs and related opposite emotion groups.

[0010] (2) Collect vocal samples according to VEM classification, and have human vocal experts evaluate the vocal samples in terms of emotion in singing and accompaniment. Use supervised learning and deep learning to train the VEM function to obtain VEM parameters and add them to the VEM library.

[0011] (3) The VEM processor is used to calibrate the rhythm of the vocal file and separate the vocal stream and the accompaniment stream. The vocal file is split into VEM-Tokens according to the rhythm, the vocal stream is converted into a VEM-Token1 sequence, the accompaniment stream is converted into a VEM-Token2 sequence, and added to the preprocessing library.

[0012] (4) Using deep learning, the dialogue expression chart, lyrics chart, VEM-Token song chart, VEM-Token accompaniment chart, and VEM-Token music score are generated respectively.

[0013] 4.2 Model for VEM-Token Beat Capture and Beat Alignment This refers to the model in "Method for Constructing VEM-Token Beat Capture and Alignment Model, CN120748450B", which specifically includes the following steps and methods (5-7): (5) For vocal files, based on the VEM-Token vocal emotion multimodal model, a beat model including beat capture and beat alignment is set to capture the beat of the vocal file. Based on the beat, the vocal file is divided into VEM-Token sequences, and the starting point and ending point of the beat in each VEM-Token are marked.

[0014] (6) Set the starting point alignment model, including: The vocal files, including the sample files and the user files generated by the user imitating the sample files, are divided into VEM-Token1 sequences and VEM-Token2 sequences, respectively. Based on the starting point of each VEM-Token1, a starting point fine-tuning step is used to adjust the starting point of the corresponding VEM-Token2 one by one, so that it is aligned with the starting point of the corresponding VEM-Token1.

[0015] For the dialogue segments and rhythms included in the voice dialogue files in social interactions, taking the first segment as a reference, starting from the second segment, a starting point fine-tuning step is adopted to adjust the starting point of each VEM-Token in each segment one by one, so as to align with the starting point of the VEM-Token at the corresponding position in the first segment, until all loop segments are completed.

[0016] (7) Set the endpoint alignment model, including: Based on the endpoint of each VEM-Token1, an endpoint fine-tuning step is used to adjust the endpoint of the corresponding VEM-Token2 one by one, so that it is aligned with the endpoint of the corresponding VEM-Token1.

[0017] For each segment of the dialogue, taking the first segment as a reference, starting from the second segment, a fine-tuning step is used to adjust the endpoint of each VEM-Token in each segment one by one, so that it is aligned with the endpoint of the VEM-Token at the corresponding position in the first segment, until all loop segments are completed.

[0018] 4.3 VEM-Token Vocal Emotion Multimodal Modified Model This refers to the model in "Construction Method of VEM-Token Vocal Emotion Multimodal Modification Model, CN120853611B", which specifically includes the following 8 and 9 steps and methods: (8) Collect sample files and user files. Based on the VEM-Token model, use beat capture and VEM-Token segmentation to obtain the VEM-Token1 sequence of the sample files and the VEM-Token2 sequence of the user files respectively. Based on the VEM-Token1 sequence, use beat alignment for all VEM-Token2 sequences to generate the VEM-Token2 sequence.

[0019] (9) Based on the VEM parameters included in the VEM-Token model, identify the VEM parameters of the VEM-Token1 sequence, determine the modification scheme by the user, process the VEM parameters of the VEM-Token2 sequence, and modify and generate the VEM-Token2 sequence that matches the style of the VEM-Token1 sequence.

[0020] 4.4 A Layered Fusion Method for VEM-Token Sentiment Synchronization Functions This refers to the model in "A Method for Layered Fusion of VEM-Token Sentiment Synchronization Function, CN120913602B", which specifically includes the following steps and methods: 10, 11, and 12. (10) The VEM-Token vocal emotion multimodal model is adopted. The song file is divided into VEM-Token sequences in units of beats. The emotion synchronization function is set as VEM-sync. VEM-sync includes a sequence of VEM-sync vectors that correspond to and are aligned with the VEM-Token sequence. The VEM-sync vector includes synchronization content and a synchronization pointer pointing to the corresponding VEM-Token beat.

[0021] (11) Based on the VEM-Token sequence, calculate and obtain the synchronization content of the VEM-sync vector sequence. The synchronization content includes beat attributes and emotion attributes, where the emotion attributes include more than one emotion name, emotion value, and emotion weight.

[0022] (12) The emotion weight function includes: using hierarchical processing of song files to obtain the corresponding VEM-sync vector sequence, and using one or a combination of multi-layer weighted scanning, recurrent neural network, long short-term memory network, self-attention mechanism and retrieval enhancement to generate forward propagation, backward propagation and omnidirectional propagation to obtain the emotion synchronization function with the song file.

[0023] 4.5 VEM-Token World Model Robot Expression Functions This refers to the "Method for Constructing Expression Functions of VEM-Token World Model Robots, CN120951102A", which specifically includes the following steps 13 and 14: (13) Constructing the VEM-fe expression function model, including: synthesizing one or more expression vectors representing emotional input into the output synthetic expression, aligning the expression vectors in the current beat, eliminating expression conflicts between two or more expression vectors, and transitioning from the synthetic expression of the current beat to the synthetic expression of the next beat, so that the robot can understand emotions and learn expressions.

[0024] (14) The robot uses a physical expression generator or an animated expression generator to form an expression display. The expression driving model is connected by the VEM-fe expression function model to transmit more than one synthetic expression and display facial expressions.

[0025] 4.6 Model Construction Method for VEM-4D Dynamic Skin Structure and Dynamic Expressions This refers to the "VEM-4D Dynamic Skin Structure and Dynamic Expression Model Construction Method, CN121437772A", which specifically includes the following 15 to 18 steps and methods: (15) The VEM-4D dynamic skin includes a 4D dynamic skin and a driving module and a 4D expression model, wherein: (16) The 4D dynamic skin includes two or more electromagnets. The electromagnets include parallel electromagnets composed of soft magnetic materials and excitation coils. The parallel electromagnets are arranged side by side at interval A. The front of the 4D dynamic skin includes a skin that is adhered to the magnetic poles on the front of the parallel electromagnets. The skin is made of an elastic and stretchable membrane.

[0026] (17) The drive module applies an excitation current to the excitation coil, which generates an attractive or repulsive force between the magnetic poles at both ends of the electromagnet, thereby driving the skin to produce expansion and contraction deformation.

[0027] (18) The VEM-4D dynamic skin also includes a 4D expression model, which provides static and dynamic expression instructions to the driving module to drive the skin to stretch and deform.

[0028] 4.7 VEM - Multimodal Emotional Skin System This refers to the "VEM-Multimodal Emotional Skin System, application number 202610065064.2", which specifically includes the following steps and methods (19 to 23): (19) VEM - Multimodal Emotional Skin System, including electromagnets, soft magnetic colloids and multimodal mutual inductance system.

[0029] (20) An electromagnet includes a soft magnetic core and an excitation coil. The excitation current passes through the excitation coil to generate magnetic poles in the electromagnet. Two or more electromagnets are arranged at intervals to form an electromagnet array. The intervals are filled with soft magnetic colloid to form a mutual inductance magnetic circuit, which constitutes the emotional skin.

[0030] (21) Soft magnetic colloid is a mixture of soft magnetic material particles and elastic colloid, and has the resilience to restore its original shape when static.

[0031] (22) The multimodal mutual inductance system receives and executes the expression command, provides excitation current to the electromagnet, and generates magneto-induced peristalsis on the epidermis of the emotional skin to simulate the expression.

[0032] (23) The multimodal mutual inductance system also has a touch sensing function. When an external object applies a touch action on the surface, the touch action is converted and calculated to form a touch signal.

[0033] 4.8 A method and system for hyperparallel alignment This refers to "A Method and System for Hyperparallel Alignment, CN110647665B" and "A Method and System for Hyperparallel Alignment, CN114780805B", specifically including the following 24 to 26 steps and methods: (24) The comparison unit is composed of table entries and key entries, and the key entry data and the table entry row data in the table entries are compared in a super-parallel manner.

[0034] (25) Based on the width of the table entry, and based on the comparison algorithm, the substring is processed by including a mask bit or a specified bit to generate one or more selection options of the substring, and the one or more selection options of the substring are input into the table entry in combination; based on the width of the substring, the parent string is cyclically shifted and truncated and input into the keyword item, so that the parent string is compared with the table entry in the form of a data stream on the keyword item, and the result is output.

[0035] (26) The comparison unit is composed of table entries and key entries, and the key entry data and the table entry row data in the table entries are compared in a super-parallel manner.

[0036] (27) Input the parent string into the table entry, input the substring into the key item, and compare and output the results in a loop.

[0037] Insufficiency of existing technical methods (1) Existing technical solutions cannot solve the problem of craniofacial emotion bionics in multimodal robots.

[0038] (2) Unable to solve the problem of delicate facial expression output and touch sensitivity of the robot's cranium.

[0039] (3) It cannot solve end-to-end multimodal emotional interaction. Summary of the Invention

[0040] To address the shortcomings of existing technologies, our team proposes a novel “VEM-Multimodal Robotic Craniofacial Emotional Bionic System”. This solution integrates emotion-based multimodal technology to achieve system integration of the robotic craniofacial structure.

[0041] The purpose and intent of this invention are achieved by the following structure and method: 1. VEM-Robotic Craniofacial End-to-End Emotional Bionic System This invention, as a VEM-robot craniofacial end-to-end emotional bionic system, includes, but is not limited to, the following systems: Including robotic craniofacial structures and end-to-end emotional bionic systems, among which: The robotic craniofacial system includes bionic eyeballs, bionic skin, a bionic mouth, bionic ears, and a bionic neck mounted on a craniofacial support.

[0042] The end-to-end emotional bionic system captures visual, auditory, lip-touch, and skin-touch input environmental perception information from bionic eyes, bionic ears, bionic mouth, and bionic skin. Based on the environmental perception information, it calculates and generates emotional bionic facial expression commands, which are output from bionic eyes, bionic skin, bionic mouth, bionic ears, and bionic neck. This results in eye movements, eyelid movements, facial skin movements, sound output, jaw movements, mouth movements, and neck movements, achieving end-to-end bionic facial expression output from input to output.

[0043] 2. Bionic Eyeball Based on the aforementioned basic solution, the bionic eyeball of the present invention includes, but is not limited to, a combination of one or more of the following systems: A bionic eyeball includes an eyeball assembly and an eye movement mechanism, specifically including: The eye assembly includes a cornea, iris, pupil, lens, and image sensor arranged in sequence, with their planar optical centers aligned so that the optical centers coincide with the central optical axis. The image sensor collects visual perception information from the bionic eye, where the central optical axis is located in the visual macula. The eye movement mechanism includes a stepper motor and a drive.

[0044] An eye movement mechanism that connects two bionic eyeballs using an eyeball control module drives the eyeball assemblies of both eyes to rotate synchronously in a two-dimensional plane perpendicular to the central optical axis, so that the central optical axis of the eyeball assemblies of both eyes points synchronously in the same direction within the field of view, thus achieving binocular synchronization.

[0045] Preferably, the eye control module includes the connection and control of the lens and image sensor to realize automatic focusing, automatic shutter and image acquisition.

[0046] 3. Bionic Eyeball II Based on the aforementioned solutions, this invention, in terms of eyeball bionic functions and eyelid control functions, includes, but is not limited to, one or more combinations of the following: The bionic eyeball also includes bionic eyeball expression commands and eyelid control expression commands, specifically including: The end-to-end emotional bionic system controls the eyeball control module through bionic eyeball expression commands, achieving control including visual positioning, attention capture, and eyelid linkage, including the upper and lower eyelids.

[0047] Visual positioning includes: marking the axis perpendicular to the center of the robot's eyes as the forward axis, marking the macula of the bionic eyeball as the coordinate 0 point, marking the horizontal angle between the central optical axis and the forward axis as the azimuth angle, and marking the vertical angle between the central optical axis and the forward axis as the pitch angle. The end-to-end emotional bionic system transmits the azimuth and pitch angles to the eyeball control module through bionic eyeball expression commands, drives the movement of the bionic eyeball, achieves visual positioning, and the image at the visual positioning point falls on the macula.

[0048] Attention capture includes: The end-to-end emotional bionic system uses images of interactive objects, images of objects that suddenly enter the visual field, and pre-set images to be captured captured by the bionic eyeballs as sudden events, generates bionic eyeball expression commands, uses the azimuth and pitch angles of the sudden events as visual positioning directions, submits them to the eyeball control module, and executes visual positioning.

[0049] The eyelid linkage includes: the end-to-end emotional bionic system controls the eyelids by controlling facial expressions through eyelid control commands. The eyelids are made of bionic skin. The end-to-end emotional bionic system controls the eyelids to achieve linkage with the bionic eyeballs for emotional bionic expressions including looking straight ahead, looking sideways, looking up, looking down, ignoring, and rolling the eyes.

[0050] Preferably, visual positioning includes setting positioning calibration.

[0051] 4. Bionic skin Based on the aforementioned solutions, this invention, in the field of bionic skin, includes, but is not limited to, one or more combinations of the following: Bionic skin includes a multimodal mutual sensing system, specifically including: The multimodal sensing system connects to and executes facial expression commands issued by the end-to-end emotional bionic system, driving the bionic skin to produce peristalsis to simulate facial expressions.

[0052] When an external object applies a touch to the bionic skin, it triggers the bionic skin's attention capture, perceives the sudden event, and the multimodal mutual sensing system converts the touch action into a touch signal that inputs the sensory information, which is then input to the end-to-end emotional bionic system.

[0053] Bionic skin includes the lower eyelid, upper eyelid, brow bone, cheekbone and cheeks, lips and tongue.

[0054] Preferably, the lower eyelid includes a lacrimal gland, and the cheek or tongue includes a salivary gland. The lacrimal gland and salivary gland are controlled by an end-to-end emotional bionic system. The lacrimal gland and salivary gland include a micro-water supply system. The lacrimal gland water supply outlet is located on the inner side of the lower eyelid, and the salivary gland water supply outlet is located on the inner side of the cheek or the underside of the tongue. The water supply system includes a water bag, a water pump, a water pipe, a water outlet, and a water pump control system.

[0055] The end-to-end emotional bionic system generates tears and saliva through tear gland expression commands and salivary gland expression commands.

[0056] The end-to-end emotional bionic system generates expression commands based on the placement of the bionic skin through a multimodal mutual sensing system, and outputs simulated expressions on the bionic skin.

[0057] Preferably, the bionic skin also includes bionic facial color, which includes, under the connection and control of an end-to-end emotional bionic system, locally changing the color of the bionic skin based on facial expression commands through touch sensing, sound content recognition, visual content recognition and emotional bionic facial color rules.

[0058] 5. Bionic mouth Based on the aforementioned solutions, this invention, in the field of bionic skin, includes, but is not limited to, one or more combinations of the following: The bionic mouth also includes a bionic jaw, specifically including: The bionic mandible includes a mandibular pivot, a mandibular motion mechanism, and bionic skin. The bionic mandible is connected to and controlled by an end-to-end emotional bionic system. Based on perceptual information and mandibular motion rules, it outputs bionic facial expressions through mandibular expression commands. The bionic skin of the mandible is made of bionic skin, including the lower half of the cheek, the interconnected lower lip and upper lip, and forms a bionic oral cavity.

[0059] Mouth-related bionics includes lower cheek bionics, jaw movement bionics, and lip-like facial expressions.

[0060] Lip bionic expressions also include sensory input of lip touch movements, including the perception of both active robot touch and passive touch applied from the outside.

[0061] Preferably, the bionic mandibular inner oral cavity includes a bionic vocal cord, which includes an amplifier and a speaker. The frequency range covers at least 100Hz to 4000Hz, the speaker power is not less than 1W, and the speaker volume is adjustable. The bionic vocal cord is connected to and receives control from an end-to-end emotional bionic system. Based on the perception information and vocal content, it outputs vocal content, vocal expression, and mandibular movement through vocal expression commands.

[0062] The bionic mandibular inner oral cavity includes a bionic tongue, with peristaltic movements including tongue curling, tongue protrusion, and tongue flattening.

[0063] Preferably, the bionic mandibular inner oral cavity includes bionic teeth.

[0064] Preferably, the mouth-feature bionics includes an air pump, air tube, air vent, and oral cavity interface to simulate inhalation and exhalation.

[0065] 6. Bionic ears Based on the aforementioned solutions, the present invention, in terms of the bionic mouth including the bionic mandible, includes, but is not limited to, one or more combinations of the following: The bionic ear uses a pickup and amplifier installed near the middle ear of both ears, connected to and controlled by an end-to-end emotional bionic system, to perceive sound content and capture attention to sound.

[0066] By using binaural sound source localization, the system can perceive the location of sounds and capture attention to those sounds, thus achieving biomimetic dialogue attention.

[0067] It can listen to sudden sound sources, sense and capture attention, and work with bionic eyes and a bionic neck to achieve attention transfer.

[0068] Preferably, the microphone includes a sound source localization mechanism that reflects sound through the contour structure of the ear.

[0069] Preferably, attention capture includes: the end-to-end emotional bionic system taking the sounds captured by the bionic ear, including the sounds of interactive objects, sudden auditory intrusions, and pre-set sounds to be captured, as sudden events, generating bionic eye expression commands, taking the sound source location of the sudden event as the auditory localization direction, submitting it to the eye control module, and performing visual localization.

[0070] Preferably, the bionic ear includes a sound source localization calibration.

[0071] 7. Bionic Neck Based on the aforementioned solutions, this invention, in terms of bionic ears, includes, but is not limited to, one or more combinations of the following: The bionic neck uses a three-dimensional coordinate motion system, which supports the robot's cranial and facial movements, including nodding, shaking, and swaying. It is connected to and controlled by an end-to-end emotional bionic system, and realizes the bionic movement of the head by issuing facial expression commands based on perceptual information and head movement rules.

[0072] The rules for head movement include expressing specific facial expressions through bionic head movements such as nodding, shaking, and tilting the head, based on the content of the dialogue, visual positioning, and sound source positioning.

[0073] Preferably, the head movement rules also include, when the avoidance rules are triggered, nodding, shaking the head, and swaying the head to avoid the movement.

[0074] 8. AI Learning Based on the aforementioned solutions, this invention, in the area of ​​bionic necks, includes, but is not limited to, one or more combinations of the following: Collect typical facial expression videos, and based on the correspondence between facial expressions and expression commands, use AI learning to obtain an expression library of facial expressions and expression commands.

[0075] Preferably, facial expression videos of specific interactive objects are collected, and an expression library of facial expressions and expression commands of specific interactive objects is obtained by using AI learning based on the correspondence between facial expressions and expression commands of specific interactive objects.

[0076] Based on real-time facial expressions, search the expression library and extract expression commands.

[0077] 9. Environmental perception also includes Based on the aforementioned solutions, the present invention, in terms of environmental perception, includes, but is not limited to, one or more combinations of the following: Integrate any or a combination of the following environmental perception items into an end-to-end emotion-based bionic system for management and attention capture: It employs temperature and vibration sensors installed in the auricle to monitor ambient temperature and vibration in real time. A wireless network access module is used to monitor wireless network access performance in real time. Real-time analysis of signals and quality acquired by bionic eyes, ears, and skin; Real-time monitoring of the quality parameters of external and internal power supplies; The robot employs water level sensors and smoke and flame sensors to monitor the water level, smoke, and flames around its skull in real time. The robot uses satellite positioning sensors to monitor its location in real time and triggers an anti-theft alarm. Set sleep and wake-up rules, and enter and wake up the robot according to the sleep and wake-up rules; Set alarm and alarm deactivation rules, and enter alarm and alarm deactivation modes according to the alarm and alarm deactivation rules; Configure external communication interfaces and protocols, including encryption and decryption mechanisms.

[0078] 10. Computational functions for end-to-end emotion-inspired bionic systems Based on the aforementioned scheme, the present invention, in terms of the computational function of the end-to-end emotional bionic system, specifically includes, but is not limited to, one or more combinations of the following: Using a beat model compatible with VEM-Token beat capture and alignment model construction, beats are established and labeled for environmental perception and facial expression output commands. The following function set and facial expression library based on the beat model are established. The facial expression command for the current beat is calculated based on the environmental perception information of the next beat. The typical formula for this command includes the following formula: , (1; Formula (1) is an expression command. ,in, It is a sequence of perception functions, where 'a' is the emotion weight, 'k' is the current beat number, and 'n' is the environmental perception information label. , (2; Formula (2) is the environment perception function labeled i. Where d is the perceived vector value, and p is the number of vectors of the environmental perception function. , (3; Formula (3) is an expression command. ,in, This is a sequence of facial expression vectors, where k is the current beat number and m is the number of facial expression command vectors. , (4; Formula (4) is the VEM-sync emotion synchronization function algorithm in the method of hierarchical fusion using VEM-Token emotion synchronization function compatible with VEM-Token. In this method, VEM_sync(t) is the hidden state, f is the activation function, W_x and W_h are trainable emotion weight matrices, b is the bias vector, and the output of VEM_sync(t) is used as the emotion weight function. The result is used as the emotion weight a and stored in the expression library.

[0079] 11. Extended Modules Based on the aforementioned solution, the present invention, in terms of extended modules, specifically includes, but is not limited to, one or more of the following combinations: The long-term interactive memory module, through which the end-to-end emotional bionic system connects and configures, allows the system to remember the characteristics of the objects interacting with the robot, collect key interactive content and emotional features, so as to achieve the persistence of emotional states in cross-session interactions in the next interaction.

[0080] Preferably, a personalized emotion adapter is used. The end-to-end emotion bionic system connects to this adapter and summarizes the personalized emotional characteristics of the interactive object based on the historical data of the interactive object, so as to optimize the emotion parameters in the next interaction.

[0081] Preferably, the hyperparallel comparison sentiment computing module is connected to the end-to-end sentiment bionic system. This module is used to process historical multimodal sentiment data in real time and adopts a chip-level single-pulse hyperparallel comparison method to achieve hardware-accelerated collaborative computing of sentiment models.

[0082] Preferably, the storage of historical data includes local storage on the robot or off-site storage connected via a network.

[0083] 12. Purpose and Intent of the Invention The purpose and intent of this invention, the VEM-robot craniofacial end-to-end emotion bionic system, is as follows: (1) It solves the shortcomings of existing technologies and realizes the VEM-robot craniofacial end-to-end emotional bionic system.

[0084] (2) An end-to-end emotional bionic solution was designed, which supports the emotional multimodal beat model, VEM-sync emotional synchronization model and VEM-fe expression function model. No human-computer interaction scheme needs to be preset manually, and no manual remote control operation or intervention is required.

[0085] (3) The craniofacial model of the complete robot with emotional bionics was realized, and the end-to-end processing of the entire emotional communication process was completed.

[0086] 13. Beneficial effects of the invention (1) It can realize the cranial design of humanoid robots, truly realize end-to-end human-machine emotional communication, and support the development of specific application products for niche markets.

[0087] (2) Significantly reduce the cost of industrial production and provide software interfaces for secondary development.

[0088] (3) Realize various anthropomorphic end-to-end emotional communication, such as anthropomorphic interactive functions like "blushing when touched". Attached Figure Description

[0089] List of attached images: Figure 1 Schematic diagram of the VEM-robot craniofacial end-to-end emotional bionic system Figure 2 Schematic diagram of the optical principle of a bionic eyeball Figure 3 Schematic diagram of the appearance of a bionic eyeball Figure 4 Diagram of eye movements Figure 5 Schematic diagram of bionic skin structure Figure 6 : A schematic diagram of bionic neck movement Figure 7 Schematic diagram of bionic mandibular movement Figure 8 Diagram of the cheek area Detailed description of the attached diagram: For detailed descriptions of each accompanying drawing, please refer to the description of each drawing corresponding to the specific embodiments. It should be emphasized that these drawings are merely one illustration of the innovative concept of this invention and are simply schematic diagrams. Specific dimensions are not labeled for the system and structural parts in the drawings; users of this invention can provide detailed design dimensions based on specific applications. Therefore, the accompanying drawings are not intended to limit the innovative concept of this invention. Users of this invention can certainly draw other types of drawings and their resulting understanding and explanation based on industry-standard knowledge. Detailed Implementation

[0090] The objectives and intentions of this invention are achieved through the following specific embodiments. It should be noted that each specific embodiment has its own specific use and industrial applicability. Therefore, the following embodiments do not encompass all system features and implementation steps of this invention, and the accompanying drawings are merely one implementation method and do not constitute a limitation of this invention. The description in the claims of this invention is the core summary of the invention.

[0091] Examples of specific embodiments of the present invention are as follows: VEM-Robotic Craniofacial End-to-End Emotional Bionic System Diagram Explanation This embodiment mainly includes, but is not limited to, the following schematic diagrams, which are described in detail in the accompanying drawings. Figures 1 to 8 .

[0092] System Implementation Instructions This embodiment mainly includes system implementations 1 to 11. Part of these is the basic system of this application, and the other part consists of optional combinations based on actual applications. This is explained below: In this specification, a logical line refers to a paragraph. The word "furthermore" in a logical line indicates that the logical line is a required option for the next step of the previous logical line, while "preferably" indicates that the logical line is not a required option but an optional option for optimization in some applications.

[0093] Unless otherwise specified, these systems are not entirely necessary.

[0094] Unless otherwise specified, their order is not required.

[0095] Unless otherwise specified, the selection of materials and design parameters is not mandatory.

[0096] Based on the specific task requirements, the patent implementer makes the preferred and further selections.

[0097] 1. VEM-Robotic Craniofacial End-to-End Emotional Bionic System This invention, as a VEM-robot craniofacial end-to-end emotional bionic system, includes, but is not limited to, the following systems: Including robotic craniofacial structures and end-to-end emotional bionic systems, among which: The robotic craniofacial system includes bionic eyeballs, bionic skin, a bionic mouth, bionic ears, and a bionic neck mounted on a craniofacial support.

[0098] The end-to-end emotional bionic system captures visual, auditory, lip-touch, and skin-touch input environmental perception information from bionic eyes, bionic ears, bionic mouth, and bionic skin. Based on the environmental perception information, it calculates and generates emotional bionic facial expression commands, which are output from bionic eyes, bionic skin, bionic mouth, bionic ears, and bionic neck. This results in eye movements, eyelid movements, facial skin movements, sound output, jaw movements, mouth movements, and neck movements, achieving end-to-end bionic facial expression output from input to output.

[0099] Figure 1 This is a schematic diagram of the VEM-robot craniofacial end-to-end emotional bionic system, and also a structural schematic diagram of this embodiment. The right side of the diagram is a conceptual diagram of the robot's craniofacial system in this embodiment. It is a fully simulated human craniofacial system, possessing all the craniofacial organs of a natural human, and all of these organs are emotionally bionic and dynamic. The left side of the diagram is a logical block diagram of the end-to-end emotional bionic system of this system. It is important to emphasize that "end-to-end" here refers to a direct, non-human-interventional (e.g., no human-operated remote control) emotional bionic system from input to output. The input end refers to the environmental perception information captured by the bionic eyes, ears, mouth, and skin on the robot's craniofacial system, such as information about what is seen, heard, touched, or kissed. The output end refers to the output of emotional expressions generated through eye movements, eyelid movements, facial skin movements, sound output, jaw movements, mouth movements, and neck movements. The environmental perception information generated at the input end is fed into the VEM-model, which is an emotion-inspired model. Based on the environmental perception information at the input end, the model generates emotion-inspired facial expression commands and sends them to the output end to generate movement expressions that conform to emotion-inspired behavior, such as head movements, eye expressions, jaw movements, emotional voices, mouth expressions, and facial expressions, in order to complete the emotion-inspired expression.

[0100] 2. Bionic Eyeball Based on the aforementioned basic solution, the bionic eyeball of the present invention includes, but is not limited to, a combination of one or more of the following systems: A bionic eyeball includes an eyeball assembly and an eye movement mechanism, specifically including: The eye assembly includes a cornea, iris, pupil, lens, and image sensor arranged in sequence, with their planar optical centers aligned so that the optical centers coincide with the central optical axis. The image sensor collects visual perception information from the bionic eye, where the central optical axis is located in the visual macula. The eye movement mechanism includes a stepper motor and a drive.

[0101] Furthermore, an eye movement mechanism is used to connect two bionic eyeballs via an eyeball control module, which drives the eyeball assemblies of both eyes to rotate synchronously in a two-dimensional plane perpendicular to the central optical axis, so that the central optical axis of the eyeball assemblies of both eyes points synchronously in the same direction within the field of view, thus achieving binocular synchronization.

[0102] Preferably, the eye control module includes the connection and control of the lens and image sensor to realize automatic focusing, automatic shutter and image acquisition.

[0103] The cornea, iris, and pupil include cornea 1, iris 1, and pupil 1, which are biomimetic representations of the human eye's appearance, and cornea 2, iris 2, and pupil 2, which are biomimetic representations of the human eye's function.

[0104] In the case of biomimetic appearance, the cornea 1, iris 1, and pupil 1 are transparent at the optical axis. The iris 1 is constructed based on the iris image of the simulated person, and an aperture needs to be added between the pupil 1 and the image sensor.

[0105] In the case of using functional bionics, the cornea 2, iris 2, and pupil 2 are not only transparent at the optical axis, but the iris 2 is also constructed according to the iris image of the simulated person, and the pupil 2 realizes the aperture function.

[0106] Preferably, the cornea is made of a transparent material in the shape of an eyeball, the iris is made of a colored pattern that mimics the human eye, the pupil is a transparent opening in the middle of the iris, the image sensor includes a color image sensor and a black and white image sensor, and the aperture and lens support adaptive linkage with the image sensor, so that the eyeball assembly supports fixed focus, zoom, fixed aperture, automatic aperture and automatic shutter.

[0107] Figure 2 This is a schematic diagram of the optical principle of a bionic eye. The diagram includes the cornea, iris, pupil, lens, and image sensor of the bionic eye. The image sensor also includes the position of its central optical axis within the macula. The yellow circle on the left is a magnified, rotated view of the macula. Assuming there is a visual target directly opposite the bionic eye, an image of the target is generated in the macula based on the principle of convex lens imaging—visual positioning imaging. Figure 2 In the text, the bionic eyeball is actually a camera. While aperture, shutter speed, and focus are not explicitly mentioned, these are basic functions that a modern camera should possess. Figure 2 It is naturally included, but it is not shown due to space limitations.

[0108] It should be noted that, Figure 2 This is merely an illustrative diagram. In the future, with the development of AI, similar bionic eyeball systems may emerge, at which point this invention can directly utilize suitable kits. Therefore, this illustrative diagram is not intended to limit the innovation of this application, but rather the claims of this application shall prevail.

[0109] Figure 3 This is a schematic diagram of a bionic eyeball. It's important to note that this differs from a regular camera; it's designed and manufactured based on the shape of a human eye. This includes the cornea, iris, and pupil. The lens, image sensor, zoom, aperture, and shutter are all contained within the bionic eyeball. The iris, in particular, needs to be constructed according to the image of the actual iris of the person being replicated.

[0110] 3. Bionic Eyeball II Based on the aforementioned solutions, this invention, in terms of eyeball bionic functions and eyelid control functions, includes, but is not limited to, one or more combinations of the following: The bionic eyeball also includes bionic eyeball expression commands and eyelid control expression commands, specifically including: The end-to-end emotional bionic system controls the eyeball control module through bionic eyeball expression commands, achieving control including visual positioning, attention capture, and eyelid linkage, including the upper and lower eyelids.

[0111] Visual positioning includes: marking the axis perpendicular to the center of the robot's eyes as the forward axis, marking the macula of the bionic eyeball as the coordinate 0 point, marking the horizontal angle between the central optical axis and the forward axis as the azimuth angle, and marking the vertical angle between the central optical axis and the forward axis as the pitch angle. The end-to-end emotional bionic system transmits the azimuth and pitch angles to the eyeball control module through bionic eyeball expression commands, drives the movement of the bionic eyeball, achieves visual positioning, and the image at the visual positioning point falls on the macula.

[0112] Attention capture includes: The end-to-end emotional bionic system uses images of interactive objects, images of objects that suddenly enter the visual field, and pre-set images to be captured captured by the bionic eyeballs as sudden events, generates bionic eyeball expression commands, uses the azimuth and pitch angles of the sudden events as visual positioning directions, submits them to the eyeball control module, and executes visual positioning.

[0113] The eyelid linkage includes: the end-to-end emotional bionic system controls the eyelids by controlling facial expressions through eyelid control commands. The eyelids are made of bionic skin. The end-to-end emotional bionic system controls the eyelids to achieve linkage with the bionic eyeballs for emotional bionic expressions including looking straight ahead, looking sideways, looking up, looking down, ignoring, and rolling the eyes.

[0114] Preferably, visual positioning includes setting positioning calibration.

[0115] The eyelids also include actions that can be performed independently, such as blinking, closing the eyes, staring, squinting, turning the eyes, and wandering.

[0116] Figure 4 This is a schematic diagram of eye movement. To facilitate the explanation of visual positioning, this application introduces the concepts of pitch angle and azimuth angle in a three-dimensional polar coordinate system. For binocular vision, visual positioning is based on pitch angle and azimuth angle, ignoring line-of-sight distance.

[0117] Figure 4 In this approach, a polar coordinate system is established with respect to the front, left and right sides, and top and bottom of the craniofacial region. The macula of the eyeball is used as the origin of the polar coordinate system, and the line of sight directly in front is used as the central optical axis. Furthermore, a bipolar coordinate system is established symmetrically for the left and right eyeballs, with the visual positioning of both eyes as parallel lines to a pair of polar coordinate systems. It is important to note that the visual positioning here only considers positioning at infinity, ignoring binocular triangulation positioning near the eyeballs. This application claims that such binocular triangulation positioning can be supported by modifying the binocular positioning algorithm.

[0118] 4. Bionic skin Based on the aforementioned solutions, this invention, in the field of bionic skin, includes, but is not limited to, one or more combinations of the following: Bionic skin includes a multimodal mutual sensing system, specifically including: The multimodal sensing system connects to and executes facial expression commands issued by the end-to-end emotional bionic system, driving the bionic skin to produce peristalsis to simulate facial expressions.

[0119] When an external object applies a touch to the bionic skin, it triggers the bionic skin's attention capture, perceives the sudden event, and the multimodal mutual sensing system converts the touch action into a touch signal that inputs the sensory information, which is then input to the end-to-end emotional bionic system.

[0120] Furthermore, the bionic skin includes the lower eyelid, upper eyelid, brow bone, cheekbone and cheek, lips and tongue.

[0121] Preferably, the lower eyelid includes a lacrimal gland, and the cheek or tongue includes a salivary gland. The lacrimal gland and salivary gland are controlled by an end-to-end emotional bionic system. The lacrimal gland and salivary gland include a micro-water supply system. The lacrimal gland water supply outlet is located on the inner side of the lower eyelid, and the salivary gland water supply outlet is located on the inner side of the cheek or the underside of the tongue. The water supply system includes a water bag, a water pump, a water pipe, a water outlet, and a water pump control system.

[0122] The end-to-end emotional bionic system generates tears and saliva through tear gland expression commands and salivary gland expression commands.

[0123] The end-to-end emotional bionic system generates expression commands based on the placement of the bionic skin through a multimodal mutual sensing system, and outputs simulated expressions on the bionic skin.

[0124] Preferably, the bionic skin also includes bionic facial color, which includes, under the connection and control of an end-to-end emotional bionic system, locally changing the color of the bionic skin based on facial expression commands through touch sensing, sound content recognition, visual content recognition and emotional bionic facial color rules.

[0125] Using color-changing colored lights and a slow-transmission film with a rough surface, the light is transmitted to the outer surface of the bionic skin at designated locations on the forehead, cheekbones, and cheeks, either within the bionic skin or on the intracranial side of the bionic skin. This is achieved through an end-to-end emotional bionic system that connects and controls the transmission of light to achieve bionic facial color.

[0126] Preferably, an electrochromic film is used, which is placed on the surface or middle layer of the bionic skin at designated locations on the forehead, cheekbone and cheeks, and is connected and controlled by an end-to-end emotional bionic system to achieve bionic facial color.

[0127] Simulated facial expressions include raised eyebrows, lowered eyebrows, frowning, puffed-out cheeks, pulled-up cheeks, upturned corners of the mouth, and downturned corners of the mouth.

[0128] The bionic skin is compatible with the VEM-multimodal emotional skin system, which includes an electromagnet, a soft magnetic colloid, and a multimodal mutual inductance system. The multimodal mutual inductance system receives and executes facial expression commands, provides excitation current to the electromagnet, and generates magneto-induced peristalsis in the bionic skin to mimic facial expressions. When an external object applies a touch action to the bionic skin, the multimodal mutual inductance system converts the touch action and calculates it to form a touch signal. The multimodal mutual inductance system is connected to and receives control from the end-to-end emotional bionic system.

[0129] Preferably, the bionic skin is compatible with the VEM-multimodal emotional skin system, including electric dragging, pneumatic dragging, and a multimodal sensing system. The multimodal sensing system receives and executes facial expression commands, provides power for electric and pneumatic dragging, and generates electric and pneumatic peristalsis on the bionic skin to mimic facial expressions. When an external object applies a touch action to the bionic skin, the multimodal sensing system converts the touch action and calculates it to form a touch signal. The multimodal sensing system is connected to and receives control from the end-to-end emotional bionic system.

[0130] Figure 5 This is a schematic diagram of a biomimetic skin structure. As a preferred embodiment, this application adopts the design scheme of the bidirectional skin with magnetostrictive peristaltic expression generation and touch perception from the invention patent applications "VEM-4D Dynamic Skin Structure and Dynamic Expression Model Construction Method, CN120951102A" and "VEM-Multimodal Emotional Skin System, Application No. 2026100650642".

[0131] Figure 5 Specifically, it includes: a back plate, a sealing sleeve, a skin, a soft magnetic colloid, parallel electromagnets, an excitation coil, a front magnetic pole, a back magnetic pole, a transverse partition, a transverse partition support, a skin positioning post, and concave and convex electromagnets, etc. Among them, the soft magnetic colloid is a mixture of soft magnetic material particles and an elastic colloid. This mixture not only fills the gaps between the electromagnets, but also forms a magnetic circuit together with the magnetic poles of the electromagnets.

[0132] Regarding the selection of electromagnets: 1. Sheet electromagnets are mainly used for thinner skin, such as the eyelids, forehead, and hands. Facial expressions on these areas are relatively simple to simulate and can be achieved using simple magnetostriction. 2. Parallel electromagnets, due to their larger size, require thicker skin for installation. Their advantage lies in the fact that, within an array of multiple parallel electromagnets, the number of front poles can be greater, allowing for more complex magnetostriction formation on the skin through different north and south pole configurations, thus simulating more subtle facial expressions. 3. Concave-convex electromagnets are mainly used here to attract and repel the back poles of parallel electromagnets, creating depressions and protrusions on the skin to enhance the effect of facial expression simulation. For example, localized depressions on the skin can simulate dimples, while localized protrusions on the cheekbones can simulate a broad smile. Furthermore, combinations of these electromagnets can also be used to simulate expressions and reactions on other areas of skin; this is just one example.

[0133] Based on the distribution of electromagnets on the craniofacial skin, it is recommended to arrange the electromagnets in a 2×2mm matrix, with approximately several hundred electromagnets, to provide a more delicate reproduction of skin expressions.

[0134] In addition, a heat dissipation device is optional. When heat dissipation is not required, the sealing shell is simply made of a non-sealed material, and there is no need for heat dissipation fluid inlets and outlets. When heat dissipation is required, the sealing shell is simply made of a sealed material, and heat dissipation fluid inlets and outlets are required. It should be noted that the subcutaneous tissue uses foamed silicone material, and a high foaming ratio is used so that in the 4D dynamic skin, most of the bubbles become interconnected spaces, in which heat dissipation fluid is filled. A circulation pump is connected to the heat dissipation fluid inlet and outlet to form a circulating heat dissipation system.

[0135] In addition, the coils of the parallel electromagnets and the concave-convex electromagnets are led to the outside of the sealing sleeve and fixed to the 4D dynamic skin through a cluster connector. The skin is made of an elastic film that can be washed and made up with makeup, and it adheres to the front magnetic pole of the parallel electromagnet. When the front electromagnet moves, it causes the skin to stretch, deform and move.

[0136] The function of the diaphragm support is to maintain the fixed spacing between the diaphragm and the back plate. When a concave-convex electromagnet is energized, it generates an attractive or repulsive force on the back poles of a nearby parallel electromagnet, limiting the deformation of the diaphragm within the range of the diaphragm support. The function of the skin positioning post is to position the skin in three dimensions, preventing it from moving due to the energization of the electromagnet.

[0137] Inside the bionic skin, color-changing light-emitting tubes and light-absorbing liquid crystal materials are arranged to match the generation of bionic skin color.

[0138] 5. Bionic mouth Based on the aforementioned solutions, this invention, in the field of bionic skin, includes, but is not limited to, one or more combinations of the following: The bionic mouth also includes a bionic jaw, specifically including: The bionic mandible includes a mandibular pivot, a mandibular motion mechanism, and bionic skin. The bionic mandible is connected to and controlled by an end-to-end emotional bionic system. Based on perceptual information and mandibular motion rules, it outputs bionic facial expressions through mandibular expression commands. The bionic skin of the mandible is made of bionic skin, including the lower half of the cheek, the interconnected lower lip and upper lip, and forms a bionic oral cavity.

[0139] Furthermore, mouth-related bionics includes lower cheek bionics, jaw movement bionics, and lip-based bionic expressions.

[0140] Lip bionic expressions also include sensory input of lip touch movements, including the perception of both active robot touch and passive touch applied from the outside.

[0141] Preferably, the bionic mandibular inner oral cavity includes a bionic vocal cord, which includes an amplifier and a speaker. The frequency range covers at least 100Hz to 4000Hz, the speaker power is not less than 1W, and the speaker volume is adjustable. The bionic vocal cord is connected to and receives control from an end-to-end emotional bionic system. Based on the perception information and vocal content, it outputs vocal content, vocal expression, and mandibular movement through vocal expression commands.

[0142] The bionic mandibular inner oral cavity includes a bionic tongue, with peristaltic movements including tongue curling, tongue protrusion, and tongue flattening.

[0143] Preferably, the bionic mandibular inner oral cavity includes bionic teeth.

[0144] Preferably, the mouth-feature bionics includes an air pump, air tube, air vent, and oral cavity interface to simulate inhalation and exhalation.

[0145] in: The lower cheek is biomimetic to drive the lips to produce mouth shapes including upturning, downturning, and baring teeth to create expressions.

[0146] Mandibular motion bionics includes the mandibular motion of a robot mimicking the speech of a real person, driven by a mandibular motion mechanism that moves the mandible up and down around the mandibular axis. The mandibular motion mechanism also includes a stepper motor and a transmission, or an electromagnet and a transmission.

[0147] Furthermore, lip expression bionics includes localized peristalsis of the bionic skin in the lower and upper lip areas, including pursing, pouting, puckering, and sucking.

[0148] 6. Bionic ears Based on the aforementioned solutions, the present invention, in terms of the bionic mouth including the bionic mandible, includes, but is not limited to, one or more combinations of the following: The bionic ear uses a pickup and amplifier installed near the middle ear of both ears, connected to and controlled by an end-to-end emotional bionic system, to perceive sound content and capture attention to sound.

[0149] By using binaural sound source localization, the system can perceive the location of sounds and capture attention to those sounds, thus achieving biomimetic dialogue attention.

[0150] It can listen to sudden sound sources, sense and capture attention, and work with bionic eyes and a bionic neck to achieve attention transfer.

[0151] Preferably, the microphone includes a sound source localization mechanism that reflects sound through the contour structure of the ear.

[0152] Preferably, attention capture includes: the end-to-end emotional bionic system taking the sounds captured by the bionic ear, including the sounds of interactive objects, sudden auditory intrusions, and pre-set sounds to be captured, as sudden events, generating bionic eye expression commands, taking the sound source location of the sudden event as the auditory localization direction, submitting it to the eye control module, and performing visual localization.

[0153] Preferably, the bionic ear includes a sound source localization calibration.

[0154] 7. Bionic Neck Based on the aforementioned solutions, this invention, in terms of bionic ears, includes, but is not limited to, one or more combinations of the following: The bionic neck uses a three-dimensional coordinate motion system, which supports the robot's cranial and facial movements, including nodding, shaking, and swaying. It is connected to and controlled by an end-to-end emotional bionic system, and realizes the bionic movement of the head by issuing facial expression commands based on perceptual information and head movement rules.

[0155] Furthermore, the head movement rules include expressing specified facial expressions through biomimetic head movements, nodding, shaking, and swaying, based on the dialogue content, visual positioning, and sound source positioning.

[0156] Preferably, the head movement rules also include, when the avoidance rules are triggered, nodding, shaking the head, and swaying the head to avoid the movement.

[0157] The bionic neck includes the rules of head movement.

[0158] The head movement rules include initiating three-dimensional polar coordinate movement based on the sudden event captured by attention, turning the head to face the front of the skull towards the location of the sudden event, and coordinating with the bionic eyeballs and bionic ears to adjust the position coordinates.

[0159] Figure 6 This is a schematic diagram of bionic neck movement. In the diagram, the black grid lines represent the robot's skull and face, while the colored lines represent the neck movement coordinate system. The origin of the coordinate system is set at the middle of the lower jaw, using an orthogonal three-axis coordinate system: up / down, left / right, and front / back. This corresponds to head shaking, nodding, and tilting movements. It's important to note that these three axes are linked to achieve omnidirectional head movement, mimicking all the movements of a human head. The bionic motion mechanism employs a three-axis stepper motor, electromagnets, and a reduction gear mechanism, as well as a spherical three-dimensional motion mechanism.

[0160] Figure 7 This is a schematic diagram of biomimetic mandibular movement. (Example) Figure 6 Similarly, the movement of the jaw is also linked by a three-coordinate system to meet the up-and-down movement of the jaw when the robot speaks, as well as the biomimetic movements of the mouth expressions, including the forward and backward extension of the jaw and the pouting of the mouth.

[0161] The mandibular motion mechanism employs a three-axis stepper motor, electromagnet, and reduction gear, as well as a spherical three-dimensional motion mechanism. The origin of the three-axis is set at the midpoint of the mandibular rotation axis within the oral cavity.

[0162] Figure 8 This is a schematic diagram of the cheek area. The designated areas in the diagram are named forehead, brow bone, upper eyelid, lower eyelid, nose, upper lip, lower lip, jaw, and cheek; these are common names for the craniofacial cheek. In this embodiment, bionic skin is used, with a makeup-compatible and washable elastic film adhered to its surface. For easy replacement, this film can be attached to the bionic skin using a removable elastic film material.

[0163] 8. AI Learning Based on the aforementioned solutions, this invention, in the area of ​​bionic necks, includes, but is not limited to, one or more combinations of the following: Collect typical facial expression videos, and based on the correspondence between facial expressions and expression commands, use AI learning to obtain an expression library of facial expressions and expression commands.

[0164] Preferably, facial expression videos of specific interactive objects are collected, and an expression library of facial expressions and expression commands of specific interactive objects is obtained by using AI learning based on the correspondence between facial expressions and expression commands of specific interactive objects.

[0165] Based on real-time facial expressions, search the expression library and extract expression commands.

[0166] 9. Environmental perception also includes Based on the aforementioned solutions, the present invention, in terms of environmental perception, includes, but is not limited to, one or more combinations of the following: Integrate any or a combination of the following environmental perception items into an end-to-end emotion-based bionic system for management and attention capture: It employs temperature and vibration sensors installed in the auricle to monitor ambient temperature and vibration in real time. A wireless network access module is used to monitor wireless network access performance in real time. Real-time analysis of signals and quality acquired by bionic eyes, ears, and skin; Real-time monitoring of the quality parameters of external and internal power supplies; The robot employs water level sensors and smoke and flame sensors to monitor the water level, smoke, and flames around its skull in real time. The robot uses satellite positioning sensors to monitor its location in real time and triggers an anti-theft alarm. Set sleep and wake-up rules, and enter and wake up the robot according to the sleep and wake-up rules; Set alarm and alarm deactivation rules, and enter alarm and alarm deactivation modes according to the alarm and alarm deactivation rules; Configure external communication interfaces and protocols, including encryption and decryption mechanisms.

[0167] 10. Computational functions for end-to-end emotion-inspired bionic systems Based on the aforementioned scheme, the present invention, in terms of the computational function of the end-to-end emotional bionic system, specifically includes, but is not limited to, one or more combinations of the following: Using a beat model compatible with VEM-Token beat capture and alignment model construction, beats are established and labeled for environmental perception and facial expression output commands. The following function set and facial expression library based on the beat model are established. The facial expression command for the current beat is calculated based on the environmental perception information of the next beat. The typical formula for this command includes the following formula: , (1; Formula (1) is an expression command. ,in, It is a sequence of perception functions, where a is the emotion weight, k is the current beat number, and n is the environmental perception information label; , (2; Formula (2) is the environment perception function labeled i. Where d is the perceived vector value, and p is the number of vectors of the environmental perception function; , (3; Formula (3) is an expression command. ,in, Let k be the sequence of facial expression vectors, where k is the current beat number and m is the number of facial expression command vectors. Preferably, , (4; Formula (4) is the VEM-sync emotion synchronization function algorithm in the method of hierarchical fusion using VEM-Token emotion synchronization function compatible with VEM-Token. In this method, VEM_sync(t) is the hidden state, f is the activation function, W_x and W_h are trainable emotion weight matrices, b is the bias vector, and the output of VEM_sync(t) is used as the emotion weight function. The result is used as the emotion weight a and stored in the expression library.

[0168] The emoji library is compatible with the VEM library, which includes at least VEM modalities, styles, emotion names, and emotion values. These are VEM-sync vector sequences generated by supervised learning and reinforcement learning of typical songs guided by human vocal experts, and deep learning performed by the VEM-Token vocal emotion multimodal model.

[0169] Preferably, the VEM library also includes an emotion-based bionic library established based on psychological classification, and a mapping between multi-head self-attention mechanism tracking and eye movements, eyelid movements, and neck movements.

[0170] The end-to-end approach includes a process from the input end of emotion perception to the output end of emotion output, without relying on the intervention of tokens based on natural language processing (NLP) and without the need for human remote control intervention.

[0171] Preferably, the end-to-end approach includes a process from the input end of emotion perception to the output end of emotion output, including the intervention of tokens based on natural language processing (NLP) and the intervention of human remote control.

[0172] 11. Extended Modules Based on the aforementioned solution, the present invention, in terms of extended modules, specifically includes, but is not limited to, one or more of the following combinations: The long-term interactive memory module, through which the end-to-end emotional bionic system connects and configures, memorizes the characteristics of objects interacting with the robot, collects key interactive content and emotional features, and enables the persistence of emotional states across conversations in subsequent interactions. Here, the long-term and cross-conversation memory is at least greater than 24 hours.

[0173] Preferably, a personalized emotion adapter is used. The end-to-end emotion bionic system connects to this adapter and summarizes the personalized emotional characteristics of the interactive object based on the historical data of the interactive object, so as to optimize the emotion parameters in the next interaction.

[0174] Preferably, the hyperparallel comparison sentiment computing module is connected to the end-to-end sentiment bionic system. This module is used to process historical multimodal sentiment data in real time and adopts a chip-level single-pulse hyperparallel comparison method to achieve hardware-accelerated collaborative computing of sentiment models.

[0175] Preferably, the storage of historical data includes local storage on the robot or off-site storage connected via a network.

Claims

1. A VEM-robot craniofacial end-to-end emotional bionic system, characterized in that: Including robotic craniofacial structures and end-to-end emotional bionic systems, among which: The robotic craniofacial structure includes bionic eyeballs, bionic skin, bionic mouth, bionic ears, and a bionic neck mounted on a craniofacial support. The end-to-end emotional bionic system captures visual, auditory, lip-touch, and skin-touch input environmental perception information from bionic eyes, bionic ears, bionic mouth, and bionic skin. Based on the environmental perception information, it calculates and generates emotional bionic facial expression commands, which are output from bionic eyes, bionic skin, bionic mouth, bionic ears, and bionic neck. This results in eye movements, eyelid movements, facial skin movements, sound output, jaw movements, mouth movements, and neck movements, achieving end-to-end bionic facial expression output from input to output.

2. The system according to claim 1, characterized in that, A bionic eyeball includes an eyeball assembly and an eye movement mechanism, specifically including: The eyeball assembly includes a cornea, iris, pupil, lens, and image sensor arranged in sequence, with their planar optical centers aligned so that the optical centers coincide with the central optical axis. The image sensor collects visual perception information from the bionic eyeball, where the central optical axis is located in the visual macula. The eyeball motion mechanism includes a stepper motor and a transmission device. An eye movement mechanism connecting two bionic eyeballs via an eyeball control module drives the eyeball assemblies to rotate synchronously in a two-dimensional plane perpendicular to the central optical axis, so that the central optical axes of both eyeball assemblies point synchronously in the same direction within the field of view, achieving binocular synchronization; and / or, The eye control module includes the connection and control of the lens and image sensor, enabling autofocus, automatic shutter, and image acquisition.

3. The system according to claim 2, characterized in that, The bionic eyeball also includes bionic eyeball expression commands and eyelid control expression commands, specifically including: The end-to-end emotional bionic system controls the eyeball control module through bionic eyeball expression commands, achieving control including visual positioning, attention capture, and eyelid linkage, including the upper and lower eyelids; Visual positioning includes: marking the axis perpendicular to the center of the robot's eyes as the forward axis, marking the macula of the bionic eyeball as the coordinate 0 point, marking the horizontal angle between the central optical axis and the forward axis as the azimuth angle, and marking the vertical angle between the central optical axis and the forward axis as the pitch angle. The end-to-end emotional bionic system transmits the azimuth and pitch angles to the eyeball control module through bionic eyeball expression commands, drives the movement of the bionic eyeball, achieves visual positioning, and the image at the visual positioning point falls on the macula. Attention capture includes: The end-to-end emotional bionic system uses images of interactive objects, images of objects that suddenly enter the visual field, and pre-set images to be captured captured by the bionic eyeballs as sudden events, generates bionic eyeball expression commands, uses the azimuth and pitch angles of the sudden events as visual positioning directions, submits them to the eyeball control module, and executes visual positioning. Eyelid linkage includes: an end-to-end emotional bionic system controlling the eyelids via facial expression commands. The eyelids are composed of bionic skin, and the end-to-end emotional bionic system controls the eyelids to achieve linkage with the bionic eyeballs, responding to bionic emotional expressions such as looking straight ahead, squinting, looking up, looking down, ignoring, and rolling the eyes; and / or, Visual positioning includes setting positioning calibration.

4. The system according to claim 3, characterized in that, Bionic skin includes a multimodal mutual sensing system, specifically including: The multimodal sensing system connects to and executes the facial expression commands issued by the end-to-end emotional bionic system, driving the bionic skin to output peristalsis to simulate facial expressions; When an external object applies a touch to the bionic skin, it triggers the bionic skin's attention capture, perceives the sudden event, and the multimodal mutual sensing system converts the touch action into a touch signal that inputs the sensory information, which is then input to the end-to-end emotional bionic system. Bionic skin includes the lower eyelid, upper eyelid, brow bone, cheekbone, and cheeks, and / or, lips and tongue; and / or, The lower eyelid includes the lacrimal gland, and the cheek or tongue includes the salivary gland. The lacrimal gland and salivary gland are controlled by an end-to-end emotional bionic system, which includes a micro-water supply system. The lacrimal gland water supply outlet is located on the inner side of the lower eyelid, and the salivary gland water supply outlet is located on the inner side of the cheek or the underside of the tongue. The water supply system includes a water bag, a water pump, a water pipe, a water outlet, and a water pump control system. The end-to-end emotional bionic system generates tears and saliva through tear gland expression commands and salivary gland expression commands; The end-to-end emotional bionic system generates facial expression commands based on the placement of the bionic skin through a multimodal sensing system, and outputs simulated facial expressions on the bionic skin; and / or, Bionic skin also includes bionic facial color, which involves locally changing the color of the bionic skin under the connection and control of an end-to-end emotional bionic system, based on touch sensing, sound content recognition, visual content recognition and emotional bionic facial color rules, through facial expression commands.

5. The system according to claim 4, characterized in that, The bionic mouth also includes a bionic jaw, specifically including: The bionic mandible includes a mandibular rotation axis, a mandibular motion mechanism, and mandibular bionic skin. The bionic mandible is connected to and controlled by an end-to-end emotional bionic system. Based on sensory information and mandibular motion rules, it can output bionic facial expressions through mandibular expression commands. The mandibular bionic skin is made of bionic skin, including the lower half of the cheek, the interconnected lower lip and upper lip, and forms a bionic oral cavity. Mouth-based bionics includes lower cheek bionics, jaw movement bionics, and lip-based facial expressions. Lip bionic expressions also include sensory input of lip tactile movements, including the perception of both active robotic touch and passive external touch; and / or, The bionic mandibular inner oral cavity includes a bionic vocal cord, which includes an amplifier and a speaker. The frequency range covers at least 100Hz to 4000Hz, the speaker power is not less than 1W, and the speaker volume is adjustable. The bionic vocal cord is connected to and receives control from an end-to-end emotional bionic system. Based on the perception information and vocal content, it outputs vocal content, vocal expression and mandibular movement through vocal expression commands. The bionic mandibular inner oral cavity includes a bionic tongue with peristaltic movements including tongue protrusion, tongue thrusting, and tongue flattening; and / or, The bionic mandibular medial oral cavity includes bionic teeth; and / or, The mouth-feature bionics includes an air pump, trachea, air vent, and oral cavity interface to simulate inhalation and exhalation.

6. The system according to claim 5, characterized in that, Bionic ears include: The bionic ear uses a pickup and amplifier installed near the middle ear of both ears, connected to and controlled by an end-to-end emotional bionic system, to perceive sound content and capture attention to sound. By using binaural sound source localization, the system can perceive the location of sounds and capture attention to them, thus achieving biomimetic dialogue attention. It can listen to sudden sound sources, sense and capture attention, and coordinate with bionic eyes and a bionic neck to achieve attention shifting; and / or, The pickup includes a mechanism that reflects sound through the contours of the ear to locate the sound source; and / or, Attention capture includes: the end-to-end emotional bionic system using the bionic ear to capture sounds from interactive objects, sudden auditory intrusions, and pre-set sounds to be captured as sudden events, generating bionic eye expression commands, using the sound source localization of the sudden event as the auditory localization direction, submitting it to the eye control module, and executing visual localization; and / or, The bionic ear includes setting sound source localization calibration.

7. The system according to claim 6, characterized in that, Bionic necks include: The bionic neck uses a three-dimensional coordinate motion system, which supports the robot's cranial and facial movements, including nodding, shaking, and swaying. It connects to an end-to-end emotional bionic system and is controlled by it. Through facial expression commands, it realizes the bionic movement of the head based on perceptual information and head movement rules. Head movement rules include expressing specified facial expressions through biomimetic head movements such as nodding, shaking, and tilting the head, based on dialogue content, visual localization, and sound source localization; and / or, The head movement rules also include head nodding, head shaking, and head swaying when the avoidance rules are triggered, based on the set avoidance rules.

8. The system according to claim 7, characterized in that, This also includes the use of AI learning, specifically including: Collect typical facial expression videos, and based on the correspondence between facial expressions and expression commands, use AI learning to acquire an expression library of facial expressions and expression commands; and / or, Collect facial expression videos of specific interactive objects, and use AI learning to obtain an expression library of facial expressions and expression commands based on the correspondence between facial expressions and expression commands of specific interactive objects. Based on real-time facial expressions, search the expression library and extract expression commands.

9. The system according to claim 8, characterized in that, Environmental perception also includes: Integrate any or a combination of the following environmental perception items into an end-to-end emotion-based bionic system for management and attention capture: It employs temperature and vibration sensors installed in the auricle to monitor ambient temperature and vibration in real time. A wireless network access module is used to monitor wireless network access performance in real time. Real-time analysis of signals and quality acquired by bionic eyes, ears, and skin; Real-time monitoring of the quality parameters of external and internal power supplies; The robot employs water level sensors and smoke and flame sensors to monitor the water level, smoke, and flames around its skull in real time. The robot uses satellite positioning sensors to monitor its location in real time and triggers an anti-theft alarm. Set sleep and wake-up rules, and enter and wake up the robot according to the sleep and wake-up rules; Set alarm and alarm deactivation rules, and enter alarm and alarm deactivation modes according to the alarm and alarm deactivation rules; Configure external communication interfaces and protocols, including encryption and decryption mechanisms.

10. The system according to claim 9, characterized in that, The computational functions of an end-to-end emotion-inspired bionic system include: Using a beat model compatible with VEM-Token beat capture and alignment model construction, beats are established and labeled for environmental perception and facial expression output commands. The following function set and facial expression library based on the beat model are established. The facial expression command for the current beat is calculated based on the environmental perception information of the next beat. The typical formula for this command includes the following formula: ， （1）； Formula (1) is an expression command. ,in, It is a sequence of perception functions, where a is the emotion weight, k is the current beat number, and n is the environmental perception information label; ， （2）； Formula (2) is the environment perception function labeled i. Where d is the perceived vector value, and p is the number of vectors of the environmental perception function; ， （3）； Formula (3) is an expression command. ,in, For a sequence of facial expression vectors, k is the current beat number, and m is the number of facial expression command vectors; and / or, ，（4）； Formula (4) is the VEM-sync emotion synchronization function algorithm in the method of hierarchical fusion using VEM-Token emotion synchronization function compatible with VEM-Token. In this method, VEM_sync(t) is the hidden state, f is the activation function, W_x and W_h are trainable emotion weight matrices, b is the bias vector, and the output of VEM_sync(t) is used as the emotion weight function. The result is used as the emotion weight a and stored in the expression library.

11. The system according to claim 10, characterized in that, Also includes: The long-term interactive memory module, through which the end-to-end emotional bionic system connects and configures, allows the system to remember the characteristics of the objects interacting with the robot, collect key interactive content and emotional features, so as to achieve the persistence of emotional state in cross-session interaction in the next interaction; And / or, Personalized emotion adapter: The end-to-end emotion bionic system connects to this adapter and summarizes the personalized emotional characteristics of the interactive object based on the historical data of the interactive object, so as to optimize the emotional parameters in the next interaction. And / or, The hyperparallel comparison sentiment computing module, connected to the end-to-end sentiment bionic system, is used to process historical multimodal sentiment data in real time. It employs a chip-level, single-pulse hyperparallel comparison method to achieve hardware-accelerated collaborative computation of sentiment models; and / or, Historical data storage includes local storage on the robot or off-site storage connected via a network.

Images