Intelligent doll interaction method, device and medium

Abstract

The application discloses an intelligent doll interaction method and device and a medium, and relates to the technical field of artificial intelligence, and comprises the following steps: acquiring an image frame sequence containing a user collected by a camera; performing real-time visual semantic analysis on the image frame sequence to extract a visual state vector of the user; mapping the visual state vector into eye expression rendering control parameters and eye view angle rendering control parameters; and performing dynamic drawing update on an eye image presented on a screen of an intelligent doll according to the eye expression rendering control parameters and the eye view angle rendering control parameters, so that the eye image expression matches the emotional state of the user and the gaze direction adapts to the spatial position of the user. The above scheme establishes a real-time mapping and dynamic driving mechanism from visual perception to screen rendering parameters, realizes the coordination and unity of emotional expression and gaze tracking in eye contact interaction, and improves the naturalness and immersion of the interaction.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, device and medium for interacting with intelligent dolls. Background Technology

[0002] As smart dolls and companion robots are increasingly used in home entertainment and emotional support, improving the naturalness and immersion of the interaction between these products and users has become a focus of technological development.

[0003] In existing technologies, smart dolls typically integrate microphones and speakers to enable voice interaction. Some products further include displays for facial expressions and cameras for capturing environmental images. During a typical interaction, the camera detects the user's face to trigger preset responses; for example, when a specific user is recognized, a pre-stored static facial image is displayed on the screen, or a pre-choreographed sequence of greeting actions is played. It is evident that the relationship between the visual information captured by the camera and the content displayed on the screen is a discrete trigger mapping. The visual perception result merely serves as a switch signal to invoke preset display content and does not participate in the real-time generation and continuous evolution of the displayed content.

[0004] Specifically, the eye images displayed on the screens of existing smart dolls are typically looping animations driven by fixed resources or simple rules. They cannot dynamically adjust the expression of the eyes based on the user's real-time emotional state, nor can they adaptively follow the direction of the eyes based on the user's spatial movement. This lack of dynamic correlation between visual perception and display feedback limits the naturalness of emotional expression and the accuracy of gaze tracking in the eye interactions of smart dolls, making it difficult to simulate the smooth interactive effects of real-time adjustment of posture based on visual observation in interpersonal interactions. Summary of the Invention

[0005] This invention provides an intelligent doll interaction method, device, and medium, aiming to solve the technical problem of: how to provide an interaction method that can map the user's visual state information collected by the camera into the rendering control parameters of the intelligent doll's screen eye image in real time and drive the dynamic evolution of eye expression and gaze direction.

[0006] In a first aspect, embodiments of the present invention provide a smart doll interaction method, comprising: Acquire a sequence of image frames containing the user captured by the camera; Real-time visual semantic analysis is performed on the image frame sequence to extract the user's visual state vector; The visual state vector is mapped to eye expression rendering control parameters and eye viewpoint rendering control parameters; Based on the eye expression rendering control parameters and the eye perspective rendering control parameters, the eye image presented on the smart doll screen is dynamically drawn and updated so that the expression presented in the eye image matches the user's emotional state and the gaze direction presented in the eye image adapts to the user's spatial position.

[0007] Optionally, the visual state vector includes age attribute parameters, gender attribute parameters, emotional state parameters, ambient illumination parameters, spatial location parameters, and continuous emotional intensity scores, wherein the spatial location parameters include the user's planar coordinate position and depth distance estimate in the image frame. The step of performing real-time visual semantic parsing on the image frame sequence to extract the user's visual state vector includes: Face detection is performed on each frame of the image frame sequence to obtain a sequence of face bounding box coordinates; Kalman filtering is applied to the face bounding box coordinate sequence for tracking, a unique tracking identifier is assigned to the same user, and a motion trajectory is generated. Extract the image region corresponding to the latest face bounding box in the motion trajectory, and input the image region in parallel into the age estimation network, gender classification network, emotion recognition network and ambient illumination estimation network to obtain the age attribute parameter, the gender attribute parameter, the emotion state parameter and the ambient illumination parameter respectively; The continuous emotion intensity score is calculated based on the emotion category confidence vector output by the emotion recognition network. The planar coordinate position is determined based on the geometric center coordinates of the latest face bounding box in the image frame, and the depth distance estimate is calculated based on the ratio of the area of ​​the bounding box to the preset reference area.

[0008] Optionally, mapping the visual state vector to eye expression rendering control parameters includes: The age attribute parameter, the gender attribute parameter, and the emotional state parameter are used as input features and input into the pre-trained eye expression generation model. Through forward reasoning of the eye expression generation model, eye expression rendering control parameters that integrate age features, gender features, and emotional features are output. Alternatively, the age attribute parameter, the gender attribute parameter, and the emotional state parameter can be used as a joint query key to match and search in a pre-built eye expression mapping table to obtain the eye expression rendering control parameters corresponding to the joint query key. The eye expression mapping table stores the mapping relationship between different age groups, genders, and emotional state combinations and corresponding eye expression rendering control parameters. The eye expression mapping table is dynamically updated from the cloud server via an over-the-air (OTA) communication interface.

[0009] Optionally, mapping the visual state vector to eye expression rendering control parameters further includes: Determine whether the ambient illuminance parameter is lower than a preset illuminance threshold; If the ambient illuminance parameter is lower than the preset illuminance threshold, a first eye expression adjustment command is generated. The first eye expression adjustment command is used to increase the diameter of the pupil region in the eye image and increase the highlight reflection intensity of the pupil region. If the ambient illuminance parameter is not lower than the preset illuminance threshold, a second eye expression adjustment command is generated. The second eye expression adjustment command is used to maintain the reference diameter size and reference specular reflection intensity of the pupil region in the eye image. The first eye expression adjustment command or the second eye expression adjustment command is used as a control component of the eye expression rendering control parameters.

[0010] Optionally, mapping the visual state vector to eye expression rendering control parameters further includes: The continuous emotion intensity scores are subjected to time series smoothing filtering to obtain a smoothed emotion intensity curve. Calculate the instantaneous rate of change of the smoothed emotion intensity curve; When the instantaneous rate of change exceeds a preset positive mutation threshold, a dynamic blinking drive signal is generated. The dynamic blinking drive signal is used to drive the eye image to perform a rapid blinking action sequence. The dynamic blinking drive signal is used as a control component of the eye expression rendering control parameters.

[0011] Optionally, mapping the visual state vector to eye-view rendering control parameters includes: Based on the horizontal and vertical offset pixel values ​​of the planar coordinate position relative to the center point of the image frame, calculate the user's first gaze deviation angle in the horizontal direction and the second gaze deviation angle in the vertical direction. Based on the depth distance estimate, calculate the interpupillary distance convergence compensation coefficient between the user and the screen; The first gaze deviation angle, the second gaze deviation angle, and the pupillary distance convergence compensation coefficient are input into the eye rotation model to obtain the first offset vector of the left pupil in the eye image and the second offset vector of the right pupil in the eye image. The first offset vector and the second offset vector are used as the eye view rendering control parameters.

[0012] Optionally, the method further includes: Acquire the audio stream captured by the microphone, and perform speech recognition processing on the audio stream to obtain the text sequence and speech emotion feature vector; The text sequence is input into the semantic understanding model to obtain the semantic intent encoding; Cross-modal feature alignment and fusion are performed on the user's facial expression features in the visual state vector and the voice emotion feature vector to obtain a multimodal emotion fusion vector; The semantic intent encoding and the multimodal emotion fusion vector are input into the eye expression generation network to obtain multimodal-driven eye expression rendering control parameters; The eye expression rendering control parameters are corrected and updated based on the multimodal driving eye expression rendering control parameters.

[0013] Optionally, before acquiring the sequence of image frames containing the user captured by the camera, the method further includes: The screen of the smart doll is controlled to be in a low-power off state, while the camera is kept capturing environmental images at the first sampling frame rate; The environmental image is subjected to moving target detection. When a moving target is detected, the camera sampling frame rate is increased to a second sampling frame rate, which is greater than the first sampling frame rate. Face detection is performed on the image frames acquired after increasing the sampling frame rate. When the number of image frames that continuously detect face regions reaches a preset frame threshold, it is determined that the user's lingering gaze condition is met. In response to meeting the user's lingering gaze condition, the screen of the smart doll is lit up and the step of performing real-time visual semantic analysis on the image frame sequence is executed; The step of detecting moving targets in the environmental image includes: Frame difference operation is performed on corresponding pixels of adjacent frames in the environmental image to obtain a difference image; The number of pixels in the differential image whose pixel values ​​exceed a first preset threshold is counted. When the number of pixels exceeds a second preset threshold, it is determined that a moving target has been detected.

[0014] Secondly, embodiments of the present invention also provide a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described method.

[0015] Thirdly, embodiments of the present invention also provide a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the above-described method.

[0016] This invention provides a method, device, and medium for intelligent doll interaction. The invention acquires a sequence of user image frames captured by a camera and performs real-time visual semantic analysis to extract a visual state vector containing the user's emotional state and spatial location. This visual state vector is then mapped to eye expression rendering control parameters and eye viewpoint rendering control parameters. Based on these control parameters, the eye images displayed on the intelligent doll screen are dynamically rendered and updated, ensuring that the expression in the eye images matches the user's emotional state in real time and that the gaze direction of the eye images adapts to the user's spatial location. Thus, the above processing establishes a real-time mapping and dynamic driving mechanism between camera visual perception information and screen eye image rendering parameters. This overcomes the technical limitation that visual state information cannot directly participate in the continuous generation and evolution of displayed content, achieving dynamic coordination and unity of emotional expression and gaze tracking in intelligent doll eye interaction, and enhancing the naturalness and immersion of the interaction process. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart illustrating an intelligent doll interaction method provided in an embodiment of the present invention; Figure 2 This is a schematic block diagram of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0021] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0022] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0023] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0024] Please see Figure 1 This invention provides an intelligent doll interaction method. The method uses a camera to perceive the user's visual state in real time and converts this visual state into dynamic changes in the expression and perspective of the eyes displayed on the intelligent doll's screen, thus achieving a visual response of the eye image to the user's state. Specifically, the method includes the following steps: S1, acquire the sequence of image frames containing the user captured by the camera.

[0025] In practice, the system acquires a sequence of image frames containing the user captured by a camera. The camera can be positioned on the smart doll itself, for example, in the doll's head area, to simulate the human eye's perspective.

[0026] Furthermore, the camera continuously captures environmental images within its field of view at a preset sampling frequency. When a user enters the camera's field of view, the captured image frame sequence contains the user's facial and body image information. This image frame sequence serves as the original input data source for subsequent visual semantic analysis steps. Each image frame records the user's spatial position, facial contours, and facial features, among other visual information, in the form of a pixel matrix at the corresponding capture time.

[0027] S2, perform real-time visual semantic parsing on the image frame sequence to extract the user's visual state vector.

[0028] In practice, real-time visual semantic analysis is performed on the image frame sequence to extract the user's visual state vector. Visual semantic analysis refers to using computer vision algorithms to analyze and understand the visual content in image frames, extracting a set of parameters that can quantitatively represent the user's current state. This set of parameters is organized in vector form, which is the visual state vector.

[0029] Furthermore, the visual state vector includes at least an emotional state parameter reflecting the user's facial expression and a spatial position parameter reflecting the user's geometric position relative to the camera or smart doll screen. Specifically, the emotional state parameter can be extracted by performing feature analysis on the user's facial region in the image frame, such as identifying the geometric pattern formed by facial key points, matching the current facial shape with multiple predefined emotional prototypes, and outputting the emotion category with the highest matching degree as the emotional state parameter. Furthermore, the spatial position parameter can be extracted by detecting the user's imaging position and imaging size in the image frame, such as determining the two-dimensional coordinate position of the user's facial region in the image frame, and estimating the relative distance between the user and the camera based on the perspective relationship between the facial imaging size and distance. The parsing process takes the image frame sequence as input and outputs a continuously updated visual state vector, realizing the information transformation from raw pixel data to high-level semantic parameters.

[0030] In some preferred embodiments, the visual state vector includes age attribute parameters, gender attribute parameters, emotional state parameters, ambient illumination parameters, spatial location parameters, and continuous emotional intensity scores. The spatial location parameters include the user's planar coordinate position and estimated depth distance within the image frame. The real-time visual semantic parsing of the image frame sequence to extract the user's visual state vector includes: performing face detection on each frame of the image frame sequence to obtain a face bounding box coordinate sequence; performing Kalman filtering tracking on the face bounding box coordinate sequence, assigning a unique tracking identifier to the same user, and generating a tracking vector. The motion trajectory is analyzed; the image region corresponding to the latest face bounding box in the motion trajectory is extracted, and the image region is input in parallel into an age estimation network, a gender classification network, an emotion recognition network, and an ambient illumination estimation network to obtain the age attribute parameter, the gender attribute parameter, the emotion state parameter, and the ambient illumination parameter, respectively; the continuous emotion intensity score is calculated based on the emotion category confidence vector output by the emotion recognition network; the planar coordinate position is determined based on the geometric center coordinates of the latest face bounding box in the image frame, and the depth distance estimate is calculated based on the ratio of the area size of the bounding box to the preset reference area.

[0031] In practice, the visual state vector includes age, gender, emotional state, ambient illumination, spatial location, and continuous emotional intensity score. Specifically, the spatial location parameter further includes the user's planar coordinates and estimated depth distance within the image frame. The age parameter represents the user's age group, the gender parameter represents the user's gender, the ambient illumination parameter represents the brightness of the user's environment, and the continuous emotional intensity score represents the activation level or intensity of the user's current emotional state. These parameters characterize the user's state and interaction scenario from multiple dimensions, including identity attributes, emotional state, spatial positioning, and the interaction environment, providing a multi-source information foundation for the refined control of eye expression and perspective in subsequent steps.

[0032] Furthermore, the process of performing real-time visual semantic parsing on the image frame sequence to extract the visual state vector is as follows: First, face detection is performed on each frame of the image frame sequence to obtain a sequence of face bounding box coordinates. Face detection refers to the technical process of locating the region where a face appears in an input image. This step can be implemented using an object detection algorithm based on image feature analysis, scanning and analyzing each frame of the image frame sequence to identify whether a face region exists. When a face is detected, the bounding box coordinates used to define the face region are output. The bounding box coordinates are usually represented by the vertex pixel coordinates of the rectangle and the width and height values ​​of the rectangle. For multiple consecutive frames, this step outputs a set of face bounding box coordinates arranged in chronological order of acquisition time, i.e., a sequence of face bounding box coordinates. This sequence serves as input data for subsequent object tracking and attribute analysis.

[0033] Furthermore, Kalman filtering is applied to the face bounding box coordinate sequence for tracking, assigning a unique tracking identifier to the same user and generating a motion trajectory. Kalman filtering is an algorithm for recursively optimally estimating the state of a dynamic system, suitable for predicting and updating the motion state of a target between consecutive frames. Based on the bounding box observations obtained from face detection, the detected face bounding boxes in each frame are input into the Kalman filter as observation data. The filter estimates the motion state of each face target, including position and velocity, and assigns a unique and invariant tracking identifier to each moving target based on the matching correlation results between the observations and predictions. Bounding boxes with the same tracking identifier are connected sequentially in time to form a motion trajectory, which records the position sequence of the same user within the field of vision over time. This step, using the face bounding box coordinate sequence as input and outputting motion trajectory information with a tracking identifier, solves the identity confusion problem that may occur in single-frame face detection, ensuring that the visual state of the same user can be continuously tracked in the time dimension.

[0034] Furthermore, the image region corresponding to the latest face bounding box in the motion trajectory is extracted. This image region is then fed in parallel into an age estimation network, a gender classification network, an emotion recognition network, and an ambient illumination estimation network to obtain age attribute parameters, gender attribute parameters, emotional state parameters, and ambient illumination parameters, respectively. The age estimation network is a pre-trained deep learning model. Its input is a face region image cropped from the image frame, and its output is an estimated value of the user's age or an age group classification result. The gender classification network is a pre-trained binary classification deep learning model. Its input is the face region image, and its output is the determination result of the user's gender and the corresponding confidence value. The emotion recognition network is a pre-trained multi-class deep learning model. Its input is the face region image, and its output is a confidence vector composed of the confidence values ​​corresponding to each emotion category in a predefined emotion category set. The emotion category with the highest confidence can be used as the output emotion state parameter. The ambient illumination estimation network can take a face region image or a complete image frame as input. By analyzing features such as pixel brightness distribution, contrast, and histogram in the image, it outputs an estimated value of the current ambient illumination level or an illumination level classification result. The four network models mentioned above perform calculations in parallel, all using the image region corresponding to the latest face bounding box as shared input data, and independently outputting their respective parameters, thus improving the real-time processing efficiency of the feature extraction stage.

[0035] It should be noted that among the aforementioned parallel networks, the ambient illuminance estimation network can be selected for deployment based on the actual hardware resource conditions of the product. For embedded platforms with limited computing resources, the following alternative implementation methods can be used to obtain the ambient illuminance parameters to further reduce computational overhead.

[0036] In an alternative implementation, the ambient illuminance parameter is obtained by reading the automatic exposure parameters of the camera's ISP module. When performing automatic exposure control, the camera's ISP module calculates and updates the exposure time and gain value in real time based on the current ambient light conditions to maintain the overall brightness of the output image at the target level. The darker the ambient light, the longer the exposure time and the higher the gain value are typically set by the ISP module; conversely, the brighter the ambient light, the shorter the exposure time and the lower the gain value are. Therefore, the exposure time and gain value directly reflect the intensity of the current ambient light and can be used as the basis for calculating the ambient illuminance parameter. Specifically, the current exposure time value and analog or digital gain value of the ISP module are read through the camera driver interface, and these values ​​are normalized or weighted before being output as the ambient illuminance parameter. This method eliminates the need for additional calculations on the image frame; the illuminance parameter is obtained only after reading register or driver status data once, minimizing computational overhead and response latency, making it a preferred implementation.

[0037] In another alternative implementation, the ambient illuminance parameter is obtained by statistically analyzing the pixel brightness of the image frame. Specifically, pixel brightness analysis is performed on a specified area of ​​the currently acquired image frame. The brightness value can be obtained by converting the color image to a luminance color space or directly extracting pixel values ​​from the grayscale image. The average brightness of all pixels within the area is calculated, and the average pixel brightness value is compared with a preset brightness threshold. The ambient illuminance level or illuminance grade is determined based on the comparison result. The specified area can be the region corresponding to the face bounding box, the center region of the image frame, or the entire image region. This method only involves pixel traversal and addition operations, with low computational load, making it suitable for low-computing-power scenarios.

[0038] The outputs of both alternative methods can be used as ambient illuminance parameters for the pupil adaptive adjustment step based on ambient illuminance parameters in subsequent embodiments. Neither method requires deploying a separately trained ambient illuminance estimation network, resulting in extremely low computational overhead, making them particularly suitable for embedded hardware platforms with limited computational resources.

[0039] Furthermore, a continuous emotion intensity score is calculated based on the emotion category confidence vector output by the emotion recognition network. This continuous emotion intensity score is a scalar value that continuously varies within a preset range, used to quantify the intensity or activation level of the user's currently expressed emotion, rather than simply outputting discrete emotion category labels. Calculation methods may include directly using the confidence value corresponding to the emotion category with the highest confidence in the confidence vector as the continuous emotion intensity score, or weighted summing of multiple confidence values ​​in the confidence vector, or applying a preset nonlinear mapping function to obtain an intensity score value that varies over a continuous range. This continuous emotion intensity score, as a component of the visual state vector, provides a quantitative basis for subsequent, detailed mapping of emotion intensity.

[0040] Furthermore, the planar coordinate position is determined based on the geometric center coordinates of the latest face bounding box in the image frame, and the depth distance estimate is calculated based on the ratio of the bounding box area to a preset reference area. The planar coordinate position can be represented by the horizontal and vertical coordinates of the face bounding box's geometric center in the image frame's pixel coordinate system, reflecting the user's positional offset relative to the image frame's center point or reference point within the two-dimensional imaging plane. The depth distance estimate is calculated based on the perspective projection relationship in monocular vision, i.e., for the same camera parameters and the same reference object, there is an inverse relationship between the distance between the object and the camera in the physical world and the image size it presents in the image. By pre-calibrating the reference area occupied by the standard face bounding box at a standard reference distance, the ratio of the real-time detected face bounding box area to this reference area is calculated, and the current depth distance estimate is derived by combining the ratio with the distance conversion relationship. The planar coordinate position and the depth distance estimate together constitute spatial position parameters, providing the user's positioning data in three-dimensional space for subsequent calculation of the eye's gaze direction.

[0041] Furthermore, the depth distance estimation method is applicable to close-range scenes of 0.3m to 2m, and the estimation error can reach ±20cm within this range.

[0042] In this embodiment, the above steps realize the complete extraction process from the original image frame sequence to a multi-dimensional visual state vector. The face detection step takes the image frame sequence as input and outputs the face bounding box coordinate sequence; the Kalman filter tracking step takes the bounding box coordinate sequence as input and outputs the motion trajectory with tracking markers; the multi-network parallel inference step takes the image region corresponding to the latest bounding box in the motion trajectory as input and outputs age attribute parameters, gender attribute parameters, emotional state parameters, and environmental illumination parameters; the emotion confidence vector is further calculated to obtain a continuous emotion intensity score; the bounding box geometric information is used to calculate the spatial position parameters, and all parameters converge to form the visual state vector.

[0043] The technical solution of this embodiment integrates age, gender, emotional state, ambient illumination, spatial location, and continuous emotional intensity scores into the visual state vector. This characterizes the user's state and interaction scenario from multiple perceptual dimensions, including identity attributes, emotional state, spatial positioning, and the interaction environment, providing rich and accurate driving information for the refined rendering of eye expressions and perspectives in subsequent steps. Furthermore, Kalman filtering is used to track face detection results across consecutive frames, effectively eliminating inter-frame data fluctuations and identity correspondence confusion caused by detection jitter or brief occlusions, ensuring the continuity and consistency of visual states over time. Furthermore, multiple attribute recognition networks process the same face region image in parallel, improving computational efficiency while maintaining feature extraction accuracy, enabling the visual semantic parsing step to meet the response latency requirements of real-time interaction. Furthermore, depth distance estimates are calculated based on the ratio of the bounding box area to the reference area, allowing the user's spatial distance information to be obtained using a single camera, providing a simple and effective implementation method for 3D localization in gaze tracking.

[0044] S3, map the visual state vector into eye expression rendering control parameters and eye viewpoint rendering control parameters.

[0045] In practice, the visual state vector is mapped to eye expression rendering control parameters and eye perspective rendering control parameters. Further, mapping refers to converting the extracted parameter values ​​in the visual state vector into a set of instructions or parameters that can be used to control graphics rendering, based on pre-established conversion rules from visual parameters to rendering parameters. Further, the eye expression rendering control parameters are a set of values ​​used to regulate the visual representation of the eye image in terms of shape, brightness, and color; for example, they can control the curvature of the upper eyelid, the arc shape of the lower eyelid, the angle of the corner of the eye, and the degree of pupil opening and closing. Further, the eye perspective rendering control parameters are a set of values ​​used to regulate the pupil position in the eye image to achieve changes in the gaze direction; for example, they can control the offset of the left pupil relative to the left eye socket reference point and the offset of the right pupil relative to the right eye socket reference point. The mapping process receives visual state vectors as input data. Based on pre-defined transformation relationships, it associates emotional state parameters with eye expression rendering control parameters, so that different emotional states correspond to different combinations of expression control parameters. It also associates spatial position parameters with eye viewpoint rendering control parameters, so that different spatial positions correspond to different combinations of viewpoint control parameters. The eye expression rendering control parameters and eye viewpoint rendering control parameters output by the mapping process together constitute the control data set driving the update of the eye image.

[0046] In some preferred embodiments, mapping the visual state vector to eye expression rendering control parameters includes: inputting the age attribute parameter, the gender attribute parameter, and the emotional state parameter as input features into a pre-trained eye expression generation model; and outputting eye expression rendering control parameters that fuse age features, gender features, and emotional features through forward inference of the eye expression generation model.

[0047] In practice, the process of mapping the visual state vector to eye expression rendering control parameters includes: inputting age attribute parameters, gender attribute parameters, and emotional state parameters as input features into a pre-trained eye expression generation model; and outputting eye expression rendering control parameters that integrate age features, gender features, and emotional features through forward inference of the eye expression generation model.

[0048] Furthermore, the eye expression generation model is a deep neural network model pre-trained on a large amount of labeled sample data. Training samples can include real human eye images of users at different ages, genders, and under different emotional states, or artistically stylized doll eye images, each with corresponding eye morphology parameter annotations. During the training phase, the model learns the mapping relationship between the input age, gender, and emotional state parameters and the output eye expression rendering control parameters. The model input consists of feature data composed of age, gender, and emotional state parameters, and the model output is the eye expression rendering control parameters. These control parameters can include multiple control terms used to define eye morphology, such as control point coordinate sequences for controlling the upper eyelid curve shape, control point coordinate sequences for controlling the lower eyelid curve shape, parameters for controlling the position and angle of the inner and outer corners of the eyes, adjustment parameters for controlling the pupil dilation ratio, and adjustment parameters for controlling the color or brightness of the iris region, etc.

[0049] Furthermore, through forward inference calculations using a neural network model, the model can automatically integrate the combined effects of different age characteristics, gender characteristics, and emotional characteristics on eye expression. For example, for users whose age attribute parameter indicates an older age group, the model output can make the periocular area of ​​the eye image exhibit subtle texture features consistent with age; for users whose gender attribute parameter indicates female, the model output can make the eye shape curve of the eye image exhibit corresponding soft features; for users whose emotional state parameter indicates a specific emotion, the model output can make the eyelid opening and closing degree and brow ridge shape of the eye image exhibit typical features matching that emotion. Because the neural network model has the ability to capture the nonlinear interaction relationships between multidimensional features, its output eye expression rendering control parameters can naturally integrate age, gender, and emotional factors, generating coordinated, unified, and expressive eye expression control data.

[0050] This embodiment uses age, gender, and emotional state parameters from the visual state vector as input data. Through forward inference calculation of the eye expression generation model, it outputs eye expression rendering control parameters. These control parameters serve as one of the inputs for subsequent screen rendering steps, driving fine-tuning of the eye image in terms of shape and style.

[0051] The technical solution of this embodiment uses a pre-trained eye expression generation model to fuse age, gender, and emotional state parameters, replacing simple rule-based parameter mapping or linear interpolation. This allows the generated eye expression rendering control parameters to reflect the complex interaction between individual user characteristics and emotional expression. Furthermore, the non-linear fitting capability of the neural network model ensures a high degree of coordination and smooth transition in terms of age-appropriateness, appropriate gender feature presentation, and accuracy of emotional expression, avoiding unnatural expressions or deviations caused by overly rigid mapping rules. The eye expression rendering control parameters generated by fusing multi-dimensional features further simplify the input interface of the rendering step and improve the data transfer efficiency between expression generation and image rendering.

[0052] For example, the eye expression generation model can be implemented based on a multilayer perceptron (MLP) or convolutional neural network (CNN) architecture. Its training process includes: acquiring facial image samples containing different ages, genders, and emotional states, as well as corresponding target eye morphological parameter annotations; inputting the sample data into the eye expression generation model, calculating the loss function value between the predicted eye morphological parameters output by the model and the target eye morphological parameters; updating the model parameters based on the loss function value until the model converges.

[0053] Alternatively, in some preferred embodiments, mapping the visual state vector to eye expression rendering control parameters includes: using the age attribute parameter, the gender attribute parameter, and the emotional state parameter as a joint query key, performing a matching search in a pre-constructed eye expression mapping table to obtain the eye expression rendering control parameters corresponding to the joint query key; wherein, the eye expression mapping table stores the mapping relationship between different age stages, different genders, and different emotional state combinations and the corresponding eye expression rendering control parameters, and the eye expression mapping table is dynamically updated from the cloud server via an over-the-air (OTA) communication interface.

[0054] In practical implementation, the process of mapping visual state vectors to eye expression rendering control parameters can be replaced by a static mapping table scheme instead of a neural network inference scheme. The eye expression mapping table is a data structure pre-built before system deployment or during operation. It uses discretized combinations of age stage, gender, and emotional state as index keys, and the corresponding eye expression rendering control parameters as storage values. The construction of the mapping table can be based on statistical analysis of a large number of labeled samples or offline inference results of neural network models under specific input conditions, solidifying the optimal or typical outputs in the continuous parameter space into a discrete table. For example, the age attribute parameter can be discretized into stages such as child, youth, middle-aged, and elderly; the gender attribute parameter can be discretized into male and female; and the emotional state parameter can be discretized into categories such as joy, sadness, anger, surprise, and neutral. The mapping table then contains complete entries for all combinations of the above dimensions, with each entry corresponding to a set of eye expression rendering control parameters. These control parameters can also include the upper eyelid curve control point coordinate sequence, the lower eyelid curve control point coordinate sequence, eye corner angle parameters, pupil dilation ratio parameters, etc.

[0055] During real-time interaction, the system uses the age, gender, and emotional state parameters output from the visual semantic parsing step to categorize age and gender into the closest discrete stage or category, and matches the emotional state parameter to the discrete emotional category with the highest confidence. A lookup is then performed in the mapping table using a joint query key composed of these three parameters. Since the mapping table is stored using a data structure that supports fast retrieval, such as a hash table or ordered array, the lookup operation can be completed in constant time complexity. The computational overhead is far lower than the forward inference process of a neural network model, making it suitable for embedded hardware platforms with limited computing resources.

[0056] Furthermore, the eye expression mapping table is not fixed but supports dynamic updates via OTA (Over-The-Air) from the cloud server through a wireless communication interface. The smart doll can have a built-in wireless communication module such as WiFi or Bluetooth, establishing a connection with the cloud server periodically or upon receiving an update push. The cloud server can generate a new version of the eye expression mapping table based on newly collected large-scale user interaction data, iterative optimizations of the doll's expression style by the design team, or updates to expression resources for specific holiday themes, and then package and send the updated mapping table data to the smart doll. Upon receiving the data, the smart doll replaces or incrementally updates its local mapping table, thereby continuously enriching and optimizing the expressiveness of eye interactions without requiring firmware upgrades or hardware replacements.

[0057] This embodiment uses the age, gender, and emotional state parameters in the visual state vector as joint query keys to obtain the eye expression rendering control parameters through a lookup table. It is suitable for implementation scenarios that require extremely low computational latency or have limited hardware resources.

[0058] The technical solution in this embodiment replaces the neural network model with a pre-built eye expression mapping table for forward inference, transforming the mapping from multi-dimensional parameters to rendering control parameters into a fast lookup operation of a discrete table. This significantly reduces computational complexity and processing latency while maintaining expressive power. The mapping table covers discrete combinations of different age groups, genders, and emotional states, ensuring basic coverage and diversity of output expressions. Furthermore, dynamic updates to the mapping table are supported via OTA (Over-The-Air) updates, allowing the eye interaction expressiveness of the smart doll to be continuously optimized and expanded throughout the product's lifecycle. New expression styles or themed expression resources can be obtained without firmware upgrades, combining the dual advantages of low hardware cost and high flexibility.

[0059] Furthermore, in a preferred embodiment, the above-mentioned static mapping table scheme can be combined with the neural network model scheme to balance table lookup efficiency and facial expression.

[0060] Specifically, the age attribute parameter, the gender attribute parameter, and the emotional state parameter are used as a joint query key to perform a matching search in a pre-constructed eye expression mapping table to obtain initial eye expression rendering control parameters. Simultaneously, the age attribute parameter, the gender attribute parameter, and the emotional state parameter are used as input features and fed into a pre-trained fine-tuned neural network model for forward inference to obtain a set of corrected offsets from the model output.

[0061] This fine-tuning neural network model differs from the previously described eye expression generation model in its output layer definition: the eye expression generation model outputs complete eye expression rendering control parameters, while the fine-tuning neural network model outputs expression fine-tuning parameters, which are the adjustment increments or coefficients corresponding to each control component in the initial eye expression rendering control parameters. For example, expression fine-tuning parameters may include the offset of eyelid opening and closing amplitude, the offset of eye corner angle, and the adjustment factor of pupil scaling ratio, etc.

[0062] The basic components of a fine-tuned neural network model include an input layer, several hidden layers, and an output layer. The input layer receives a feature vector composed of age, gender, and emotional state parameters. The hidden layers can be fully connected, with 2 to 4 layers depending on computational power. Each layer contains several neurons, and activation functions such as ReLU or LeakyReLU can be used. The number of neurons in the output layer is the same as the number of control components in the initial eye expression rendering control parameters. Each output neuron corresponds to a correction offset or adjustment coefficient for one control component. The output layer activation function can be Tanh or a linear function, allowing the output value to be either positive or negative, thus achieving positive enhancement or negative weakening of the baseline expression.

[0063] The training process of fine-tuning the neural network model includes: acquiring a training sample set, where each set of training samples contains input age attribute parameters, gender attribute parameters, and emotional state parameters, as well as labeled target expression rendering control parameters; using a pre-built eye expression mapping table, querying the input parameters of each set of training samples to obtain the initial eye expression rendering control parameters; subtracting the initial eye expression rendering control parameters from the target expression rendering control parameters component by component to obtain the target correction offset for each control component, which serves as the training label; inputting the input parameters of each set of training samples into the fine-tuning neural network model for forward propagation to obtain the model's predicted correction offset; calculating the loss function value between the predicted correction offset and the target correction offset, where the loss function can be either mean squared error loss or smoothed L1 loss; updating the model parameters based on the loss function value using the backpropagation algorithm, iterating the training until the model converges.

[0064] The initial eye expression rendering control parameters and the expression fine-tuning parameters are superimposed or weighted and fused according to their corresponding components to obtain the final eye expression rendering control parameters. During the superposition and fusion process, the effective range of the expression fine-tuning parameters can be limited to a preset numerical range to ensure that the fine-tuned expression does not deviate from the baseline expression style range defined by the mapping table.

[0065] In this hybrid implementation, the mapping table lookup provides a baseline expression that matches the current user's age, gender, and emotional combination, ensuring rapid generation and comprehensive coverage of the basic expression. The fine-tuned neural network model, based on the baseline expression, finely adjusts each control component of the baseline expression according to continuous values ​​of the input parameters, enabling the output expression to reflect subtle differences between individual user characteristics, such as nuanced expressions of different emotional intensities within the same emotion category. This hybrid approach maintains the low latency of the mapping table while leveraging the non-linear fitting capability of the fine-tuned neural network model to enhance the subtlety and personalization of expression expression, making it suitable for product configurations with high requirements for interactive experience and sufficient computational redundancy.

[0066] In some preferred embodiments, mapping the visual state vector to eye expression rendering control parameters further includes: determining whether the ambient illuminance parameter is lower than a preset illuminance threshold; if the ambient illuminance parameter is lower than the preset illuminance threshold, generating a first eye expression adjustment instruction, the first eye expression adjustment instruction being used to increase the diameter of the pupil region in the eye image and increase the highlight reflection intensity of the pupil region; if the ambient illuminance parameter is not lower than the preset illuminance threshold, generating a second eye expression adjustment instruction, the second eye expression adjustment instruction being used to maintain the reference diameter and reference highlight reflection intensity of the pupil region in the eye image; and using the first eye expression adjustment instruction or the second eye expression adjustment instruction as a control component of the eye expression rendering control parameters.

[0067] In specific implementation, the process of mapping the visual state vector to eye expression rendering control parameters also includes: determining whether the ambient illuminance parameter is lower than a preset illuminance threshold; if the ambient illuminance parameter is lower than the preset illuminance threshold, generating a first eye expression adjustment instruction, which is used to increase the diameter of the pupil region in the eye image and enhance the highlight reflection intensity of the pupil region; if the ambient illuminance parameter is not lower than the preset illuminance threshold, generating a second eye expression adjustment instruction, which is used to maintain the reference diameter and reference highlight reflection intensity of the pupil region in the eye image; and using the first eye expression adjustment instruction or the second eye expression adjustment instruction as a control component of the eye expression rendering control parameters.

[0068] Specifically, the ambient illuminance parameter is provided by the output of the ambient illuminance estimation network in the above embodiments, and its value reflects the brightness level of the ambient light in which the user is located. A preset illuminance threshold is a predetermined reference illuminance value used to distinguish between a normally adequate ambient light state and a relatively dim ambient light state. The judgment step compares the ambient illuminance parameter with the preset illuminance threshold. When the ambient illuminance parameter is lower than the preset illuminance threshold, it indicates that the user is in a dimly lit environment, and the system generates a first eye expression adjustment instruction. The first eye expression adjustment instruction includes instructions for adjusting the rendering parameters of the pupil area, including increasing the pupil diameter drawing size by a specified ratio relative to a preset reference diameter size, and correspondingly increasing the brightness value and display size of the highlight reflection area within the pupil area relative to the preset reference value. This adjustment simulates the physiological dilation of the pupil in a dark environment, while enhancing highlight reflection helps to highlight the presence and vividness of the eye image against a dark background. When the ambient illuminance parameter is not lower than the preset illuminance threshold, it indicates that the ambient light is normal or adequate, and the system generates a second eye expression adjustment instruction. The second eye expression adjustment command instructs that the baseline diameter size and baseline specular reflection intensity parameters of the pupil area be kept unchanged, without performing any additional adjustments.

[0069] Furthermore, the generated adjustment instructions, as a control component of the eye expression rendering control parameters, together with the control parameters output by the neural network model in Example 3, constitute a complete set of eye expression rendering control parameters, which are then input into the screen rendering stage. During rendering, the rendering engine integrates the instructions of all control components to determine the shape and attributes of each visual element in the final eye image presented on the screen. This pupil adaptive adjustment mechanism enables the intelligent doll's eye expressions to not only respond to the user's identity attributes and emotional state, but also to make a biomimetic adaptive response to changes in the lighting of the interactive environment, enriching the environmental perception dimension of eye expression expression.

[0070] The technical solution of this embodiment introduces an ambient illumination parameter and adaptively adjusts the diameter and highlight reflection intensity of the pupil area accordingly. This allows the eye image displayed on the smart doll's screen to simulate the physiological response characteristics of a real biological pupil to changes in light intensity. Automatically enlarging the pupil size and enhancing the highlight reflection visual effect in low-light environments not only improves the visual recognition of the eye image under dim conditions but also creates a human-like interactive experience, as if the doll can perceive ambient light and make corresponding physiological adjustments. Furthermore, this environmental adaptive adjustment feature, combined with expression rendering based on the user's identity and emotions, enriches the expressive layers of the eyes, ensuring that the smart doll's eye interaction maintains a natural and vivid visual performance under different ambient lighting conditions.

[0071] Optionally, in some preferred embodiments, in standby mode, the screen continuously plays a preset slow breathing frame animation sequence to simulate the micro-breathing sensation of a biological pupil slowly expanding / contracting, with a cycle of about 4 seconds; the pupil diameter variation range of the breathing frame animation is 90px to 95px.

[0072] In some preferred embodiments, mapping the visual state vector to eye expression rendering control parameters further includes: performing time-series smoothing filtering on the continuous emotion intensity score to obtain a smoothed emotion intensity curve; calculating the instantaneous rate of change of the smoothed emotion intensity curve; generating a dynamic blinking drive signal when the instantaneous rate of change exceeds a preset positive mutation threshold, the dynamic blinking drive signal being used to drive the eye image to perform a rapid blinking action sequence; and using the dynamic blinking drive signal as a control component of the eye expression rendering control parameters.

[0073] In specific implementation, the process of mapping the visual state vector to eye expression rendering control parameters also includes: performing time-series smoothing filtering on continuous emotion intensity scores to obtain a smoothed emotion intensity curve; calculating the instantaneous rate of change of the smoothed emotion intensity curve; when the instantaneous rate of change exceeds a preset positive mutation threshold, generating a dynamic blinking drive signal, which is used to drive the eye image to perform a rapid blinking action sequence; and using the dynamic blinking drive signal as a control component of the eye expression rendering control parameters.

[0074] The continuous emotion intensity score, provided by the above embodiments, dynamically changes over time, reflecting the temporal evolution of the user's emotional activation level. First, a smoothing filter is applied to the time series of the continuous emotion intensity score; this can be achieved using a moving average filter or an exponentially weighted moving average filter. The smoothing filter suppresses high-frequency interference components caused by detection noise or transient fluctuations in the score series, resulting in a smoother and more stable emotion intensity curve. This ensures the stability of subsequent rate of change calculations and avoids false triggering of subsequent steps due to instantaneous noise fluctuations.

[0075] Furthermore, based on the smoothed emotion intensity curve, the change or derivative of this curve at the current moment relative to the previous moment is calculated as the instantaneous rate of change. The instantaneous rate of change reflects the speed and direction of the evolution of the user's emotion intensity over time. When the instantaneous rate of change is positive and large, it indicates that the user's emotion intensity is undergoing a rapid increase, such as a rapid shift from a relatively calm state to a state of surprise or excitement. A preset positive mutation threshold is a pre-calibrated threshold value for the rate of change; when the instantaneous rate of change exceeds this threshold, it means that the user's emotion intensity has undergone a significant and rapid positive jump.

[0076] Furthermore, in response to the above conditions being met, the system generates a dynamic blinking drive signal. The dynamic blinking drive signal is a sequence of timing control signals used to trigger the eye image to execute a rapid blinking sequence. The rapid blinking sequence includes a continuous frame animation process where the eyelids quickly close from an open state to a fully closed or partially closed state, and then quickly open back to an open state. The closing phase lasts for a short time, and the overall movement simulates the natural blinking response of humans when emotionally excited, suddenly distracted, or unexpectedly stimulated. The dynamic blinking drive signal, as a control component of the eye expression rendering control parameters, is input into the rendering pipeline along with other expression control parameters. During rendering, the timing control instructions contained in the dynamic blinking drive signal temporarily override or adjust the eyelid opening and closing related control parameters, driving the eye image to complete a full blinking action. After the action is completed, the eye image returns to the normal expression determined by other control components.

[0077] Furthermore, this dynamic blinking-driven mechanism introduces an instant feedback behavior based on dynamic emotional changes to the eye-eye interaction of intelligent dolls, so that the response of eye expressions is no longer limited to the static mapping of steady-state emotion categories, but can capture and respond to the instantaneous features of emotional fluctuations.

[0078] The technical solution of this embodiment performs temporal smoothing filtering and rate of change analysis on continuous emotion intensity scores, and automatically triggers a dynamic blinking sequence when a significant positive change in emotion intensity is detected. This allows the intelligent doll to provide immediate and subtle visual feedback responses to the transient changes in the user's emotions. Furthermore, compared to solutions that adjust the static shape of the eyes solely based on steady-state emotion categories, this dynamic blinking mechanism more closely resembles the subconscious micro-expression behaviors that humans exhibit during emotional fluctuations or attention shifts, enhancing the agility and anthropomorphism of eye contact interaction. Furthermore, by using the dynamic blinking drive signal as an independent control component of the eye expression rendering control parameters, the triggering logic of the blinking action is decoupled from the parameter generation logic of conventional expression rendering. This ensures both the continuous stability of steady-state expression and a vivid embellishment effect during emotional climaxes.

[0079] In some preferred embodiments, mapping the visual state vector to eye view rendering control parameters includes: calculating a first gaze deviation angle in the horizontal direction and a second gaze deviation angle in the vertical direction based on the horizontal and vertical offset pixel values ​​of the planar coordinate position relative to the center point of the image frame; calculating the interpupillary distance convergence compensation coefficient between the user and the screen based on the depth distance estimate; inputting the first gaze deviation angle, the second gaze deviation angle, and the interpupillary distance convergence compensation coefficient into the eye rotation model to obtain a first offset vector of the left pupil in the eye image and a second offset vector of the right pupil in the eye image; and using the first offset vector and the second offset vector as the eye view rendering control parameters.

[0080] In specific implementation, the process of mapping the visual state vector to eye view rendering control parameters includes: calculating the user's first gaze deviation angle in the horizontal direction and the second gaze deviation angle in the vertical direction based on the horizontal and vertical offset pixel values ​​of the planar coordinate position relative to the center point of the image frame; calculating the interpupillary distance convergence compensation coefficient between the user and the screen based on the depth distance estimation value; inputting the first gaze deviation angle, the second gaze deviation angle, and the interpupillary distance convergence compensation coefficient into the eye rotation model to obtain the first offset vector of the left pupil in the eye image and the second offset vector of the right pupil in the eye image; and using the first offset vector and the second offset vector as eye view rendering control parameters.

[0081] The planar coordinate position and depth distance estimates are provided by the spatial position parameters in Example 2. The planar coordinate position uses the geometric center point of the image frame as the reference origin, calculating the lateral and vertical offset pixel values ​​of the user's face bounding box relative to this reference origin. Using the camera's own imaging parameters or a pre-calibrated conversion relationship between pixel offset and physical angle, the lateral offset pixel values ​​can be converted into a first horizontal gaze angle, and the vertical offset pixel values ​​into a second vertical gaze angle. For example, based on the ratio of the camera's horizontal field of view to the total number of horizontal pixels in the image, the angular change corresponding to each pixel offset can be determined, and then the pixel offset can be converted into an angle value. The gaze angle characterizes the degree of deviation of the user's face relative to the camera's optical axis in the horizontal and vertical directions.

[0082] Furthermore, the depth distance estimate is used to calculate the interpupillary distance convergence compensation coefficient between the user and the screen. When the user is close to the screen, their eyes converge to focus on the virtual eye image on the nearby screen, meaning both pupils move inward simultaneously. The interpupillary distance convergence compensation coefficient quantifies the magnitude of this physiological accommodation effect; its value is negatively correlated with the depth distance estimate, meaning the closer the distance, the greater the required convergence compensation. By introducing this compensation coefficient, the rendered pupils can exhibit an inward shift effect consistent with real visual physiology, thus simulating a natural, stereoscopic gaze. The interpupillary distance convergence compensation coefficient can be calculated using a pre-established mapping table of depth distance and convergence compensation amount or a fitting function.

[0083] Furthermore, the eye movement model is a geometric computational model that calculates the pupil offset vectors of both eyes based on the gaze deviation angle and convergence compensation parameters. The model receives the first gaze deviation angle, the second gaze deviation angle, and the pupillary distance convergence compensation coefficient as input data. Internally, the model performs calculations based on the geometric relationship between the visual axes of both eyes in three-dimensional space, outputting the offset vectors of the left pupil relative to the reference center of the left eye socket and the right pupil relative to the reference center of the right eye socket. The offset vectors can be represented using normalized relative coordinates or absolute pixel offsets. The model calculation ensures that the output pupil offset vectors allow the gaze of both eyes to converge at a target point corresponding to the user's facial position in three-dimensional space. The calculated first and second offset vectors are output as eye viewpoint rendering control parameters to the rendering step. The rendering engine adjusts the drawing position of the pupils in the eye image accordingly, achieving a gaze-tracking effect for the user's vision.

[0084] In this embodiment, the plane coordinate position and depth distance estimation values ​​in the spatial position parameters are used as input data. After the gaze deviation angle is calculated, the pupil distance convergence compensation is calculated, and the eye rotation model is solved, the accurate pupil offset vector of both eyes is output as the direct control data to drive the change of eye angle.

[0085] The technical solution of this embodiment calculates the horizontal and vertical gaze deviation angles using planar coordinate positions, calculates the interpupillary distance convergence compensation coefficient using depth distance estimation, and then generates pupil offset vectors through an eye rotation model. This achieves accurate mapping calculation from the user's coordinate position in the two-dimensional image to the three-dimensional gaze direction of the smart doll's eyes. This solution fully considers the actual impact of the distance between the user and the screen on the convergence angle of the eyes. In close-range interaction scenarios, interpupillary distance convergence compensation ensures that the gaze focus of the eye image converges more realistically at the user's face, avoiding unnatural problems such as gaze focus divergence or non-convergence of the eyes that may occur due to neglecting distance factors. Furthermore, the eye view rendering control parameters are output in the form of pupil offset vectors, directly affecting the parameter settings for the pupil drawing position in the rendering step. The data flow is clear and unambiguous, effectively ensuring the real-time performance of gaze tracking calculation and the accuracy of gaze direction presentation.

[0086] S4. Based on the eye expression rendering control parameters and the eye perspective rendering control parameters, dynamically draw and update the eye image presented on the smart doll screen so that the expression presented in the eye image matches the user's emotional state and the gaze direction presented in the eye image adapts to the user's spatial position.

[0087] In practice, the eye images displayed on the smart doll's screen are dynamically drawn and updated based on the eye expression rendering control parameters and the eye perspective rendering control parameters. The smart doll screen is a display device placed on the doll's face area to display eye images. The eye images displayed on the screen can be programmatically generated vector graphics or bitmap images synthesized from pre-set materials. Furthermore, dynamic drawing and updating refers to the real-time redrawing or refreshing of the pixel content of the eye images displayed on the screen based on the eye expression rendering control parameters and the eye perspective rendering control parameters input at each moment. Furthermore, regarding eye expression, the rendering process adjusts the shape parameters of the eyelid curve, the size parameters of the pupil area, the angle parameters of the corners of the eyes, and the brightness and position parameters of the highlight reflection area in the eye image according to the eye expression rendering control parameters, so that the expression presented by the eye image adapts to the user's emotional state.

[0088] For example, when the emotional state parameter indicates that the user is in a positive and happy state, the rendered eye image may show features such as the upper eyelid curving upward and the corners of the eyes slightly raised; when the emotional state parameter indicates that the user is in a depressed state, the rendered eye image may show features such as the eyelid drooping and the corners of the eyes straight or tilted downward.

[0089] Furthermore, regarding the eye's perspective, the rendering process adjusts the relative drawing position of the pupil within the eye socket region in the eye image based on the eye perspective rendering control parameters, ensuring that the gaze direction of the eye image points towards the user's spatial location. For example, when the user's spatial position parameter indicates that the user is located slightly to the left in front of the doll, both pupils are rendered with an appropriate distance offset to the right, creating a visual effect of the gaze looking to the left. When the user's position moves, the pupil offset is continuously adjusted accordingly, ensuring that the gaze always follows the user. Through this dynamic rendering and update process, the smart doll screen presents an eye image in real time that combines emotional expression and gaze tracking capabilities, forming a closed-loop interaction from visual perception to visual feedback.

[0090] This embodiment constructs a complete data processing chain from image acquisition, semantic parsing, parameter mapping to image rendering through the above four main steps. The image frame sequence serves as the raw data input, which is transformed into a visual state vector through visual semantic parsing. The visual state vector is then used to generate eye expression rendering control parameters and eye viewpoint rendering control parameters through mapping rules. These two types of control parameters jointly drive the dynamic rendering and updating of the eye image on the screen, and finally, the visible changes in the shape of the eye image serve as the interactive feedback output.

[0091] The technical solution of this embodiment acquires a sequence of image frames containing the user in real time through a camera and performs visual semantic analysis to extract visual state vectors reflecting the user's emotional state and spatial location, enabling the smart doll to perceive the user's real-time state. Furthermore, the visual state vectors are mapped to eye expression rendering control parameters and eye viewpoint rendering control parameters, and the eye images displayed on the screen are dynamically drawn and updated accordingly. This allows the eye images to adjust their expression synchronously according to the user's emotional changes and to adjust their gaze direction synchronously according to the user's positional movement. Furthermore, this solution overcomes the limitations of traditional smart dolls that can only present pre-set static expressions or fixed perspectives, establishing a real-time correlation and dynamic mapping mechanism between visual perception and display feedback. This improves the naturalness of emotional expression and the accuracy of gaze tracking in the smart doll's eye interaction, creating a more immersive interactive experience for the user.

[0092] In some preferred embodiments, the method further includes: acquiring an audio stream captured by a microphone and performing speech recognition processing on the audio stream to obtain a text sequence and a speech emotion feature vector; inputting the text sequence into a semantic understanding model to obtain a semantic intent encoding; performing cross-modal feature alignment and fusion on the user's facial expression features in the visual state vector and the speech emotion feature vector to obtain a multimodal emotion fusion vector; inputting the semantic intent encoding and the multimodal emotion fusion vector into an eye expression generation network to obtain multimodal driven eye expression rendering control parameters; and correcting and updating the eye expression rendering control parameters according to the multimodal driven eye expression rendering control parameters.

[0093] In practice, the process begins by acquiring an audio stream from a microphone and performing speech recognition processing to obtain a text sequence and a speech emotion feature vector. The microphone can be integrated into the smart doll itself to continuously collect sound signals from its environment. The speech recognition processing includes an automatic speech recognition stage, which converts the speech signals in the audio stream into a corresponding text sequence, reflecting the literal content expressed by the user. Simultaneously, the speech recognition processing also includes a speech emotion recognition stage. This stage analyzes the acoustic features in the audio stream, including pitch variation trajectories, emphasis modulation patterns, speech rate variations, and formant distribution characteristics, to extract a speech emotion feature vector that characterizes the emotional tendency inherent in the speech. This speech emotion feature vector reflects the emotional tone conveyed in the user's speech.

[0094] Furthermore, the text sequence is input into a semantic understanding model to obtain semantic intent encoding. The semantic understanding model can be a neural network model based on natural language processing techniques, used for semantic parsing and encoding of text sequences of variable length. This model converts the text sequence into a fixed-dimensional semantic intent encoding vector. The semantic intent encoding vector captures the core semantic information of the user's voice commands or expressions, such as the type of intent expressed, including asking questions, issuing instructions, expressing evaluations, or making statements, as well as key parameter information associated with the intent.

[0095] Furthermore, cross-modal feature alignment and fusion are performed on the user's facial expression features and speech emotion feature vectors in the visual state vector to obtain a multimodal emotion fusion vector. The visual state vector is provided by the real-time visual semantic parsing step in Example 1, and the user's facial expression features included therein are specifically the feature vectors output by the intermediate layer of the emotion recognition network. These feature vectors retain fine-grained information about facial expressions in the high-dimensional feature representation before inputting to the classification layer. Alternatively, as a simplified implementation, the user's facial expression features can also be directly replaced by the one-hot encoded vector of the emotion state parameter. Cross-modal feature alignment refers to mapping facial expression features from the visual modality and speech emotion features from the auditory modality to the same feature representation space, so that the vector representations of features expressing similar emotional connotations in different modalities are close to each other or comparable in this space. Feature alignment can be achieved by introducing a contrastive learning strategy or a modal adversarial training strategy during the model training phase. Specifically, a cross-modal alignment method based on contrastive learning can be used to map visual facial expression features and vocal emotion features to the same-dimensional embedding space. A contrastive loss function is then used to bring cross-modal features expressing the same emotion closer together in this space, while features expressing different emotions are pushed apart. The aligned features can then be fused using concatenation or an attention mechanism to obtain the multimodal emotion fusion vector.

[0096] Aligned facial expression features and speech emotion features are fused using methods such as feature concatenation, weighted summation, or gated fusion mechanisms to obtain a multimodal emotion fusion vector. This fusion vector integrates emotional information from two perceptual channels: facial visual expression and speech intonation, exhibiting higher accuracy and robustness compared to single-modal emotion representations.

[0097] Furthermore, the semantic intent encoding and the multimodal emotion fusion vector are jointly input into the eye expression generation network to obtain multimodal-driven eye expression rendering control parameters. The eye expression generation network is a neural network model whose input is the joint representation data of the semantic intent encoding and the multimodal emotion fusion vector, and whose output is the eye expression rendering control parameters. During the training phase, the network learns the mapping relationship between semantic intent information and fused emotion information to appropriate eye expression performance. For example, when the semantic intent encoding indicates that the user is speaking or responding to content with positive emotional connotations and the multimodal emotion fusion vector represents a positive emotional tendency, the network output control parameters can make the eye image present an expression matching the positive emotion; when the semantic intent encoding indicates that the user is issuing a command or request and the fused emotion representation is neutral, the network output control parameters can make the eye image present an expression of focused listening.

[0098] Furthermore, the eye expression rendering control parameters are corrected and updated based on the multimodal-driven eye expression rendering control parameters. This correction and update can be achieved by weighted fusion of the multimodal-driven eye expression rendering control parameters and the eye expression rendering control parameters generated based on the pure visual modality, according to preset weights. Alternatively, in specific interactive scenarios, the multimodal-driven eye expression rendering control parameters can replace the original visual-driven expression rendering control parameters. The corrected and updated eye expression rendering control parameters are then fed into the screen rendering step, ultimately driving the screen to display an eye image that integrates both speech semantic and emotional information.

[0099] Through the above steps, this embodiment introduces speech modal information into the decision-making process for generating eye expressions, forming an eye-expression interaction mechanism driven by the fusion of visual and auditory perception. Speech recognition and semantic understanding provide the user's interactive intent, speech emotion recognition provides supplementary information on emotional expression, and cross-modal fusion technology ensures the consistent integration of multi-source perceptual information, ultimately outputting a more context-adaptive eye expression performance.

[0100] The technical solution of this embodiment performs cross-modal alignment and fusion processing on facial expression features in the visual state vector, compensating for the perceptual uncertainty of a single visual modality under adverse conditions such as noise interference in noisy environments, partial occlusion of the user's face, or unclear facial expressions. Furthermore, the multimodal emotion fusion vector integrates emotional expression information from both visual and auditory perception channels, improving the accuracy and robustness of the overall emotion recognition results. Furthermore, by inputting semantic intent encoding and the multimodal emotion fusion vector together into the eye expression generation network, the eye feedback of the intelligent doll not only responds to the user's emotional state but also matches the semantic context of the user's voice dialogue, thereby improving the naturalness and empathetic expressiveness of the interaction process. Furthermore, the multimodal driver output corrects and updates the original eye expression rendering control parameters, achieving effective fusion of visual and auditory perception information at the rendering decision level, further enriching the intelligence level of the intelligent doll's eye interaction.

[0101] In some preferred embodiments, before acquiring the image frame sequence containing the user captured by the camera, the method further includes: controlling the smart doll screen to be in a low-power off-screen state and maintaining the camera to acquire environmental images at a first sampling frame rate; performing motion target detection on the environmental images, and when a motion target is detected, increasing the camera sampling frame rate to a second sampling frame rate, the second sampling frame rate being greater than the first sampling frame rate; performing face detection on the image frames acquired after increasing the sampling frame rate, and determining that the user's lingering gaze condition is met when the number of consecutively detected image frames of a face region reaches a preset frame number threshold; and, in response to meeting the user's lingering gaze condition, lighting up the smart doll screen and performing the step of real-time visual semantic parsing of the image frame sequence.

[0102] In practice, firstly, the smart doll's screen is kept in a low-power off-screen state, while the camera continues to capture environmental images at a first sampling frame rate. When no user is detected approaching or looking at the doll, the smart doll's screen remains off or displays only a static, simple image with extremely low brightness to reduce power consumption. The camera operates at a relatively low first sampling frame rate, such as one frame per second or one frame per two seconds, to control the power consumption of image acquisition and front-end processing at a low level.

[0103] Furthermore, moving target detection is performed on the environmental images. When a moving target is detected, the camera's sampling frame rate is increased to a second sampling frame rate, which is higher than the first sampling frame rate. Moving target detection can employ lightweight image processing methods based on inter-frame difference analysis or background subtraction algorithms to analyze and judge environmental images continuously acquired at low frame rates. Once a moving object with a changing position or shape is detected in the image, it indicates that a user may have entered the camera's field of view or is approaching the smart doll.

[0104] In some preferred embodiments, the moving target detection is implemented using the frame difference method, which includes: performing frame difference operation on corresponding pixels of adjacent frames in the environmental image to obtain a difference image; counting the number of pixels in the difference image whose pixel values ​​exceed a first preset threshold; and determining that a moving target has been detected when the number of pixels exceeds a second preset threshold.

[0105] In practice, the frame difference method is a motion detection algorithm based on the analysis of pixel grayscale value changes between adjacent image frames. Specifically, for an environmental image sequence acquired at a first sampling frame rate, two temporally adjacent frames are selected. The pixel grayscale values ​​at corresponding positions in the two frames are subtracted pixel by pixel, and the absolute value of the difference is used to construct a difference image. The difference image is then binarized, marking pixels whose values ​​exceed a preset motion pixel threshold as moving pixels. The total number of moving pixels is counted, and when the number of moving pixels exceeds a preset moving pixel threshold, a moving target is determined to exist in the current frame.

[0106] The frame difference method involves only pixel-level subtraction and comparison operations, requiring no background model maintenance or complex feature extraction. Its computational load is extremely low, making it suitable for continuous execution by an MCU or low-power processor in low-power standby mode for smart dolls. By adjusting the motion pixel threshold and the number of motion pixels threshold, the sensitivity of motion detection can be controlled, striking a balance between filtering false detections caused by minor changes in ambient light or sensor noise and ensuring timely detection of user approach. The intermittent execution of the frame difference method, combined with a low sampling frame rate, enables continuous monitoring of environmental changes without significantly increasing power consumption.

[0107] Once the frame difference method detects a moving target, the system increases the camera's sampling frame rate from a lower first sampling frame rate to a higher second sampling frame rate, such as 15 frames per second or higher. This increased frame rate allows the camera to capture more continuous and detailed image sequences, providing a high-quality data foundation for subsequent, more accurate face detection steps.

[0108] Furthermore, face detection is performed on the image frames acquired after increasing the sampling frame rate. When the number of image frames in which a face region is continuously detected reaches a preset frame count threshold, it is determined that the user's lingering gaze condition is met. The face detection can use the same or a lighter face detection algorithm as in Embodiment 2 to determine the presence of face regions in each image frame acquired at the second sampling frame rate. The preset frame count threshold is used to distinguish between a user's brief, accidental passing by and the behavior of intentionally stopping and looking at the doll. For example, it can be set to determine that the lingering gaze condition is met if a face region is stably detected in at least ten consecutive image frames. This continuous frame count determination mechanism effectively filters out false wake-up situations caused by the user unintentionally scanning the camera or briefly passing by.

[0109] Furthermore, in response to meeting the user's gaze condition, the smart doll's screen is illuminated, and a step of real-time visual semantic analysis of the image frame sequence is performed. The smart doll's screen switches from a low-power off state to a normal illuminated display state, and initiates the complete interactive process of visual semantic analysis, parameter mapping, and dynamic rendering and updating of eye images as described in Embodiment 1 and subsequent embodiments. Thus, when the smart doll detects that the user is consciously gazing at it, it actively illuminates the screen and initiates eye interaction, exhibiting an interactive posture of gaze following and facial expression response.

[0110] This embodiment constructs a step-by-step wake-up chain through the above steps, from low-power environment monitoring, motion-triggered sampling frame rate enhancement, face persistence confirmation, to active interaction initiation. Each stage progressively improves the granularity of perception analysis and the investment of computing resources, minimizing the overall power consumption level in standby mode while ensuring timely response to user interaction intentions.

[0111] The technical solution of this embodiment effectively reduces the overall power consumption of the smart doll during periods of no interaction by running the camera at a lower sampling frame rate and performing lightweight moving target detection in standby mode, thus helping to extend the battery life in battery-powered scenarios. Furthermore, the sampling frame rate is increased and the computationally intensive face detection process is initiated only when a moving target is detected, achieving hierarchical and on-demand allocation of computing resources. Using continuous multi-frame face detection results as the basis for determining gaze, the system effectively filters out false wake-up triggers that may occur due to environmental disturbances or unintentional user passage, ensuring the accuracy and reliability of the wake-up mechanism. Furthermore, in response to meeting the gaze condition, the screen is actively lit and the eye-tracking interaction process is initiated, allowing the smart doll to naturally enter an interactive state simply by perceiving the user's gaze behavior, without requiring the user to issue a voice wake-up command or perform touch operations. This improves the convenience and proactivity of interaction initiation, creating a human-like interactive experience where the doll senses the user's arrival and proactively greets them.

[0112] Specifically, the steps of acquiring image frame sequences captured by the camera, performing real-time visual semantic parsing on the image frame sequences to extract visual state vectors, and mapping the visual state vectors to eye expression rendering control parameters and eye viewpoint rendering control parameters are executed on the first processor. The first processor can be an edge AI chip with a built-in neural network processing unit, such as the RV1126B. Its integrated NPU can perform hardware-accelerated inference for deep learning models such as age estimation networks, gender classification networks, emotion recognition networks, and ambient illumination estimation networks, thereby controlling power consumption while meeting real-time requirements.

[0113] After generating the eye expression rendering control parameters and eye view rendering control parameters, the first processor does not directly drive the display screen to draw images. Instead, it sends these control parameters to the second processor via the UART bus. The UART bus is an asynchronous serial communication interface suitable for short-distance data exchange between processors. The communication data can be encapsulated using a predefined frame format. For example, a frame header of 0xAA 0x55 identifies the start of a frame. The frame body sequentially contains the values ​​of each control component of the eye expression rendering control parameters and the values ​​of each offset vector of the eye view rendering control parameters. A check byte can be appended to the end of the frame to ensure transmission reliability.

[0114] The second processor can be an MCU, such as an ESP32-S3, which receives control parameter data sent by the first processor via a UART bus. After receiving a complete data frame, the second processor parses the frame data, extracts the eye expression rendering control parameters and eye perspective rendering control parameters, and executes the step of dynamically drawing and updating the eye image displayed on the smart doll screen according to these control parameters. The second processor connects to the display screen via a display interface such as SPI or parallel port. The display screen can be a circular TFT LCD module, such as a GC9D01, used to display the programmably generated eye image. The second processor adjusts the shape of the eye image according to the eye expression rendering control parameters and adjusts the drawing position of the pupil within the eye socket area according to the eye perspective rendering control parameters, driving the display screen to complete the real-time refresh of the eye image.

[0115] The aforementioned dual-processor architecture assigns computationally intensive AI visual inference tasks to the first processor, which is equipped with NPU acceleration capabilities, and assigns real-time-critical screen rendering control tasks to the second processor, which has a low-latency response. Control parameters are transferred between the two processors via a UART bus. This architecture allows each processor to be independently selected and optimized based on task characteristics, ensuring real-time interaction while helping to reduce overall system cost and power consumption.

[0116] Please see Figure 2 , Figure 2 This is a schematic block diagram of a computer device provided in an embodiment of this application. The computer device 500 can be a terminal or a server, wherein the server can be a standalone server or a server cluster composed of multiple servers.

[0117] The computer device 500 includes a processor 502, memory, and a network interface 505 connected via a system bus 501. The memory may include a non-volatile storage medium 503 and internal memory 504. The processor 502 may include a neural network processing unit (NPU) for accelerating AI model inference.

[0118] The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. When the computer program 5032 is executed, it causes the processor 502 to execute a smart doll interaction method.

[0119] The processor 502 provides computing and control capabilities to support the operation of the entire computer device 500.

[0120] The internal memory 504 provides an environment for the execution of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute a smart doll interaction method.

[0121] The network interface 505 is used for network communication with other devices. Those skilled in the art will understand that the above structure is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device 500 to which the present application is applied. A specific computer device 500 may include more or fewer components than shown in the figures, or combine certain components, or have different component arrangements.

[0122] The processor 502 is used to run a computer program 5032 stored in a memory to implement the steps of any of the above method embodiments.

[0123] It should be understood that in the embodiments of this application, the processor 502 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0124] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program may be stored in a storage medium, which is a computer-readable storage medium. The computer program is executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.

[0125] Therefore, the present invention also provides a storage medium. This storage medium may be a computer-readable storage medium. The storage medium stores a computer program. When executed by a processor, the computer program causes the processor to perform the steps of any of the above-described method embodiments.

[0126] The storage medium is a physical, non-transient storage medium, such as a USB flash drive, external hard drive, read-only memory (ROM), magnetic disk, or optical disk, or any other physical storage medium capable of storing program code. The computer-readable storage medium can be non-volatile or volatile.

[0127] Furthermore, the above method can be stored in the Flash memory of the embedded device as firmware and executed collaboratively by the NPU and MCU.

[0128] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0129] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of each unit is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0130] The steps in the method of this invention can be adjusted, merged, or reduced in order according to actual needs. The units in the device of this invention can be merged, divided, or reduced according to actual needs. Furthermore, the functional units in the various embodiments of this invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0131] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0132] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0133] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Since these modifications and variations fall within the scope of the claims and their equivalents, this invention also intends to include these modifications and variations.

[0134] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for interacting with an intelligent doll, characterized in that, include: Acquire a sequence of image frames containing the user captured by the camera; Real-time visual semantic analysis is performed on the image frame sequence to extract the user's visual state vector; The visual state vector is mapped to eye expression rendering control parameters and eye viewpoint rendering control parameters; Based on the eye expression rendering control parameters and the eye perspective rendering control parameters, the eye image presented on the smart doll screen is dynamically drawn and updated so that the expression presented in the eye image matches the user's emotional state and the gaze direction presented in the eye image adapts to the user's spatial position.

2. The intelligent doll interaction method according to claim 1, characterized in that, The visual state vector includes age attribute parameters, gender attribute parameters, emotional state parameters, environmental illumination parameters, spatial location parameters, and continuous emotional intensity scores. The spatial location parameters include the user's planar coordinate position and depth distance estimate in the image frame. The step of performing real-time visual semantic parsing on the image frame sequence to extract the user's visual state vector includes: Face detection is performed on each frame of the image frame sequence to obtain a sequence of face bounding box coordinates; Kalman filtering is applied to the face bounding box coordinate sequence for tracking, a unique tracking identifier is assigned to the same user, and a motion trajectory is generated. The image region corresponding to the latest face bounding box in the motion trajectory is extracted, and the image region is input in parallel into an age estimation network, a gender classification network, an emotion recognition network, and an ambient illumination estimation network to obtain the age attribute parameter, the gender attribute parameter, the emotion state parameter, and the ambient illumination parameter, respectively; or, the ambient illumination parameter is obtained by reading the automatic exposure parameters of the camera ISP module; or, the ambient illumination parameter is obtained by performing pixel brightness statistics on the image frame; The continuous emotion intensity score is calculated based on the emotion category confidence vector output by the emotion recognition network. The planar coordinate position is determined based on the geometric center coordinates of the latest face bounding box in the image frame, and the depth distance estimate is calculated based on the ratio of the area of ​​the bounding box to the preset reference area.

3. The intelligent doll interaction method according to claim 2, characterized in that, Mapping the visual state vector to eye expression rendering control parameters includes: The age attribute parameter, the gender attribute parameter, and the emotional state parameter are used as input features and input into the pre-trained eye expression generation model. Through forward reasoning of the eye expression generation model, eye expression rendering control parameters that integrate age features, gender features, and emotional features are output. Alternatively, the age attribute parameter, the gender attribute parameter, and the emotional state parameter can be used as a joint query key to match and search in a pre-built eye expression mapping table to obtain the eye expression rendering control parameters corresponding to the joint query key. The eye expression mapping table stores the mapping relationship between different age groups, genders, and emotional state combinations and corresponding eye expression rendering control parameters. The eye expression mapping table is dynamically updated from the cloud server via an over-the-air (OTA) communication interface.

4. The intelligent doll interaction method according to claim 2, characterized in that, Mapping the visual state vector to eye expression rendering control parameters also includes: Determine whether the ambient illuminance parameter is lower than a preset illuminance threshold; If the ambient illuminance parameter is lower than the preset illuminance threshold, a first eye expression adjustment command is generated. The first eye expression adjustment command is used to increase the diameter of the pupil region in the eye image and increase the highlight reflection intensity of the pupil region. If the ambient illuminance parameter is not lower than the preset illuminance threshold, a second eye expression adjustment command is generated. The second eye expression adjustment command is used to maintain the reference diameter size and reference specular reflection intensity of the pupil region in the eye image. The first eye expression adjustment command or the second eye expression adjustment command is used as a control component of the eye expression rendering control parameters.

5. The intelligent doll interaction method according to claim 2, characterized in that, Mapping the visual state vector to eye expression rendering control parameters also includes: The continuous emotion intensity scores are subjected to time series smoothing filtering to obtain a smoothed emotion intensity curve. Calculate the instantaneous rate of change of the smoothed emotion intensity curve; When the instantaneous rate of change exceeds a preset positive mutation threshold, a dynamic blinking drive signal is generated. The dynamic blinking drive signal is used to drive the eye image to perform a rapid blinking action sequence. The dynamic blinking drive signal is used as a control component of the eye expression rendering control parameters.

6. The intelligent doll interaction method according to claim 2, characterized in that, Mapping the visual state vector to eye-view rendering control parameters includes: Based on the horizontal and vertical offset pixel values ​​of the planar coordinate position relative to the center point of the image frame, calculate the user's first gaze deviation angle in the horizontal direction and the second gaze deviation angle in the vertical direction. Based on the depth distance estimate, calculate the interpupillary distance convergence compensation coefficient between the user and the screen; The first gaze deviation angle, the second gaze deviation angle, and the pupillary distance convergence compensation coefficient are input into the eye rotation model to obtain the first offset vector of the left pupil in the eye image and the second offset vector of the right pupil in the eye image. The first offset vector and the second offset vector are used as the eye view rendering control parameters.

7. The intelligent doll interaction method according to claim 1, characterized in that, Also includes: Acquire the audio stream captured by the microphone, and perform speech recognition processing on the audio stream to obtain the text sequence and speech emotion feature vector; The text sequence is input into the semantic understanding model to obtain the semantic intent encoding; Cross-modal feature alignment and fusion are performed on the user's facial expression features in the visual state vector and the voice emotion feature vector to obtain a multimodal emotion fusion vector; The semantic intent encoding and the multimodal emotion fusion vector are input into the eye expression generation network to obtain multimodal-driven eye expression rendering control parameters; The eye expression rendering control parameters are corrected and updated based on the multimodal driving eye expression rendering control parameters.

8. The intelligent doll interaction method according to claim 1, characterized in that, Before acquiring the sequence of image frames containing the user captured by the camera, the method further includes: The screen of the smart doll is controlled to be in a low-power off state, while the camera is kept capturing environmental images at the first sampling frame rate; The environmental image is subjected to moving target detection. When a moving target is detected, the camera sampling frame rate is increased to a second sampling frame rate, which is greater than the first sampling frame rate. Face detection is performed on the image frames acquired after increasing the sampling frame rate. When the number of image frames that continuously detect face regions reaches a preset frame threshold, it is determined that the user's lingering gaze condition is met. In response to meeting the user's lingering gaze condition, the screen of the smart doll is lit up and the step of performing real-time visual semantic analysis on the image frame sequence is executed; The step of detecting moving targets in the environmental image includes: Frame difference operation is performed on corresponding pixels of adjacent frames in the environmental image to obtain a difference image; The number of pixels in the differential image whose pixel values ​​exceed a first preset threshold is counted. When the number of pixels exceeds a second preset threshold, it is determined that a moving target has been detected.

9. A computer device, characterized in that, The computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, can implement the method as described in any one of claims 1-8.

Images