A flexible robot facial expression generation system driven by a needle screen array

Abstract

This invention discloses a flexible robot facial expression generation system based on a needle-screen array, belonging to the field of machine intelligence. It includes the following modules: a needle-screen array configuration and system initialization module, an expression motion database construction and management module, a needle-screen action primitive sampling and response modeling module, an integrated expression-needle trajectory-residual model training module, an online expression parsing and target trajectory generation module, a distributed execution and closed-loop control module, an online parameter update module, and a health management module. This invention achieves rapid generation of robot facial expressions and generates natural and diverse robot facial expressions by employing a hierarchical action execution and online system self-calibration mechanism, constructing a module-level low-rank residual basis and a three-layer closed-loop control architecture, and introducing dynamic expression primitive modeling and region weight labeling based on an expression motion database. This solves the problems of large expression generation latency and difficulty in coordinating complex expressions in existing technologies.

Description

Technical Field

[0001] This invention relates to the field of machine intelligence technology, and in particular to a flexible robot facial expression generation system based on a needle-screen array driven by a needle screen array. Background Technology

[0002] Facial expressions are an important carrier of human emotions and intentions. Achieving highly realistic facial expressions in flexible robots has always been a key technological direction in fields such as human-computer interaction, social companionship, and medical rehabilitation. To enhance the naturalness and emotional expression capabilities of human-computer interaction, the industry generally hopes that robots can present continuous skin deformation expressions that closely resemble human faces.

[0003] Existing robotic facial expression generation systems typically employ a simulated facial skin or flexible artificial skin on the outer layer of the head, with mechanisms driven by actuators such as servos or joysticks positioned inside the skin to apply mechanical action and induce the deformation required for facial expressions. Servos, as a common actuator, can drive corresponding mechanisms or linkages to achieve facial movements and expression changes. Meanwhile, existing technologies also include methods that introduce tiny airbags into facial mechanisms and coordinate with servos to control these mechanisms and achieve movements. Furthermore, besides directly driving facial expressions with servos, there are also solutions that simultaneously control mechanisms using servos and airbags to achieve facial expressions.

[0004] For example, Chinese invention patent CN117697772B discloses an intelligent flexible bionic facial expression robot, including: a model head, a skeleton, a drive module, and an epidermal layer; one side of the skeleton is attached to one side of the model head, and the skeleton forms a first sliding groove and a first slot; the drive module includes a mounting frame, an eyebrow drive module and an eyelid drive module connected to the mounting frame; the eyebrow drive module includes a first drive source, a first crank and a first rocker arm, the first drive source is connected to the mounting frame to drive the first crank, and the first crank drives the first rocker arm; the epidermal layer is attached to the other side of the skeleton opposite the model head, and the epidermal layer forms a first region and a second region facing the skeleton, the first rocker arm and the second rocker arm both pass through the first sliding groove so that the first contact part is recessed or wrapped in the first region and the second contact part is recessed or wrapped in the second region, when the first drive source drives the first crank, the ends of the first rocker arm and the second rocker arm asynchronously drive the first region and the second region of the epidermal layer to undergo flexible deformation to produce facial expressions.

[0005] The above-mentioned technology has at least the following technical problems: In existing robotic facial expression generation technologies, the inflation and deflation of pneumatic cavities involves a complete dynamic process of pressure build-up and release. The compressibility of the gas itself leads to a significant time lag in pressure changes and transmission. Furthermore, factors such as pipe length, flow rate limitations, and valve switching response speed within the pneumatic system further affect the airflow response efficiency and distribution uniformity. The facial material itself also exhibits elastic hysteresis, rebound, and fatigue effects after multiple deformations, resulting in non-negligible differences in deformation response under the same control command in different cycles. Therefore, in actual expression switching or rapid micro-expression presentation, the system often exhibits insufficiently rapid pressure response and facial deformation dynamics, requiring a certain stabilization time before reaching the target expression form. This leads to a large overall expression generation delay, making it difficult to achieve fast switching and high-precision expression control.

[0006] Meanwhile, for the generation of complex expressions, such as asymmetrical expressions, cross-regional linked expressions, or multiple superimposed actions, existing systems often need to simultaneously control multiple airbags and servo channels to achieve multi-regional linked deformation. Since the flexible skin is a continuous structure, the coupling effect between different regions is significant; the inflation and deflation of any airbag will affect the stress and deformation state of surrounding and even more distant regions. The pressure adjustment between pneumatic cavities and the overall mechanical response of the skin exhibit obvious strong nonlinearity and dynamic mutual influence between channels. As a result, it is difficult to directly and linearly superimpose single or simple control commands to generate complex expressions. The mapping relationship between the actual expression and the expected shape is difficult to establish and maintain stably, easily leading to unpredictability of combined expressions, decreased consistency of repeated calibration, increased difficulty in synchronization and coordination between channels, and decreased control precision of detailed parts. This further results in the technical difficulties of achieving stable reproduction of complex expressions, rapid switching, and high-precision detail restoration. Summary of the Invention

[0007] To address the technical problems of high latency in generating facial expressions for robots, difficulty in automatically compensating for edge effects and system noise, and difficulty in achieving natural and coordinated complex expressions in existing technologies, this invention provides a flexible robot facial expression generation system based on a needle-screen array. The technical solution is as follows: A flexible robot facial expression generation system based on a needle-screen array is provided, comprising: The needle screen array configuration and system initialization module is used to configure high-density and low-density needle screen sub-arrays according to the functional division of facial regions. Each sub-array has an independent structure, driver and interface. During system initialization, standard action parameters are collected to complete the calibration. The facial expression motion database construction and management module is used to collect facial expression motion data, organize it into sample data containing expression type, time, region labeling and weight, store it in a structured manner and label the parameters; The needle curtain action primitive sampling and response modeling module is used to execute needle curtain action primitives in each zone, collect needle-level commands, states and surface deformation observation data, and construct a response model and response dictionary for the relationship between needle-level behavior and local deformation. The integrated model training module for facial expression-needle trajectory-residual is used to train a model that parses facial expression commands into target deformation trajectory, needle-level target displacement trajectory and residual compensation parameters based on facial expression motion database, region weights and response dictionary, and establishes an integrated mapping relationship. The online facial expression parsing and target trajectory generation module is used by the main controller to call the model to generate target deformation trajectory, needle-level target trajectory and residual parameters when the main controller receives facial expression instructions, and organizes them into sub-region instruction data packets; The distributed execution and closed-loop control module is used to send instruction data packets to the controllers of each sub-area module, perform closed-loop control and adjustment according to the needle-level target trajectory, and the main controller arranges multiple expression primitive sequences. The online parameter update module is used to record needle-level status, deformation observation and target trajectory, update control parameters based on historical recursion and maintain status information.

[0008] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following: 1. The flexible robot facial expression generation system based on needle array drive provided by this invention arranges a high-density needle array that can deform bidirectionally in the robot's facial region, uses needle-level linear execution units and position readback sensors to achieve rapid closed-loop control of needle tip normal displacement, and configures a distributed control architecture at the array sub-region level with independent module controllers and main controllers working together. This allows expression commands to be mapped into needle-level target trajectories in real time and converge rapidly in the outer loop of local deformation, thereby significantly shortening the response time required from receiving expression commands to the stable presentation of facial surface deformation. This improves the dynamic bandwidth and rapid switching capability of robot facial expression generation, effectively solving the problem of large delays in expression switching and micro-expression presentation caused by the pressure establishment and release process of pneumatic cavities being limited by gas compressibility, pipeline and valve flow, and material hysteresis and rebound.

[0009] 2. This invention divides the facial region into functional zones and configures needle curtain subarrays of different densities in key expression areas and auxiliary areas respectively. Combined with a module-level response dictionary and low-rank residual basis constructed under the drive of action primitives, the expression-target deformation-needle trajectory integrated model is used to parse the expression type, intensity and regional weight issued by the upper layer into the target surface deformation trajectory and needle-level displacement trajectory of each zone. This enables complex expressions such as asymmetrical expressions, cross-regional linked expressions and multi-action superimposed expressions to maintain spatial coordination, smooth temporal transition and high consistency in repeated execution. In this way, it realizes the stable reproduction and controllable superposition of complex combined expressions on continuous facial skin. It effectively solves the problems of significant regional coupling, strong nonlinearity between channels and dynamic mutual influence when the flexible facial skin is a continuum, which makes it difficult to linearly superimpose combined expressions, reduce the consistency of calibration and repetition, and make it difficult to synchronize coordination and control details. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 A flowchart illustrating a business method for a flexible robot facial expression production system based on a needle-screen array driven, as provided in this application embodiment; Figure 2 A schematic diagram of a flexible robot facial expression production system based on a needle-screen array driven by an embodiment of this application; Figure 3 A structural diagram of a flexible robot facial device based on a needle-screen array driven according to an embodiment of this application; Figure 4 This is a timing diagram illustrating distributed execution and closed-loop control provided in an embodiment of this application. Detailed Implementation

[0012] Embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of the present disclosure are shown in the drawings, it should be understood that embodiments of the present disclosure may be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure.

[0013] It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure. In the description of the embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "this embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc., may refer to different or the same objects.

[0014] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0015] like Figure 1 The diagram shown is a flowchart of a business method for a flexible robot facial expression production system based on a needle-screen array driven according to an embodiment of this application. Figure 2 The diagram shown is a structural schematic of a flexible robot facial expression production system based on a needle-screen array driven according to an embodiment of this application. The system structure and the business methods of each system are as follows: The needle-canopy array configuration and system initialization module is used to partition the flexible robot's face and configure the needle-canopy array for each region based on the partition. For example... Figure 3 The diagram shows the structure of a flexible robot facial device based on a needle-screen array driven according to an embodiment of this application. The flexible robot face consists of a flexible skin layer, a strain sensing layer, a fine mesh conductive layer, a needle-screen driving layer, and a support frame. The flexible skin layer (also known as the biomimetic skin layer) is made of biomimetic silicone, with a thickness of approximately 0.4 mm, and its surface simulates the texture of real skin. The strain sensing layer is composed of a silver fiber conductive fabric mesh, presented in a dashed grid pattern. When the silver fiber conductive fabric is stretched or compressed, its resistance changes approximately linearly with the magnitude of the strain; the greater the strain, the greater the change in resistance. The fine mesh conductive layer is composed of TPU (Thermoplastic Polyurethane) mesh, used as a floating interlayer, with a pore size of 1 mm, presented in a diamond grid pattern. The needle-driven layer consists of an array of micro-actuated needles. To closely resemble a realistic human face and demonstrate a pulling effect, some needles are initially retracted. To protect the flexible skin layer and extend the lifespan of the robot's materials, the micro-actuated needles have spherical tips with a viscoelastic coating, enabling them to generate controllable adhesion to the flexible skin through stretching and contraction. The support frame, the innermost layer of the flexible robot's face, is composed of aluminum alloy or carbon fiber ring frames. It includes specially designed channels for wiring, allowing for the visualization of some wire routes.

[0016] In the mechanical design phase of the robot's face, a unified coordinate system for the robot's head shape and the needle array substrate is established using 3D modeling software. Multiple facial regions, including the brow bone region, periorbital region, nasal alar region, perioral region, cheekbone region, and mandibular region, are marked using curved surfaces in the 3D model. Each region is described by a set of boundary points. Fixed coordinates of the mechanical hole positions of each needle on the needle array substrate are defined and exported as a needle position list. To assign needles to facial regions, a programming tool (e.g., Python) is used to traverse the needle position list, determine whether a needle falls within a certain polygonal region, and write the region number to which the needle belongs into a configuration table.

[0017] For each facial region, an actual deployable area is obtained, denoted as A, in cm. 2 The target needle count N for that area is determined. In key expression areas, such as the brow bone, periorbital area, nasal alar area, and periorbital area, because these areas carry crucial expression details, high-density array sub-areas are set up, with a needle density ≥ 25 needles / cm². 2 In areas that assist facial expressions, such as the cheekbone and jaw areas, because these areas primarily bear auxiliary deformations, i.e., large-scale contour changes, low-density array sub-areas are set up, with a needle density of ≤16 needles / cm in these areas. 2 Considering the limitations of machining, the pinhole array is positioned accordingly. Each zone is implemented through modular subarrays, each equipped with an independent drive and sampling interface for easy zone control and maintenance. It should be noted that the facial expression zones of the robot can be adjusted by technicians according to the actual situation. This embodiment is only an example for reference and does not limit specific facial zones. Similarly, the density of pins in each area can also be adjusted by technicians.

[0018] To achieve both convex and concave deformations in the robotic flexible skin layer, a miniature linear actuator is configured at each needle position, enabling bidirectional displacement of the needle tip along the facial normal. The miniature linear actuator consists of a miniature ball screw and a stepper motor, with the needle body mounted at the actuator's output end to ensure the needle's movement direction aligns with the facial normal. During mechanical design, the zero-position of the needle tip in the undriven state is predefined, making the needle tip essentially flush with the inner surface of the bionic skin. When the actuator drives in the forward direction, the needle tip extends outward relative to the zero position, creating a convex deformation within a preset displacement range, for example, varying from 0 to 1.5 mm. When the actuator drives in the reverse direction, with the cooperation of the compliant layer, the needle tip retracts inward relative to the zero position, forming a localized concavity of a certain depth on the inner surface of the bionic skin, for example, varying from 0 to 1.0 mm. This allows the same needle unit to possess bidirectional deformation capabilities in both forward and reverse directions.

[0019] Next, the controller and communication architecture are initialized. The system first selects the communication bus type (e.g., CAN, RS485, SPI, or Ethernet) between the main controller and each module, and assigns a unique bus address (e.g., CAN identifier, RS485 address, or IP + port) to each subarray module. The main controller maintains a module information list, recording the module's identifier, bus address, region type, and pin count configuration. After system power-on, the main controller sends a broadcast message with the protocol version number and handshake sequence number via the bus. After local initialization, the module controllers return an acknowledgment, reporting their own identifier, firmware version, pin count configuration, and initial health status. Based on this, the main controller generates an online module table, marking online / offline modules, and downgrading or marking unavailable modules that do not respond or are abnormal.

[0020] Subsequently, the main controller and each module complete the communication parameters and synchronization configuration, including negotiating and setting the baud rate, data bits, and parity of the serial bus, constraining the uplink and downlink refresh frequencies and the maximum load length of a single frame, and agreeing on the arrangement order and encoding format of fields such as expression primitive identifiers, trajectory parameters, and residual coefficients in subsequent instruction frames. To ensure the time consistency of the multi-module control cycle, the main controller periodically sends time synchronization frames with absolute timestamps. After receiving these frames, each module controller fine-tunes its local timer to ensure that each module executes the control cycle under a unified time base. It should be noted that the expression primitive identifier is used to indicate a predefined expression control unit. Each expression control unit is obtained by parameterizing and normalizing expression sample data, and this application does not limit the specific construction method.

[0021] After the communication parameters are configured, the main controller establishes a status monitoring channel for each module, enabling each module to periodically report information such as temperature, current, error count, and health status. The main controller dynamically updates the system monitoring interface. If a module fails to report within the specified time or its error count exceeds the limit, it is marked as downgraded or unavailable, and its reliance on the module is reduced or avoided in subsequent expression and trajectory allocation.

[0022] Next, the system sequentially activates preset calibration actions according to the module order, such as single-needle micro-step displacement, synchronous lifting and falling of small cluster needles, and linear gradual deformation. To avoid damage to the flexible skin layer, the displacement amplitude and speed of these actions are limited to a safe range. After power-on, the main controller starts calibration sequentially according to the modules, activating only one or two sub-arrays at a time, while keeping the others stationary to avoid measurement crosstalk caused by simultaneous operation of multiple modules. The system synchronously records the issued needle displacement commands, the actual displacement read back by the sensor, and the surface deformation measurement data, storing them as calibration samples with timestamps. For each needle, the system fits the linear relationship between the command and the actual displacement to obtain the gain and bias parameters. For edge needles and inner needles, it statistically analyzes the difference in deformation response to generate edge weight correction coefficients. At the same time, it fits the conversion relationship between the sensor's original output and the actual displacement and writes it into the conversion parameter table. After completing the parameter estimation, the system writes the parameter set corresponding to each sub-array, including the driving gain and bias of each needle, edge weight coefficients, and sensor conversion parameters, into the module's local storage and the main controller's backup, respectively, as the initial calibration baseline. During normal operation, the needle tip position feedback is uniformly calculated using calibration conversion parameters. The facial expression trajectory planning module considers edge weight coefficients when allocating displacement and compensates for edge areas. The underlying controller automatically applies gain and bias parameters for drive correction when executing servo control.

[0023] The facial expression motion database construction and management module is used to establish a database of human facial expression motions. First, an expression acquisition protocol is established, clearly defining the types of expressions to be acquired, including smiles, frowns, surprise, disgust, sadness, and fear. Intensity levels and movement rhythms are designed for each expression. For ease of understanding, this embodiment uses three levels (strong, medium, and weak) and a 2-second onset, 1-second hold, and 2-second slow fall as examples. Multiple volunteers of different genders and face shapes are organized to perform facial expressions in front of a camera system according to the protocol specifications. The entire process, through parameterized settings (including expression type, intensity, and rhythm), provides a reproducible ground truth for subsequent data acquisition and annotation. The specific acquisition process should be implemented separately by technical personnel based on actual conditions; this embodiment does not describe it in detail. The final acquired complete facial expression motion samples record the entire process from a neutral expression to onset, peak, fall, and finally back to a neutral expression, including information such as the timeline, 3D keypoint coordinates, and region deformation.

[0024] The collected facial expression samples are stored in a structured manner. Each sample should include sample-level metadata and corresponding time-series data. The sample-level metadata mainly includes the sample number and corresponding subject number, expression type and intensity level, key time points of each stage of the expression, and peak duration. The time-series data includes the relative timestamps of each sample, the intensity vectors of each FACS (Facial Action Coding System) action unit, the three-dimensional coordinate data of key points, and the average deformation by region. For facial functional areas, such as the brow bone, periorbital area, nasal alae, perioral area, cheekbone, and jaw, indicators such as maximum normal displacement, normal displacement time curve, and tangential deformation parameters are stored.

[0025] To maintain the biomimetic drive and system training of facial expressions, each expression sample needs to be annotated for executability, including dynamic rhythm parameters and region weight parameters, and the overall executability level needs to be marked. Annotating the dynamic rhythm parameters requires first synthesizing the region deformations into an overall expression intensity curve, denoted as E(t). From the FACS data of a particular expression, weights w are assigned to the main action units mapped to that expression type. n And perform a weighted summation of the action unit intensities of the current frame: Where n represents the nth action unit, AU n E(t) represents the activation intensity of the nth action unit at time t. By analyzing E(t) and its derivative, the onset, peak holding, and fall phases of facial expressions are automatically identified, and parameters such as the duration and maximum rise or fall speed of each phase are extracted and written into the sample record as dynamic rhythm parameters of the sample. These parameters can be directly used to control the time and speed of onset, holding, and fall when the robot generates needle trajectories in the future.

[0026] Perform region weight parameter annotation, and calculate the deformation energy E of each functional region r throughout the entire expression cycle. r And for all regions of E r Normalization: Where, ∆s r (t) represents the normal displacement vector of region r at time t. After normalization, the region weight parameter ω is obtained between 0 and 1. r Furthermore, the sum of the weights of all regions is 1. The region weight parameter is used to guide the resource allocation of the bionic skin drive and key regions. If a certain region has an extremely low weight in most samples, it can be regarded as an auxiliary region, and its priority can be reduced in the device design.

[0027] Based on the limitations of the bionic robot platform, such as maximum deformation and speed, and combined with data quality checks, the system automatically evaluates the geometric and kinematic feasibility of each sample, assigning it a level label such as directly executable, requires scaling, or unusable. When generating bionic facial expression trajectories, the system will prioritize highly executable samples as templates to ensure the naturalness and safety of the generated actions.

[0028] The needle screen motion primitive sampling and response modeling module is used to perform facial expression sampling and construct a response dictionary on the needle screen array. Based on the regional needle density configuration in the needle screen array configuration and system initialization module, a set of standard motion primitives is designed for each array sub-region. These are short, standardized motion segments performed on several needles in each sub-region, with their spatial distribution and temporal rhythm pre-designed to ensure reusability and parameterization. Spatially, the set of standard motion primitives includes single-needle actions (e.g., raising a single needle from 0 to +∆h or pressing it to -∆h and then back to 0 within a short time), small cluster needle actions (e.g., raising or lowering a 3×3 local needle block together), auxiliary regional strip actions (e.g., raising a strip along the cheekbone segment by segment to simulate large-scale contour changes), and cross-regional continuous actions (e.g., continuous bending from the brow bone region to the periorbital region, stimulating inter-regional coupling). Temporally, it includes the start time t0 and the rise time T. r Hold time T h and fallback time T f .

[0029] Within a safe range, the aforementioned action primitives are executed sequentially, and three types of data are simultaneously collected: needle-level control commands, needle-level status readback information, and corresponding facial surface deformation observation data. First, based on the selected expression primitive and instantiated parameters (e.g., needle i, ∆h=0.8mm, action duration T=0.8s), the main controller generates a target displacement trajectory for each needle in the system within a specified time period, denoted as u. i (t m This forms a target displacement command sequence for the needle tip. This sequence is sent to each module controller via the bus and stored in the system as the target control signal for this facial expression action. For needle units that do not participate in this facial expression action, their target command value remains zero to maintain a static state. In actual implementation, the target command sequence can be stored in matrix form, denoted as Cmd[i][m], representing the target displacement value of the i-th needle at the m-th sampling time.

[0030] Next, the needle tip position response sequence is collected. During system operation, each module controller collects and reads the actual displacement of each needle within its jurisdiction through integrated displacement sensors in each control cycle, denoted as y. i (t mThe collected raw data, after being timestamped, is either cached locally or uploaded to the main controller. The actual displacement of each needle over time forms a response time series, which can be organized in a matrix manner and denoted as Pos[i][m], representing the actual displacement of the i-th needle at the m-th sampling time.

[0031] Finally, the skin surface response data is recorded. This part observes the actual deformation of the bionic skin layer after being driven by the needle curtain. The system can use surface displacement sensing arrays, depth cameras, or structured light to observe the normal displacement of the surface under the action of the needles. For example, using a depth camera, the Z-axis displacement corresponding to each functional area is extracted through depth maps or point cloud data, and the measurement results are projected onto each grid point of the array according to the surface array coordinate calibration. To align with the time-series data of the needle array, the coordinates of each sampling point need to be calibrated so that its image or physical coordinates correspond one-to-one with the displacement array or needle array. The collected surface displacement field is stored in a structured data format, denoted as Surface[m][i], which directly corresponds to the surface deformation at the location of each needle, where m is the sampling time and i represents the i-th needle.

[0032] Align all the above data with the same timestamp, and for each primitive execution, form a complete data packet sample, denoted as: These sample data are categorized and stored in the database according to subarrays and primitive types, forming a response dictionary data source.

[0033] Construct a module-level response dictionary. The response dictionary refers to the system's ability to record and retrieve corresponding needle tip position response and skin surface response data for various standard action primitives within each subarray module. Essentially, it maps specific inputs (e.g., action primitives and their parameters) to corresponding outputs (e.g., system response), forming a lookup table or small model library. The specific entry structure includes using module number, action primitive type, and action parameters as search keys, storing statistical characteristics of needle-level responses (e.g., gain, hysteresis, rise or fall time), spatial patterns of surface deformation (e.g., maximum displacement distribution in the region, typical deformation profile), and main dynamic features. During system training, repeated trials can be configured for each type of action primitive and parameter, and multiple sampling samples can be collected. Spatiotemporal averaging or filtering can be applied to the needle-level response and surface deformation to generate typical response curves and typical deformation diagrams, which are then recorded in the response dictionary, providing a data foundation for subsequent modeling and planning.

[0034] A needle-level command-displacement response model is established. This needle-level response modeling consists of two layers: a static mapping and a dynamic model. The static mapping is used to fit the nonlinear relationships such as gain, bias, dead zone, and saturation between the target displacement command and the actual displacement at the needle tip. Linear or piecewise polynomial fitting is used to obtain parameters such as gain, bias, and dead zone threshold, establishing the needle-level static response function. The dynamic model describes the time-series response characteristics of the actual displacement to the target command, including inertia and hysteresis. A first-order or second-order discrete-time model can be used, and parameters are identified through least squares or recursive identification algorithms to obtain the dynamic time constant or system coefficients. Finally, the corresponding static and dynamic parameters for each needle are saved and organized into a response model parameter set. During runtime, the target command outputs the response prediction result after static compensation and dynamic iterative calculation, realizing needle-level closed-loop control and system simulation.

[0035] Furthermore, the system constructs a low-rank residual basis to efficiently express and compensate for the difference between ideal and measured deformation. Based on the established needle-level response model and the linear superposition assumption, the ideal surface deformation distribution for each type of action primitive is first calculated. Then, the actual surface deformation is obtained using real sensor data, and the residual field between the ideal and measured values ​​is calculated. The residuals at each time step and for each primitive are vectorized to form a large-scale residual sample matrix. Principal component analysis is then used to reduce the dimensionality of the matrix, and the first K principal basis vectors (generally 3-8 are selected) are extracted to form the low-rank residual basis, denoted as {φ1, ..., φ...}. K Thus, for any new residual field r(x, y), we can obtain: Where k is the number of the low-rank residual basis, c k The residual field represents the residual field in the k-th typical residual mode φ k The magnitude of the components is used to weight φ k The coefficients φ of (x, y) k (x, y) represents the value of the k-th low-rank residual basis at coordinates (x, y). To account for the anomalous behavior of edge units, a separate edge-specific residual basis can be extracted for regions with significant edge effects. In practice, this is denoted as r(x, y, l), where l is the number of the edge-specific basis, similar to the definition of k, but used specifically for edge anomalies.

[0036] In practical applications, the new residual field can be approximated by a linear combination of the basis. Only a small number of coefficients are needed to describe most of the non-idealities and local differences, thus achieving efficient compensation and adaptive optimization.

[0037] The integrated model training module for facial expression, needle trajectory, and residual is used to train the target facial expression-needle trajectory model, needle trajectory model, and residual compensation coefficient prediction model based on the facial expression motion database construction and management module and the needle motion primitive sampling and response modeling module.

[0038] First, based on the facial expression motion database, a target facial expression deformation trajectory model is trained; for ease of explanation, this model will be referred to as Model A below. Model A uses a machine learning-based prediction model to train and fit the mapping relationship between facial expressions and target deformation trajectories. The machine learning model can be freely chosen by technicians according to the actual situation; selectable models include, but are not limited to, neural network models and regression models. This embodiment does not restrict the choice of model type. For Model A, the input data includes facial expression type, facial expression intensity level, action duration parameters (i.e., the start, hold, and fall times), and regional weight vectors (including weights for areas such as the brow bone, eye area, nose wing, and mouth area). Using supervised learning, the model learns the mapping rules from facial expression command parameters to facial expression deformation trajectories, outputting a target facial expression deformation trajectory that conforms to natural laws and personalized settings across the entire facial surface. The trajectory can be a full-time displacement sequence for each grid point or a coefficient sequence of 3 to 8 deformation basis functions. For ease of explanation, this embodiment uses the deformation basis function form as an example, and does not impose constraints on the form or number of output trajectories.

[0039] Secondly, based on the target surface deformation trajectory just obtained, a needle trajectory model is trained, which will be referred to as Model B below for ease of explanation. Similarly, Model B uses a machine learning-based prediction model to train and fit the nonlinear inverse mapping relationship between the target deformation trajectory and the needle trajectory. For Model B, the local target deformation trajectory of the input array sub-region and the state parameters of the current sub-region (including calibration parameters, edge weights, maximum displacement, and maximum velocity, etc., physical constraints) are used as sample data. Also using supervised learning, the model is trained using a large number of samples of target deformation to needle trajectory, so that the model automatically fits the nonlinear inverse mapping relationship between the local target surface motion and the needle-level motion, and outputs the target displacement time sequence (including forward deformation and reverse deformation) of each needle in the array sub-region as well as the control parameters such as velocity and acceleration of each needle.

[0040] Finally, to ensure stable facial expression control of the robot despite unavoidable changes such as hardware aging and material relaxation, a residual coefficient compensation prediction model is trained. For ease of explanation, this model will be referred to as Model C below. Model C uses a machine learning-based regression model to train the residual compensation coefficients. The regression model can be freely chosen by the technicians based on the actual situation. Possible models include, but are not limited to, multilayer perceptron models and lightweight temporal network models. This embodiment does not restrict the choice of model type. For Model C, the current expression command parameters (including expression type, expression intensity, etc.), the target deformation features of the current cycle, and the state information of the current array sub-region (including temperature, recent error statistics, etc.) are input as sample data. Supervised learning is also used. The model is trained using the residual basis coefficients calculated during historical executions and outputs the compensation coefficient vector of each array sub-region on the low-rank residual basis r(x, y), denoted as: Thus, Model C can achieve adaptive prediction and real-time compensation for expected system errors, thereby achieving adaptive facial expression precision control.

[0041] The online expression parsing and target trajectory generation module receives expression commands from the upper layer and generates target trajectories and residual predictions via models A, B, and C in the expression-trajectory-residual integrated model training module. To reduce complexity, the system establishes a preset shape base library in the preset stage, representing the surface deformation at any given time using a weighted combination of basis functions and their corresponding weight factors. Based on the expression commands sent to the controller from the upper layer (including the target expression type, expected expression intensity, initial arrival speed, holding time, fall speed, and weight ratio of different facial regions), the system uniformly plans the entire expression action process as a normalized time process, divided into stages such as the initial movement, holding, and fall. At each time point, the system takes the above expression commands and time process as input, performs inference through model A, and outputs the coefficient sequence of each deformation basis function corresponding to the current time. Combining the preset shape base library and the coefficient sequence of the deformation basis functions, the system can reconstruct the target deformation sequence of the entire facial surface at different time points. This sequence is the global target surface deformation trajectory, which covers the entire expression action process and is used to further execute subsequent steps.

[0042] Because the robot's face is divided into several drive array sub-regions, each sub-region is responsible for controlling the deformation of the corresponding part of the face. Furthermore, during system deployment and initialization, the spatial positional relationship of each drive needle and each group of needles on the robot's face has been precisely calibrated. After Model A outputs the complete global deformation trajectory, the system extracts the local surface deformation trajectory covered by each sub-region, preparing for subsequent independent control of the sub-region. For each drive needle, based on its precise position in space, Model B outputs the target displacement time sequence at each moment, i.e., the motion trajectory that the needle should achieve throughout the entire facial expression cycle. Finally, all needles in each sub-region obtain a set of target displacement time sequence sequences, serving as input for the next level of trajectory planning and physical constraints. In addition, in practical applications, Model B typically performs further data dimensionality reduction or keyframe extraction on the local deformation trajectory of each sub-region, and combines this with the current state parameters of the region (such as gain compensation, edge special processing, health status monitoring, etc.) to finally output the baseline motion trajectory of each needle. These trajectories can be target displacement points given directly in time series, or they can be given in a trajectory parameterized manner (such as key control points of splines or Bezier curves) to facilitate subsequent smoothing and constraint processing.

[0043] To further enhance the precision and adaptability of facial expression rendering, the system uses Model C to predict and correct residual coefficients to address the discrepancy between the execution results of each array sub-region and the actual skin deformation. First, the system collects target deformation data and actual skin deformation data from sensors within each facial expression cycle, comparing them at spatial sampling points and corresponding time points in each array sub-region to obtain a spatial residual distribution covering the entire cycle or key stages. This residual reflects the subtle differences between the target action and the actual execution. To improve the stability and representativeness of the residual representation, technicians can select peak moments of facial expressions, energy statistics, or other representative methods to compress the residual data, obtaining the periodic deformation residual field of each sub-region within the current cycle. For ease of subsequent data processing, the system transforms the residual field into a fixed-length residual vector according to a pre-defined spatial sampling order. Based on this vector and the low-rank residual basis obtained from offline training, the system further decomposes the residual vector and projects it onto several typical residual distribution bases. The resulting set of coefficients characterizes the strength of the residual components in each typical residual deformation mode within the current cycle. Next, the obtained residual coefficients are input into model C. Combining historical trends and system operating status information, the current residual coefficients are dynamically corrected and short-term predictions are made. Finally, the system packages the current period's facial expression primitives, needle-level baseline trajectory parameters, and corrected low-rank residual coefficients together to form a standardized module instruction data package, which is then sent to the controller of the corresponding array sub-region module.

[0044] The distributed execution and three-layer closed-loop control module is used to distribute module instruction packets from each array sub-region to the corresponding module controllers according to the communication protocol, and to perform needle-level inner-loop control, module-level deformation outer-loop control and residual application, as well as expression arrangement and multi-primitive fusion. For example... Figure 4 The diagram shown is a timing diagram of distributed execution and closed-loop control provided in an embodiment of this application. The specific control process is as follows.

[0045] For needle-level inner-loop control, after receiving the instruction packet from the main controller, the module controller first parses the current expression primitive identifier, the reference motion trajectory parameters of each needle, relevant physical constraints (such as maximum speed, acceleration, rate of change, etc.), and the residual compensation coefficient for the current cycle. Then, according to the set sampling period, the controller interpolates and flattens the trajectory parameters to generate the target displacement sequence of each needle at each sampling time, while ensuring that the trajectory meets all physical constraints. Within each sampling period, the controller collects measurement data from the needle tip position sensor in real time, and after self-calibration parameter and environmental compensation processing, obtains accurate actual displacement values. The system compares the target displacement with the actual displacement, calculates the displacement error in real time, and uses it as feedback input for closed-loop control. Simultaneously, based on the needle-level target trajectory and error signal, a composite control law (such as feedforward and proportional-integral-derivative algorithms) is used to independently drive the actuator of each needle. The output drive command undergoes amplitude limiting and filtering to prevent instantaneous impacts and abnormal vibrations. Furthermore, the control parameters can be adaptively fine-tuned based on regional attributes and historical performance. If the displacement reading remains stationary for multiple consecutive sampling periods, the abnormal needle will be automatically marked as faulty, and amplitude degradation or freezing measures will be implemented, preventing it from participating in subsequent deformation tasks.

[0046] For module-level deformation outer-loop control and residual application, within each module's outer-loop control cycle, the controller collects surface deformation observation data of the facial area covered by the module, and simultaneously acquires the target deformation reference value. By comparing the actual and target deformations, the system calculates the spatially distributed deformation residuals, projects the deformation residual vectors onto a pre-trained low-rank residual basis, extracts the coefficients of the main residual modes, and performs smoothing and fusion processing within the module controller to reduce the impact of short-term fluctuations. Based on the residual modes and the needle-surface response relationship, the controller converts the regional residuals into trajectory correction amounts for each needle. The correction magnitude in key areas can be weighted and amplified, while auxiliary areas are appropriately suppressed, ensuring that limited driving capabilities are prioritized for facial expression detail repair. Simultaneously, the needle-level target trajectory is fine-tuned in real time accordingly, enhancing the dynamic consistency and naturalness of the entire area. Model C pre-corrects the residual coefficient c and predicts the residual trend based on the current expression type and historical execution results, improving deformation accuracy and consistency without altering the upper-level expression logic.

[0047] This involves facial expression orchestration and multi-primitive fusion, specifically temporal orchestration and spatial weighting at the main controller level. Each facial expression primitive can be understood as a short, parameterized action segment, including primitive ID, temporal parameters, spatial parameters, expression intensity parameters, and an associated facial expression generation model index. For example, there's a smiling primitive with an intensity of 0.7 between 0 and 0.8 seconds, and a raised eyebrow primitive with an intensity of 0.5 between 0.2 and 0.5 seconds. The main controller, based on presets or user input, sequentially superimposes multiple facial expression primitives along a unified global timeline. Each primitive generates a weight function within its effective range to regulate its participation throughout the entire action cycle. At each moment, the system weights and superimposes the outputs of all effective primitives to form a continuous and smooth target facial expression trajectory. To achieve unilateral expressions and asymmetrical facial expressions, independent regional weights and rhythms are set for each facial region within each facial expression primitive. Finally, the region-level deformation trajectory is converted into the trajectory of each subarray and each needle by Model B and encapsulated in an instruction package before being sent to each module.

[0048] The online parameter update module is used to update some module-level parameters in real time based on observation data during normal execution. During the robot's facial expression actions, the module's main operational data is collected in real time, including the actual position feedback of each needle, the target trajectory, deformation measurement data of the surface area covered by the module, the current module-level parameter values, and operational status information such as temperature and current. This data is stored in a local buffer using a sliding time window. Then, according to a preset period (e.g., every N facial expressions or every T seconds), the cached data is periodically filtered, automatically removing data segments with saturation or abnormal signals. For valid data intervals, the system statistically analyzes features including average tracking error, regional error distribution (i.e., the ratio of the average error of the edge region to the center region), and the long-term offset of the residual parameter c. These statistics serve as the observation basis for module-level parameter updates.

[0049] For each module-level parameter, simple parameter estimation models such as step correction, first-order filtering, or recursive least squares are used for adjustment. For example, when a large long-term tracking error is detected in the overall or peripheral regions, the gain or weight coefficient is automatically increased, and vice versa. All parameter updates are recursively applied in small steps at low frequency, with upper and lower limits set to prevent abrupt changes in parameters, and updates are only performed when the data quality is reliable and the stimulus is sufficient (e.g., the facial expression amplitude exceeds a certain threshold, and there is no prolonged saturation).

[0050] The system sends the updated parameters to the module. After the module controller completes the parameter update, it writes the latest parameters into the local parameter table and marks a new parameter version number. The controller automatically reads the latest parameters in subsequent execution cycles, thus achieving a smooth transition. It should be noted that this embodiment only uses local adaptive updates as an example. In actual operation, technicians can freely choose the parameter update method according to the actual situation, and this embodiment does not impose any restrictions on this.

[0051] Example 2, building upon Example 1, addresses the issue that during long-term operation, the performance of some needles or modules may gradually decline due to factors such as unit aging, environmental drift, and local anomalies, affecting overall facial expression consistency and system availability. Therefore, a health management module is added to the online parameter update module to ensure long-term stable operation and self-diagnosis of the flexible robot facial expression generation system. A health index is introduced, maintaining the health of each needle and module separately. Needle-level health includes tracking error statistics, lag time counts (i.e., the needle's displacement reading remains stationary for a period of time), drive saturation counts, and the condition of operating temperature approaching the upper limit. Module-level health includes the needle-level health of all needles within the module, sensor normality (e.g., displacement sensor failure), and the long-term convergence of regional residuals. For continuous indicators reflecting health (such as average error and temperature rise margin), the system uses an exponential moving average method for smooth updates, ensuring that the health score reflects the current long-term operating trend. For discrete events (such as jams, overload, etc.), a counting and decay strategy is adopted. Each event is counted and accumulated, and the count of historical events is periodically decayed to prevent early anomalies from having a long-term impact on the health score. All health-related indicators are normalized and then synthesized into a final health score according to preset weights, confined within a standardized scoring range. Module-level health scores can be flexibly determined based on the weighted average, minimum value, or other synthesis methods of the health scores of each needle within the module.

[0052] When the health of a needle or module is detected to be below a preset threshold, the system automatically implements a degradation control strategy. For example, for needles with significantly abnormal health, the system proactively reduces their participation weight in the surface deformation closed loop, and may even exclude them from the local solution and control sequence. Simultaneously, the control amplitudes of their maximum displacement and velocity are tightened to prevent further damage. For modules with significantly reduced health, the system can reduce their contribution to facial expression amplitude during the expression primitive fusion stage and dynamically allocate key facial expression details to neighboring modules with higher health, maximizing the stability and consistency of the overall facial expression output. Furthermore, the system proactively reports information on modules with abnormal health, providing a basis for operation and maintenance, and preventing data from faulty modules from continuously affecting the parameter update process.

[0053] Through the above description of the implementation methods, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the above functions can be divided into different functional modules to complete all or part of the functions described above.

[0054] In the embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between devices or units, and may be electrical, mechanical, or other forms.

[0055] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units, located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0056] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0057] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to the solution, or all or part of the technical solution, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0058] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A flexible robot facial expression generation system based on a needle-screen array, characterized in that, include: The needle screen array configuration and system initialization module is used to configure high-density and low-density needle screen sub-arrays according to the functional division of facial regions. Each sub-array has an independent structure, driver and interface. During system initialization, standard action parameters are collected to complete the calibration. The facial expression motion database construction and management module is used to collect facial expression motion data, organize it into sample data containing expression type, time, region labeling and weight, store it in a structured manner and label the parameters; The needle curtain action primitive sampling and response modeling module is used to execute needle curtain action primitives in each zone, collect needle-level commands, states and surface deformation observation data, and construct a response model and response dictionary for the relationship between needle-level behavior and local deformation. The integrated model training module for facial expression-needle trajectory-residual is used to train a model that parses facial expression commands into target deformation trajectory, needle-level target displacement trajectory and residual compensation parameters based on facial expression motion database, region weights and response dictionary, and establishes an integrated mapping relationship. The online facial expression parsing and target trajectory generation module is used by the main controller to call the model to generate target deformation trajectory, needle-level target trajectory and residual parameters when the main controller receives facial expression instructions, and organizes them into sub-region instruction data packets; The distributed execution and closed-loop control module is used to send the instruction data packet to the controllers of each sub-area module, perform closed-loop control and adjustment according to the needle-level target trajectory, and the main controller arranges multiple expression primitive sequences; The online parameter update module is used to record needle-level status, deformation observation and target trajectory, update control parameters based on historical recursion and maintain status information.

2. The flexible robot facial expression generation system based on needle array drive as described in claim 1, characterized in that: The needle screen array configuration and system initialization module includes: The robot's face is logically divided into multi-functional regions. These functional regions are used to distinguish facial regions corresponding to different facial expression characteristics. High-density needle screen sub-arrays are configured for key expression regions, and low-density needle screen sub-arrays are configured for auxiliary expression regions. This ensures that the needle screen sub-arrays of each functional region are independent in terms of structural scale, drive control unit, and modular management method. Each sub-array is configured with an independent drive and sampling interface, supporting self-calibration and parameter updates under the standard action sequence during the initialization phase.

3. The flexible robot facial expression generation system based on needle-screen array drive as described in claim 1, characterized in that: The facial expression motion database construction and management module includes: Facial motion data is collected based on at least one facial motion representation method, such as FACS action unit, three-dimensional key point temporal sequence, and surface displacement field. The facial motion data is then converted into a multi-dimensional temporal sample that includes at least facial expression identifier, time parameter, region marker and dynamic rhythm parameter. Index information including facial expression type, time feature and region information is established for each sample record. Region weight parameters are generated for different facial regions, and these region weight parameters are associated with and stored with corresponding expression samples for use by the expression-needle trajectory-residual integrated model training module during training.

4. The flexible robot facial expression generation system based on needle array drive as described in claim 1, characterized in that: The needle screen motion primitive sampling and response modeling module includes: Within each array sub-region, a response model between needle-level commands and displacements is established based on needle-level control commands and needle-level status readback information, according to preset standard action primitives. A response dictionary is also established based on needle-level actions and facial surface deformation observation data to describe the local deformation distribution. The response dictionary is used to distinguish the response characteristics of key regions and auxiliary regions, and internal and edge units.

5. The flexible robot facial expression generation system based on needle array drive as described in claim 1 or 4, characterized in that: The needle screen motion primitive sampling and response modeling module also includes: The surface deformation error information in each array sub-region is spatially decomposed to extract a set of deformation basis functions to characterize the local deformation error distribution and edge effects. The set of deformation basis functions is then used as the representation basis for residual information and provided to the expression-needle trajectory-residual integrated model training module and the online expression parsing and target trajectory generation module.

6. The flexible robot facial expression generation system based on needle array drive as described in claim 1, characterized in that: The integrated training module for the expression-needle trajectory-residual model includes: Based on the facial expression motion database and the response dictionary, a unit is trained to parse facial expression commands into target deformation trajectories, a unit is trained to generate needle-level target displacement trajectories based on the target deformation trajectory and array sub-region state information, and a unit is trained to generate residual compensation parameters based on the target deformation trajectory, array sub-region state information, and deformation basis function set.

7. The flexible robot facial expression generation system based on needle-screen array drive as described in claim 1, characterized in that: The online facial expression parsing and target trajectory generation module includes: Based on the current period's facial surface deformation observation data and the previous period's execution results, the deformation residuals of each array sub-region are calculated based on the integrated model. The deformation residuals are then projected onto the corresponding set of deformation basis functions to obtain residual coefficients, forming an instruction data packet containing expression primitive identifiers, needle-level target displacement trajectory parameters, and residual coefficients.

8. The flexible robot facial expression generation system based on needle array drive as described in claim 1, characterized in that: The distributed execution and closed-loop control module includes: Within each module controller, the needle-level target displacement trajectory is reconstructed within the array sub-region based on the instruction data packet. Closed-loop control is performed on the needle level, and the difference between the local deformation observation data and the target deformation trajectory is periodically collected during the execution process. The needle-level target displacement trajectory is then corrected in real time based on the residual coefficient.

9. The flexible robot facial expression generation system based on needle-screen array drive as described in claim 1, characterized in that: The online parameter update module includes: During the execution of facial expressions, the system continuously records historical execution data related to needle-level state readback, local deformation observation, and target trajectory. Based on the historical execution data, the parameters related to array control are periodically updated using a recursive algorithm to form a parameter configuration that changes over time.

10. The flexible robot facial expression generation system based on needle-screen array drive as described in claim 1, characterized in that: The system further includes a health management module, which is used to perform real-time statistics and evaluation of the status data of the needle curtain driving unit during operation, and dynamically adjust the participation mode of the corresponding needle curtain driving unit in the overall drive control when an abnormality is detected, so as to support the system's operational health management and the redistribution of deformation tasks.

Images