User state recognition and behavior decision method and system based on end-cloud cooperation

Abstract

The application discloses a user state recognition and behavior decision method and system based on end-cloud cooperation, belongs to the field of robot intelligent interaction and end-cloud cooperative control, and aims at solving the problem that feedback lag or offline interruption is caused by the fact that the end-side computing power of a bionic accompanying robot is limited and the cloud side is dependent. Through end-side collection and recognition of multi-modal states, end-cloud shunting and gating, cloud-side fine understanding and strategy optimization, cooperative result verification and execution, and offline cache synchronization, the technical effects of low-delay basic accompanying, complex state understanding, long-term relationship modeling and weak-network continuous interaction are achieved.

Description

Technical Field

[0001] This invention relates to the field of robot intelligent interaction and edge-cloud collaborative control, and in particular to a method and system for user state recognition and behavior decision-making based on edge-cloud collaboration. Background Technology

[0002] With the development of bionic companion robots, edge perception models, and cloud-based intelligent services, companion robots are gradually transforming from simple touch feedback or voice wake-up devices into interactive terminals capable of sensing user proximity, touch, voice, posture, sleep, and basic emotions. To enable bionic companion robots to provide a more continuous companionship experience, existing solutions typically require distributing state recognition, semantic understanding, historical memory, and behavioral decision-making tasks between the robot itself and the cloud service.

[0003] However, if existing solutions primarily rely on local computation on the robot's edge, they are limited by the robot's computing power, storage space, model size, and historical data processing capabilities, making it difficult to achieve complex emotion understanding, long-term user state modeling, relational memory retrieval, and personalized companionship strategy optimization. If existing solutions primarily rely on cloud-based recognition and decision-making, they are prone to feedback delays, behavioral interruptions, or decreased companionship capabilities when the network is unstable, the cloud response is delayed, or the robot is offline. Furthermore, if cloud results are not validated in conjunction with the current edge state, the robot's current state, and the validity period of the results, expired strategies may overwrite immediate feedback or cause abrupt changes in behavioral output.

[0004] Therefore, there is a need for a user state recognition and behavior decision-making method and system that can overcome the shortcomings of the existing technologies. Summary of the Invention

[0005] One objective of this invention is to propose a user state recognition and behavior decision-making method and system based on edge-cloud collaboration. Addressing the limitations of existing technologies where local computing on the edge side struggles to handle complex state understanding and long-term relationship modeling, while centralized cloud processing is susceptible to network latency and offline status, this invention proposes a technical solution combining edge-side multimodal acquisition and coarse-grained recognition, edge-cloud traffic gating, cloud-side fine-grained understanding and strategy optimization, validity period and matching verification of collaborative results, and offline behavior maintenance and recovery synchronization. Specifically, this invention improves real-time response capabilities, complex state understanding capabilities, personalized strategy adaptation capabilities, and continuous support capabilities in weak network conditions.

[0006] This invention provides a user state recognition and behavior decision-making method based on edge-cloud collaboration, including:

[0007] S1. Collect multimodal data of user interaction with companion robot and robot current state data, and obtain end-side multimodal features, robot current state parameters and end-cloud collaborative control parameters through time alignment, normalization and feature extraction;

[0008] S2. Perform coarse-grained user state recognition based on the multimodal features of the device side, generate the device side real-time state, local state label, state evaluation parameters and basic behavior strategy, and output basic companion behavior;

[0009] S3. When the edge-cloud collaborative control parameters meet the cloud-side processing conditions and the state evaluation parameters indicate that cloud-side processing needs to be called, generate the data packet to be uploaded through the edge-cloud diversion gating and send it to the cloud side.

[0010] S4. Based on the data packet to be uploaded, historical interaction records, long-term user state model, relational memory and robot current state parameters, the cloud side generates the cloud side's fine state and collaborative processing results.

[0011] S5. The edge performs validity period verification and matching verification on the collaborative processing results, and generates or adjusts the collaborative behavior strategy based on the verification results, combined with the edge's real-time status, the robot's current state parameters, and the collaborative processing results.

[0012] S6. When the edge-cloud collaborative control parameters indicate network anomalies, upload restrictions, or unavailability of collaborative processing results, the edge maintains basic companion behavior and synchronously caches records to update the long-term user state model and relation memory after network recovery.

[0013] Optionally, S1 includes:

[0014] The camera captures images of facial expressions, gaze direction, and approach / departure trajectories.

[0015] Acquire wake words, speech text, and acoustic features through the microphone;

[0016] The touch sensor acquires the touch area, touch duration, touch rhythm, and touch pressure.

[0017] The states of being picked up, put down, shaken, inverted, and rolled are obtained through attitude sensors and inertial measurement units;

[0018] Millimeter-wave radar is used to acquire characteristics of changes in respiration, heart rate, presence, and distance.

[0019] Supplement the time period, device mode, and scene context based on application interaction data and environmental data;

[0020] The robot's current state data is obtained through the robot body controller, actuator feedback unit, and energy management unit.

[0021] Optionally, S2 includes:

[0022] The edge-side multimodal features are input into the edge-side state recognition model and rule engine. At least one of the following methods is used to identify user states: proximity state, interaction state, wake word state, simple semantic state, touch event state, holding posture state, pre-sleep state, night wake state, and basic emotional state.

[0023] The local status label is generated based on the coarse-grained user status.

[0024] Based on model output probability, multimodal consistency, sensor integrity, scene risk level, and rule consistency, edge identification confidence parameters and state uncertainty parameters are generated, and the state evaluation parameters are composed of the edge identification confidence parameters and the state uncertainty parameters.

[0025] A multidimensional set of behavioral parameters is generated based on the coarse-grained user state, the local state label, the robot's current state parameters, the locally cached relational memory summary, and the behavioral constraint parameters, and the basic behavioral strategy is generated based on the multidimensional set of behavioral parameters.

[0026] The locally cached relation memory digest is obtained from one or more updates among the historical relation memory digests cached on the edge, the relation memory digests that have recently passed the validity period verification and matching verification, or the synchronous records after network recovery. The behavior constraint parameters are generated based on the robot's current state parameters, scene context, and local safety rules.

[0027] Optionally, S3 includes:

[0028] The state evaluation parameters, network connection parameters, power supply status parameters, upload permission parameters, and scene task identifier input terminal cloud diversion gating module are used, wherein the state uncertainty parameter is calculated based on multimodal consistency, model output confidence, sensor integrity, scene risk level, historical strategy effect, and network resource status, and the preset diversion threshold is greater than the preset local execution threshold.

[0029] When the state uncertainty parameter does not exceed the preset local execution threshold and the local state label directly matches the basic behavior mapping table, the end-side immediate execution path is adopted.

[0030] When the state uncertainty parameter does not exceed the preset local execution threshold but the local state label cannot be directly matched with the basic behavior mapping table, or when the state uncertainty parameter is greater than the preset local execution threshold but does not exceed the preset diversion threshold, no multimodal conflict occurs and the current interaction request does not need to call the long-term user state model or the relation memory, a conservative execution path on the client side is adopted.

[0031] When the state uncertainty parameter exceeds the preset diversion threshold, a multimodal conflict occurs, or the current interaction request needs to call the long-term user state model or the relation memory, the data packet to be uploaded is generated.

[0032] The system dynamically adjusts subsequent upload strategies based on historical upload results, model inference performance, user feedback, and runtime resource status.

[0033] Optionally, S4 includes:

[0034] The cloud side inputs the data packet to be uploaded into the multimodal sentiment understanding model and the language understanding model, wherein the data packet to be uploaded includes the scene task identifier;

[0035] By combining the long-term user state model associated with user identifiers, the relational memory, and the robot's current state parameters, a cloud-side fine-grained state is generated, which includes emotional state, interaction willingness state, relational stage state, and behavioral preference state.

[0036] Based on the difference between the cloud-side fine state and the edge-side real-time state, the edge-side real-time state is corrected, supplemented, or refined to generate the state correction result.

[0037] Based on the historical interaction records, user feedback records, and the effect data after the execution of the companionship behavior, a behavior asset library is formed, and the strategy incremental results used to adjust the basic behavior strategy are generated.

[0038] The behavioral asset library includes one or more of the following: verified companionship behavior templates, behavioral parameter combinations, behavioral rhythm templates, state mapping templates, or behavioral combination templates.

[0039] Based on the relational memory, extract the relational memory summary applicable to the current interaction task, and generate the scene constraint parameters based on the scene task identifier and the cloud-side fine-grained state;

[0040] The collaborative processing result is composed of the cloud-side fine-grained state, the state correction result, the policy incremental result, the relation memory summary, the scenario constraint parameters, the return timestamp, the validity period identifier, and the applicable state identifier, and the collaborative processing result is sent to the edge side.

[0041] Furthermore, the strategy increment result is used to adjust the multidimensional behavior parameter set based on the real-time state of the end-side, the current state parameters of the robot, the locally cached relational memory summary, and the behavior constraint parameters, and in combination with the relational memory summary. The multidimensional behavior parameter set includes one or more of the following: eye state parameters, eye animation parameters, eye emotion parameters, voice content parameters, voice output level, purring content parameters, purring rhythm, respiratory fluctuation amplitude, heart rate rhythm, kneading frequency, kneading duration, kneading force, head movement amplitude, tail movement frequency, tail movement amplitude, limb movement parameters, trunk posture adjustment parameters, movement control parameters, light brightness, or system mode switching parameters.

[0042] The endpoint calls the corresponding behavioral asset in the behavioral asset library based on the incremental result of the strategy, and performs dynamic adjustments on the called behavioral asset by combining the endpoint's real-time state and the robot's current state parameters.

[0043] Optionally, S5 includes:

[0044] Determine whether the collaborative processing result is within a preset validity period based on the returned timestamp and validity period identifier in the collaborative processing result;

[0045] When the collaborative processing result is not within the preset validity period, the collaborative processing result is discarded or downgraded.

[0046] When the collaborative processing result is within the preset validity period, the applicable status identifier in the collaborative processing result is matched and verified with the local status label at the current time on the terminal.

[0047] When the applicable state identifier is consistent with the local state label, the cooperative behavior strategy is executed according to the end-side instantaneous state, the robot's current state parameters, the relational memory summary, and the scene constraint parameters.

[0048] When the applicable state identifier is partially consistent with the local state label, only the preset conservative parameter corresponding to the partially consistent state is called.

[0049] When the applicable status identifier is inconsistent with the local status label, the collaborative processing result is discarded or downgraded.

[0050] When generating or adjusting the collaborative behavior strategy, the robot's current state parameters are used as a mimicry life state to participate in the decision-making. When the robot's current state parameters conflict with the edge-side instantaneous state or the cloud-side fine-grained state, a comprehensive decision is made based on the robot's current state parameters, the edge-side instantaneous state, the cloud-side fine-grained state, the relationship stage state, the scene constraint parameters, and the behavior priority rules, and one or more of the following processing methods are executed: constraint, degradation, delayed execution, partial execution, alternative execution, or prohibition of execution.

[0051] Furthermore, the robot's current state parameters include one or more of the following: mimicry emotion state, activity state, system state, energy state, safety state, interaction mode state, and hardware state.

[0052] The mimicry life state is used to determine response willingness parameters, interaction intensity parameters, and behavioral rhythm parameters;

[0053] The response willingness parameter, the interaction intensity parameter, and the behavior rhythm parameter are used to constrain, downgrade, enhance, delay, partially execute, substitute, or prohibit the execution of the collaborative behavior strategy or the basic behavior strategy.

[0054] Optionally, S6 includes:

[0055] When the connection status of the network connection parameter in the cloud-edge collaborative control parameters is disconnected, the upload permission parameter indicates that uploading is prohibited, or the collaborative processing result is not returned within the preset validity period, an offline behavior queue is generated based on the most recent collaborative behavior strategy that passed the validity period verification and status matching verification, the local rule table, the cached user preference records, and the locally cached behavior assets.

[0056] When a wake word event, touch soothing event, hugging event, or sleep-related event is detected, the basic companionship behavior corresponding to the local status tag is preferentially called from the offline behavior queue. The sleep-related event includes at least one of the following: bedtime event, falling asleep event, nighttime wake-up event, morning wake-up event, or sleep mode switching event.

[0057] After the network connection is restored, the status judgments, interaction records, behavior execution results, and user feedback records generated during the offline period will be synchronized to the cloud side in chronological order.

[0058] On the other hand, the present invention also provides a user status recognition and behavior decision-making system based on edge-cloud collaboration, including:

[0059] The multimodal and robot state acquisition module is used to collect multimodal data of user-companion robot interaction and robot current state data, and generate end-side multimodal features, robot current state parameters, and end-cloud collaborative control parameters; the end-side coarse-grained recognition module is used to generate end-side instantaneous state, local state label, end-side recognition confidence parameter, state uncertainty parameter, multi-dimensional behavior parameter set, and basic behavior strategy based on the end-side multimodal features, robot current state parameters, locally cached relational memory summary, and behavior constraint parameters; the end-cloud traffic gating module is used to generate data packets to be uploaded based on the state uncertainty parameter, network connection parameter, power supply status parameter, upload permission parameter, scene task identifier, historical upload results, model inference effect, user feedback result, and running resource status, and select the end-side instant execution path, the end-side conservative execution path, or the cloud-side enhanced path; the cloud-side fine-grained... The understanding and strategy optimization module is used to generate a cloud-side fine-grained state, a behavior asset library, and collaborative processing results based on the data packet to be uploaded, historical interaction records, long-term user state model, relational memory, and the robot's current state parameters. The edge-side collaborative execution module is used to generate edge-side execution control results based on the edge-side real-time execution path, the robot's current state parameters, and the basic behavior strategy when selecting the edge-side immediate execution path or the edge-side conservative execution path, and to perform validity period verification and matching verification on the collaborative processing results when selecting the cloud-side enhanced path, and to generate a collaborative behavior strategy based on the edge-side immediate state, the robot's current state parameters, and the collaborative processing results. The offline continuous companionship and synchronization module is used to maintain basic companionship behavior, call the offline behavior queue, and synchronize cached records after network connection is restored when the network is disconnected, upload is restricted, or the collaborative processing results are unavailable.

[0060] The beneficial effects of this invention are:

[0061] 1. By collecting multimodal data and robot current state data on the edge, and generating edge-side real-time state, local state labels, state evaluation parameters and basic behavior strategies, basic companion behaviors can be directly output in scenarios such as user approach, wake word, touch, hug, sleep-related events and basic emotions, reducing reliance on real-time cloud feedback and improving the real-time feedback capability of the bionic companion robot.

[0062] 2. The edge-cloud diversion gating module selects the edge-side immediate execution path, edge-side conservative execution path, or cloud-side enhanced path based on state uncertainty, network connection, power supply status, upload permission, and scenario task identifier. The cloud side combines long-term user state model, relation memory, and robot current state parameters to generate cloud-side fine state, state correction results, and policy increment results. This can improve the user state understanding and the targeting of personalized companionship strategy generation in scenarios with complex semantics, multimodal conflicts, or those requiring historical relation memory.

[0063] 3. By performing validity period verification and matching verification on the collaborative processing results, and using the robot's current state parameters as a mimicry of life state to participate in behavioral decisions, and generating an offline behavior queue when the network is disconnected, upload is restricted, or the collaborative processing results are unavailable, and synchronously caching the records after the network is restored, the risk of expired cloud results overwriting the real-time state on the edge can be reduced, and the smoothness of behavioral output and the ability to accompany offline continuously can be maintained. Attached Figure Description

[0064] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0065] Figure 1 This is a flowchart of a user status recognition and behavior decision-making method based on edge-cloud collaboration.

[0066] Figure 2 This is a flowchart of step S3, the cloud-end traffic splitting gating step, of the present invention. Detailed Implementation

[0067] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0068] refer to Figures 1-2 User status recognition and behavior decision-making methods based on edge-cloud collaboration include:

[0069] S1. Collect multimodal data of user interaction with companion robot and robot current state data, and obtain end-side multimodal features, robot current state parameters and end-cloud collaborative control parameters through time alignment, normalization and feature extraction;

[0070] S2. Perform coarse-grained user state recognition based on the multimodal features of the device side, generate the device side real-time state, local state label, state evaluation parameters and basic behavior strategy, and output basic companion behavior;

[0071] S3. When the edge-cloud collaborative control parameters meet the cloud-side processing conditions and the state evaluation parameters indicate that cloud-side processing needs to be called, generate the data packet to be uploaded through the edge-cloud diversion gating and send it to the cloud side.

[0072] S4. Based on the data packet to be uploaded, historical interaction records, long-term user state model, relational memory and robot current state parameters, the cloud side generates the cloud side's fine state and collaborative processing results.

[0073] S5. The edge performs validity period verification and matching verification on the collaborative processing results, and generates or adjusts the collaborative behavior strategy based on the verification results, combined with the edge's real-time status, the robot's current state parameters, and the collaborative processing results.

[0074] S6. When the edge-cloud collaborative control parameters indicate network anomalies, upload restrictions, or unavailability of collaborative processing results, the edge maintains basic companion behavior and synchronously caches records to update the long-term user state model and relation memory after network recovery.

[0075] In this specific embodiment, S1 includes:

[0076] The companion robot uses a monotonically increasing system clock as a unified time base. It collects and writes the sampling results from cameras, microphones, touch sensors, attitude sensors, inertial measurement units, millimeter-wave radar, application interaction data, environmental data, and the robot's controller, actuator feedback unit, and energy management unit into a time-sorted circular buffer. The discrete time slice index of the unified time base is denoted as... The start time of each time slice is denoted as The time slice length is fixed. And the time slice update step size is fixed as This allows data from different sampling frequencies to be projected into the same time slice;

[0077] The camera captures data at 15fps with a resolution of [resolution value missing]. The RGB image frames are processed, and a capture timestamp is appended to each frame before being input into the face detection model. The face detection model Using the BlazeFace architecture and fixing the input size All convolutional layers use The convolution kernel is set to ReLU, and the number of model parameters is fixed. The algorithm outputs the face bounding box, five key points, and confidence scores. Then, it performs affine alignment on the face region and inputs it into the expression feature extraction model. The facial expression feature extraction model The MobileNetV3-Small architecture is adopted with a fixed input size. With a width factor of 0.75, the final output dimension of the fully connected layer is fixed at [value missing]. This process yields camera modal feature vectors for characterizing facial expressions. Simultaneously, head pose angles are constructed based on key points, and the gaze direction feature is formed by combining the head pose angles with the eye region grayscale gradient. Approach and departure trajectory features are constructed based on the displacement sequence and scale change sequence of the face bounding box center point within consecutive time slices. These gaze direction features, approach and departure trajectory features, and facial expression feature vectors are concatenated to form a time slice. Corresponding camera modal features ;

[0078] The microphone acquires mono audio at 16kHz and performs speech activity detection at the edge to obtain speech segments. Log-Mel spectra with a frame length of 25ms and a frame shift of 10ms are then extracted from the speech segments. The number of Mel filter banks is fixed at 80 to form acoustic features, which are then input into the wake word detection model. The wake word detection model generates a wake word event identifier. The DS-CNN architecture is used, with all convolutional kernels being depthwise separable convolutional networks. The number of layers is 8, and the number of channels is as follows: The number of parameters is fixed. Output the posterior probability of the wake word and generate a wake word event identifier through a threshold of 0.8;

[0079] Simultaneously, the same speech segment input side-stream speech recognition model To obtain the speech text, the streaming speech recognition model It consists of a Conformer encoder and a CTC decoder. The encoder has 6 layers, each with a model dimension of 256. It has 4 attention heads, a convolutional kernel size of 15, and a fixed number of parameters. The output character-level transcription sequence is used to obtain the speech text through dictionary mapping, thus forming a time slice. Corresponding microphone modal characteristics It also includes wake word event identifiers, speech text, and acoustic features;

[0080] The touch sensor samples touch coordinates and pressure values ​​at 100Hz and aggregates them within a time slice to obtain the touch area number, touch duration, touch rhythm, and touch force. The touch area number is determined by a predefined grid area where the touch coordinates fall; the touch duration is determined by the time span of consecutive touch sampling points; the touch rhythm is obtained by representing the mean and variance of the time interval sequence of adjacent touch events within the time slice; and the touch force is obtained by mapping the pressure value to Newton units using a calibration curve and then taking the mean value within the time slice, thus forming the time slice. Corresponding touch modal features ;

[0081] The attitude sensor and inertial measurement unit sample triaxial acceleration, triaxial angular velocity, and attitude quaternions at 200Hz and perform complementary filtering at the end to suppress noise. The acceleration weight of the complementary filter is fixed at 0.02 and the gyroscope weight is fixed at 0.98. Then, based on the attitude quaternions and peak acceleration, the state indicators of picking up, putting down, shaking, inverting, and rolling are detected and time slices are formed. Corresponding attitude and inertial modal features ;

[0082] The millimeter-wave radar outputs a range image at 20Hz and performs bandpass filtering and fast Fourier transform on the phase sequence within the target range gate at the end-side to extract the main frequency peaks of respiration and heart rate, respectively. Simultaneously, it generates a presence marker based on the range image energy threshold and generates range variation features based on the change of range peaks over time, thus forming a time slice. Corresponding radar mode characteristics ;

[0083] Application interaction data and environmental data are read and discretized into time slice, device mode, and scene context fields within the time slice. The time slice is mapped to a predefined time slice number by the local clock, the device mode is read from the robot's current operating mode register, and the scene context is encoded by network type, geofence, indoor lighting level, and noise level to form the time slice. Corresponding context features ;

[0084] The robot body controller, actuator feedback unit, and energy management unit read and form the robot's current state parameters within a time slice. The robot's current state parameters It includes simulated emotional state, activity state, system state, energy state, safety state, interaction mode state and hardware state, and is encoded in a fixed field order. The energy state includes battery charge state, battery temperature and charging status. The safety state includes motor overcurrent status, joint limit status and fall risk status. The hardware state includes sensor online status and actuator temperature.

[0085] End-to-cloud collaborative control parameters Within the time slice, network connection parameters, power supply status parameters, upload permission parameters, and scene task identifier are generated and included. The network connection parameters consist of the average round-trip latency of the last 5 heartbeats, packet loss rate, and connection status identifier. The power supply status parameters consist of the charging identifier and the comparison result of the battery state of charge threshold. The upload permission parameters are jointly determined by the local privacy configuration switch and the validity of the user authorization token. The scene task identifier is obtained by mapping the current interaction entry point to the task routing table.

[0086] After time alignment is completed, the modal features are concatenated in a fixed order to form end-side multimodal features:

[0087] ;

[0088] in Indicates time slice End-side multimodal features, Indicates time slice Camera modal characteristics, Indicates time slice Microphone modal characteristics, Indicates time slice Touch modal features, Indicates time slice The attitude and inertial modal characteristics, Indicates time slice Radar mode characteristics, Indicates time slice Contextual features, Indicates the time slice index, which will eventually be and The features are output together and written to the edge feature cache for use in steps S2 and S3. The vector fields are calibrated and normalized based on the preset mean and variance table, and a missing value is written when a field is missing to ensure that the subsequent state recognition can distinguish between true zero values ​​and missing values.

[0089] In this specific embodiment, S2 includes:

[0090] The edge-side coarse-grained recognition module uses edge-side multimodal features With the robot's current state parameters Reading time slices The corresponding data is retrieved from the edge feature cache, and the data of length is... Sliding time window To cover the most recent Interactive segments, in which Indicates the time slice index. This represents the time slice length and is fixed in step S1. This indicates the time slice update step size, which is fixed in step S1. Indicates the length of the time window used for edge-side identification;

[0091] End-side state recognition model A deterministic structure of "multimodal embedding-gated fusion-temporal convergence-multi-task output" is adopted, wherein the multimodal embedding layer... Six modal subvectors Two fully connected transformations are performed to unify the embedding to 64 dimensions. The first layer has an output dimension of 128 and uses ReLU as the activation function, while the second layer has an output dimension of 64 and does not use an activation function.

[0092] The gated fusion layer reads the missing modality identifiers for each modality and forms a modality usability vector. ,Will Inputting a fully connected layer yields a real vector of length 6, which is then processed by Softmax to obtain the modal weights. And for the six 64-dimensional embeddings Weighted summation yields the single-frame fused representation. If and only if When only one element is 1, the Softmax output degenerates into a single-modal weight of 1 and the rest of the weights of 0 to achieve single-modal passthrough;

[0093] The time-series convergence layer within the time window A single-layer GRU is input to obtain the temporal hidden state. The GRU has a hiding dimension of 64, a layer of 1, and a dropout value of 0. The hidden state at the last time step is taken. As a time-series converged output;

[0094] The multi-task output layer consists of two fully connected layers. The first layer has an output dimension of 32 and uses the ReLU activation function, while the second layer has an output dimension of 9 and uses the Sigmoid activation function, outputting a probability vector. ,in The nine components correspond in a fixed order to the user's proximity state, interaction state, wake word state, simple semantic state, touch event state, holding posture state, pre-sleep state, nighttime awakening state, and basic emotional state.

[0095] Rule Engine The rule state vector is generated based on the event fields and thresholds within the same time window. The wake-up word state is triggered by a wake-up word posterior probability of not less than 0.8 in step S1 and lasts for 2 seconds; the touch event state is triggered by a touch duration of not less than 0.15 seconds and lasts for 1 second after the touch ends; and the holding posture state is determined by posture and inertial modal features. The "picking up" flag is triggered when it is true and cleared when the "putting down" flag is true; the pre-sleep state is determined by contextual features. The time period in the data belongs to the pre-sleep period, the millimeter-wave radar presence is true, the dominant breathing frequency is in the range of 0.10Hz to 0.35Hz, and the root mean square of the acceleration modulus of the posture and inertial modal features within the last 10 seconds is not greater than 0.15g. The night awakening state is triggered by the wake word state or touch event state within the sleep-related period and lasts for 5 seconds. The basic emotional state is determined by the camera modal features. In-face embedding and microphone modal features Acoustic Embedded in The basic emotional output components are jointly satisfied with a probability of not less than 0.6 to trigger;

[0096] The endpoint uses a cascaded identification and confidence-gated determination process to generate the endpoint's real-time state. , among which first by generate To ensure that wake word events, touch events, hold events, and sleep-related events can take effect immediately on the device, and then... generate Used to supplement interactive states, simple semantic states, and basic emotional states, and in When a certain component is 1, the corresponding state will be in Set the middle setting to true and mark its source as the rule source;

[0097] The confidence parameter for end-side identification is set to The state uncertainty parameter is obtained by adjusting the state uncertainty parameter. The parameters are obtained by taking the complementary parts, and together they constitute the state assessment parameters. ,in Determine as follows:

[0098] ;

[0099] in This represents the confidence parameter for edge-side identification. This means cropping the input to a closed interval. The operator, Represents the model probability weights and is fixed at 1. Represents the multimodal consistency weight and is fixed at 1. The sensor integrity weight is represented and fixed at 1. Represents the scenario risk weight and is fixed at 1. Represents a probability vector Maximum component value, The multimodal consistency score is represented by the sum of penalties and complementary values ​​obtained from three types of consistency checks: consistency between camera proximity and departure trajectory features and millimeter-wave radar distance change features; consistency between wake word event identifiers and voice activity detection; and consistency between touch events and posture stability. Represents the sensor integrity score and is determined by time slices. The internal modal count can be obtained by dividing the total number of modes (6). Indicates the risk level of the scenario and is determined by Energy state and safety state in The time period numbers are obtained by looking up a table together and the battery state of charge is below The value is 1 when the motor overcurrent indicator is true or the drop risk indicator is true;

[0100] Local status label Through the The coarse-grained user states in the middle are reduced by a fixed priority table, which is in the following order from high to low: holding posture state, night wake-up state, bedtime state, wake word state, touch event state, simple semantic state, interaction state, user proximity state, and basic emotional state, so as to ensure that the state with the greatest impact on companionship behavior is labeled first.

[0101] Behavioral constraint parameters Depend on and Generated by local security rules, where when Energy state characterization of battery state of charge is lower than When the upper limit of the sound output level is set to medium and the movement control parameter is disabled, When the safety status indicator shows the motor overcurrent flag as true or the joint limit flag as true, set the limb movement parameters and tail movement amplitude to the minimum and the kneading frequency to 0. When representing nighttime periods, set the upper limit of light brightness to And set the behavioral rhythm to slow;

[0102] The client reads relational memory digests from the local cache. The relational memory summary The relational memory digest employs a fixed field structure and includes user address preferences, a list of disabled actions, soothing preferences, interaction intensity preferences, the template number of the most recent positive feedback behavior, and its timestamp. The value is updated by the historical cache on the client side, the relationship memory summary that passed the validity period check and matching check most recently, and the synchronous record after network recovery, with the largest last update timestamp overwriting the old value;

[0103] The endpoint is based on a coarse-grained user state set, and Generate a set of multidimensional behavioral parameters ,in It includes parameters such as eye expression status, eye animation, sound content, sound output level, purring rhythm, head movement amplitude, tail movement frequency, movement control parameters, light brightness, and system mode switching parameters, and uses a basic behavior mapping table to... Map to the basic companionship behavior template number, and then... Hard constraints are applied to the amplitude, frequency, output volume, and light brightness of the actions within the template, and the template is then clipped. The system performs preference substitutions between audio content and interaction intensity to form basic behavioral strategies. ,in Includes the template number of the invoked behavior and The values ​​of each parameter are determined;

[0104] End-side based on basic behavioral strategies Generate and issue control commands for basic companionship behaviors to the main controller and actuators, and simultaneously... and Write the state to the edge-side cache for the edge-cloud traffic splitting gating module in step S3 to read.

[0105] In this specific embodiment, S3 includes:

[0106] The edge-cloud traffic splitting gating module reads time slices from the edge-side state cache. Corresponding local status label With state assessment parameters and control parameters from the edge-cloud collaboration The system reads network connection parameters, power supply status parameters, upload license parameters, and scene task identifier. Indicates the time slice index. This represents the confidence parameter for end-side identification and is determined by the end-side state identification model in step S2. With the rules engine jointly determined, This represents the state uncertainty parameter and is taken as the pair in step S2. The complementary quantities, network connection parameters include connection status identifiers. Average round-trip time of the last 5 heartbeats With packet loss rate ,in and Indicates network connection, Indicates a disconnection. The unit is milliseconds (ms). The range of values ​​is The power supply status parameters include a charging indicator. With battery state of charge The upload license parameter is the upload license identifier. Furthermore, the scenario task identifier is determined jointly by the local privacy configuration switch and the validity of the user authorization token. And it is determined through the task routing table on the endpoint, where Indicates basic companionship tasks, Indicates a simple dialogue task, Indicates a security and reassurance mission, Represents the task of retrieving relational memory. This indicates a task invocation for long-term user state modeling;

[0107] The edge-cloud traffic splitting gating module in time slices Internal determination of cloud-side processing condition identifiers , where if and only if and and and or and Time setting Otherwise ;

[0108] The edge-cloud traffic splitting gating module simultaneously generates multimodal conflict identifiers. Where the multimodal consistency score obtained in step S2 is lower than 0.4 or "camera modal features" appear, Characterizing the mode features of millimeter-wave radar near the user The representation does not include "wake word event flag is true but voice activity detection is false" and "touch event state is true and gesture and inertial modal features are present". The risk of inversion or rollover is true for any one of the three types of conflicts. Otherwise ;

[0109] The edge-cloud traffic diversion gating module generates memory requirement identifiers based on scenario task identifiers. , where if and only if Time setting Otherwise To characterize the current interaction request as requiring the invocation of a long-term user state model or relational memory; and to ensure that the traffic diversion decision reflects both historical policy effectiveness and network resource status, the edge-cloud traffic diversion gating module maintains historical policy effectiveness scores. Network resource penalty score ,in The endpoint writes the result after receiving the collaborative processing result from the cloud side and completing the S5 verification. The writing rule is to map the user feedback result of this interaction and the effect data after the behavior is executed to a gain value, and update it by exponential moving average with a fixed update coefficient. Depend on and The result was obtained by looking up the table in three categories and when and Time to take ,when or Time to take ,when or Time to take ;

[0110] The edge-cloud traffic offloading gating module calculates the routing uncertainty parameters used for traffic offloading. Based on this, the following options are selected: immediate execution path on the endpoint, conservative execution path on the endpoint, or enhanced execution path on the cloud side. in Indicates the routing uncertainty parameter. This means cropping the input to a closed interval. The operator, The parameter represents the state uncertainty. The historical strategy effect penalty weight is fixed. Indicates the historical strategy effectiveness score. This represents the network resource penalty weight, which is fixed at 1. This represents the network resource penalty score. Indicates the time slice index;

[0111] The edge-cloud traffic splitting gating module has a preset local execution threshold. With shunt threshold And satisfy and maintain the basic behavior mapping table. To represent the mapping relationship from local status labels to basic companion behavior template numbers, where when As a bond energy The system retrieves a unique behavior template number, and all required behavior parameters of the template can be obtained from the multidimensional behavior parameter set generated in step S2. When providing the information, it is determined that "the local status label directly matches the basic behavior mapping table"; otherwise, it is determined that "it does not directly match".

[0112] when Furthermore, when a direct match is determined, the end-to-cloud traffic gating module selects the immediate execution path on the end side and writes the cloud-side enhanced path disable flag into the gating status register to prevent the upload from being triggered during that time slice;

[0113] when However, when it is determined that it is not a direct match, or when and and At that time, the edge-cloud traffic splitting gating module selects the conservative execution path on the edge side and marks the time slice as "to be reviewed by the cloud side" so that the cached records can be synchronized when the network conditions are met in the future;

[0114] when or or At this time, the edge-cloud traffic offloading gating module enters the cloud-side enhanced path selection process, where when Generate data packets to be uploaded in real time And sent to the cloud side, when The system does not generate data packets to be uploaded and rolls back to the conservative execution path on the client side, and writes the "Reason for cloud-side enhancement unavailability" field into the local log for offline continuous support and recovery synchronization in step S6.

[0115] Data packet to be uploaded Protocol Buffers serialization is used, and it includes a fixed set of fields and a fixed byte limit. The fixed set of fields includes the time slice index. Time slice start time Scene task identifier Local status tags Status assessment parameters Robot current state parameters , length is End-side multimodal feature window And an event trajectory summary for cloud-side verification, the event trajectory summary including wake word event identifier, touch area number and touch duration, pick-up and put-down state identifier, and peak values ​​of respiratory and heart rate frequencies, and when uploading a permission identifier. The data further includes a constrained payload of the original data segment, which is generated by Opus encoding of the most recent 1.2s audio with a fixed bitrate of 16kbps, and by JPEG encoding of the most recent 2 frames of face-aligned images with a fixed quality factor of 60. The face region is cropped on the image at the end to avoid uploading irrelevant backgrounds.

[0116] The edge-cloud traffic splitting gating module sends The system employs a TLS channel and uses AES-GCM for secondary encryption of the payload. The session key is generated by the device-side key management module every 24 hours to ensure consistent security between the upload link and data storage.

[0117] The edge-cloud traffic splitting gating module dynamically adjusts subsequent upload strategies based on historical upload results, model inference performance, user feedback, and runtime resource status. Historical upload results are maintained using the success rate and average round-trip latency of the last 50 uploads; model inference performance is maintained using the state correction magnitude returned from the cloud side and the strategy increment magnitude; user feedback is maintained using the positive feedback ratio recorded on the edge side; and runtime resource status is maintained using edge-side CPU utilization, available memory, and temperature protection indicators. Based on this, the module then... Adjust the shunt threshold within the range in steps of 0.05. And in The limited load switch allows for two modes: "Upload Feature Window Only" or "Upload Feature Window and Limited Load." This enables adaptive control of the upload frequency and content size while meeting cloud-side processing requirements. Send to the cloud side for processing in step S4.

[0118] In this specific embodiment, S4 includes:

[0119] The data packets to be uploaded sent by the cloud-side fine-grained understanding and strategy optimization module on the receiving end. Then, device authentication is completed, and the authenticated device identifier is mapped to the user identifier. Subsequently Perform deserialization, field integrity verification, and timestamp consistency verification to extract the scene task identifier. Local status tags Status assessment parameters Robot current state parameters End-side multimodal feature window And an event trajectory summary, and based on the event trajectory summary and local state labels The real-time state of the endpoint is obtained through the cloud-side state restoration table. In order to compare the differences with the cloud-based inference results, among which Indicates the time slice index. This indicates the length of the end-side window and is fixed in step S2. This represents the confidence parameter for edge-side identification. Indicates parameters of state uncertainty;

[0120] The cloud side will display the camera modal features in the multimodal feature window of the edge side. Microphone modal characteristics Touch modal features Attitude and inertial modal characteristics Radar mode characteristics With context features Expanded chronologically to a length of The sequence is input into the multimodal emotion understanding model. Generate emotion-related representations and interaction intention-related representations, wherein the multimodal emotion understanding model The model consists of a modality-aligned encoder and a temporal fusion encoder. The modality-aligned encoder maps the six modality sub-vectors of each time slice to a 128-dimensional embedding through two fully connected layers. The first layer has an output dimension of 256 and uses GELU activation, while the second layer has an output dimension of 128 and does not use an activation function. The temporal fusion encoder adopts a TransformerEncoder structure with 4 layers, 256 hidden dimensions, 4 attention heads, 1024 feedforward dimensions, and a dropout of 0.1. It also uses learnable positional encoding to handle temporal relationships within the window. The model output has two classification heads on the same hidden dimension. The first classification head outputs the probability distribution of the emotion state, and the category corresponding to the highest probability is taken as the cloud-side emotion state. The set of emotional state categories is fixed as follows: Furthermore, these correspond to calm, joy, anxiety, sadness, irritability, and drowsiness, respectively. The second classification head outputs the probability distribution of interaction intention states, and uses the category corresponding to the highest probability as the cloud-side interaction intention state. The set of interaction intention state categories is fixed as follows: And they correspond to low, medium and high respectively;

[0121] Cloud Side General Speech-text input language understanding model To generate semantic intent and key slots, wherein the language understanding model The TransformerEncoder architecture is used with 6 layers, a hidden dimension of 384, 6 attention heads, a feedforward layer dimension of 1536, and a dropout of 0.1. The vocabulary size is fixed at 8000, and an intent classification head is set to output the intent category. Slot set output with slot label header The intent category set is fixed as follows: These correspond to greetings, seeking comfort, requesting interaction, requesting to talk, requesting sleep, refusing interaction, expressing liking, expressing discomfort, calling for help, setting preferences, checking status, and ending the conversation.

[0122] Cloud side uses user identifier Load long-term user state model from persistent storage for key Relational memory Among them, long-term user state model It consists of a user-state GRU and a preference readout head. The input to the user-state GRU is a vector of interaction events arranged in the interaction order, and the interaction event vector is composed of... User feedback records and scene task identifiers The concatenation yields a GRU with a fixed hidden dimension of 128, 1 layer, and 0 dropout. The preference readhead is a two-layer fully connected network with the first layer having an output dimension of 64 using ReLU and the second layer having an output dimension of 16 using Sigmoid, with the output being the preference state vector. The 16 preference components correspond in a fixed order to preferences for sound output level, snoring rhythm, interaction intensity, eye animation, light brightness, acceptable touch area, acceptable kneading force, acceptable tail movement amplitude, acceptable movement distance, bedtime soothing, nighttime wake-up response, addressing, prohibited actions, storytelling content, dialogue length, and silent companionship.

[0123] Relational memory Employs a key-value memory structure and stores entries Storage, in The key vector is obtained by linear mapping from the interaction event vector. It is a structured value that includes the trigger state, the behavior template number used, a summary of the set of multi-dimensional behavior parameters executed, user feedback tags, and a summary of effect data. The cloud side generates a query vector based on the current interaction event vector, using the entry timestamp. The top 3 memory entries are retrieved based on dot product similarity, and their structured values ​​are compressed into relational memory summaries. And guarantee Includes the behavioral template number of the most recent positive feedback, a list of prohibited actions, address preferences, and sleep-related preferences;

[0124] Cloud side in getting and After generating the cloud-side fine state ,in Includes emotional state Interaction willingness status Relationship Stage Status With behavioral preference state Relationship stage state From long-term user state model The hidden state of the GRU is determined by the four-class readout head, and the four stages are fixed and correspond to the initial acquaintance, familiarity, stability and intimacy.

[0125] Cloud side will refine the cloud side status Real-time status at the end Perform difference comparisons and generate state correction results. ,in The system provides structured fields for the status items that need correction, the corrected status items, and the correction confidence level. When the correction confidence level is not lower than 0.7, the corrected status items are marked as mandatory; when the correction confidence level is lower than 0.7 but not lower than 0.4, the corrected status items are marked as recommended; and when the correction confidence level is lower than 0.4, the correction fields are not output to avoid abrupt changes in real-time feedback from the end side.

[0126] The cloud side builds and updates the behavior asset library based on historical interaction records, user feedback records, and the effect data after the execution of companionship behaviors. Among them, behavioral asset library Validated companionship behavior templates, behavior parameter combinations, behavior rhythm templates, state mapping templates, and behavior combination templates are stored as behavior asset entries. Each behavior asset entry contains an asset number, an applicable set of scenario task identifiers, an applicable set of state identifiers, a default set of multi-dimensional behavior parameters, an adjustable range, a rhythm description, and statistics on the last 100 uses.

[0127] The cloud platform generates an effectiveness score for each completed interaction and uses this score to determine whether to solidify the combination of behavioral parameters into a new behavioral asset item or to update the statistics and default parameters of an existing item. Calculate using the following formula:

[0128] ;

[0129] in Indicates the first The effectiveness score of each interaction. This means cropping the input to a closed interval. The operator, This represents the weight of user feedback and is fixed at 1. Indicates the first The normalized score of user feedback for each interaction is set as follows: positive feedback is 1, neutral feedback is 0.5, and negative feedback is 0.5. Indicates the interaction input weight and is fixed at 1. Indicates the first The normalized score of each interaction is obtained by mapping the duration of effective interactions to the number of user-initiated triggers, and is taken as when the duration is not less than 20 seconds and the number of user-initiated triggers is not less than 2. This represents the risk penalty weight and is fixed at 1. Indicates the first The risk normalization score of the second interaction and the robot's current state parameters If any of the motor overcurrent indicator, joint limit indicator, or drop risk indicator is true during the data transmission process, the value is set to 1; otherwise, it is set to 0. The behavior template and its multi-dimensional behavior parameter set are then solidified into a new candidate asset entry, and after accumulating 20 solidification records, it is marked as a verified asset entry.

[0130] Cloud side basis Relational Memory Summary With the robot's current state parameters Generate scene constraint parameters ,in Includes upper limits for sound output levels, light brightness, and movement range, as well as permissions for motion control and mode switching. When representing a request for sleep or a bedtime scenario, the upper limit of the sound output level is fixed to medium and the upper limit of the light brightness is fixed to [value missing]. And fix the movement control permission to be prohibited;

[0131] Cloud side-by-side behavioral asset library Filter out the task identifiers that meet the requirements of the scenario. With scene constraint parameters And the applicable status identifier includes the local status label. Or may include state correction results The set of candidate assets marked as having a mandatory corrective action status is then used to define the corresponding label. and Input policy incremental generation model Generate policy increment results for adjusting the basic behavioral policy in step S2. The strategy incremental generation model mentioned above A two-tower structure is adopted, with the default multidimensional behavioral parameter set of candidate assets encoded into a 64-dimensional asset vector in the left tower, and the right tower encoding... and The encoding is a 64-dimensional state memory vector. Both the left and right towers consist of two fully connected layers. The first layer has an output dimension of 128 and uses ReLU, while the second layer has an output dimension of 64 and does not use an activation function. The two vectors are then concatenated and fed into a fully connected scoring head to output candidate asset scores. The asset number with the highest score is selected as the asset number for the invoked action. The default multidimensional behavior parameter set of the asset entry is then mapped to the basic behavior mapping table on the client side. The incremental strategy result is obtained by subtracting the corresponding baseline parameters. ,in The system outputs data in a sparse field format, with the field set limited to parameters including eye state, eye animation, eye emotion, sound content, sound output level, purring content, purring rhythm, respiratory amplitude, heart rate, kneading frequency, kneading duration, kneading force, head movement amplitude, tail movement frequency, tail movement amplitude, limb movement parameters, trunk posture adjustment parameters, movement control parameters, and light brightness and system mode switching parameters. Each field is assigned an increment type identifier and an increment value. The increment type identifier is fixed to either an additive increment or a replacement increment, with the additive increment used to evaluate the multi-dimensional behavioral parameter set on the endpoint. The corresponding fields are updated by addition, and the incremental replacement is used to cover the corresponding fields on the client side;

[0132] Cloud side will refine the cloud side status State correction results Incremental results of the strategy Relational Memory Summary Scene constraint parameters Return timestamp Validity period label Applicable status indicators Composition of collaborative processing results And send it to the endpoint, where a timestamp is returned. Generate for cloud side Server time, validity period identifier A fixed 5s interval is used to constrain the end-side window, applicable to status indicators. The fixed label is used when filtering the candidate asset set and when If a mandatory correction is in effect, use the corrected label; otherwise, use the corrected label. This enables the end side to control the S5. Perform validity period verification and matching verification, and then invoke the asset number of the invoked action accordingly. Corresponding behavioral assets and combined with incremental strategy results Multidimensional behavioral parameter set on the opposite side Perform consistent field updates.

[0133] In this specific embodiment, S5 includes:

[0134] The edge-side collaborative execution module receives the collaborative processing results sent from the cloud side. Afterwards, Write to the collaboration results cache and read the fine-grained cloud-side state from it. State correction results Incremental results of the strategy Relational Memory Summary Scene constraint parameters Return timestamp Validity period label Applicable status indicators With behavioral asset number ,in This indicates the endpoint time slice index corresponding to the collaborative processing result and is related to the data packet to be uploaded in step S3. The time slice indexes are consistent;

[0135] The local time slice index corresponding to the receiving time at the end is denoted as And read the current local state label from the edge state cache. End-side instantaneous status Robot current state parameters Basic behavioral strategies at the end Simultaneously read the current time from the local clock. And read the clock offset obtained from the most recent time synchronization with the cloud side. ,in This represents the offset of mapping the cloud server time to the local time on the endpoint. It is updated by the endpoint based on the server timestamp during the handshake process each time a TLS connection is successfully established, and smoothed using an exponential moving average with a fixed update coefficient of 0.8.

[0136] Based on the returned timestamp With expiration date Perform validity period verification and generate collaborative result weights. ,in:

[0137] ;

[0138] in This indicates the weight of the collaborative results and is used to perform discarding or deweighting. This means cropping the input to a closed interval. The operator, Indicates the local time slice on the end side At the present moment, This indicates that the cloud side generates collaborative processing results. The server timestamp at that time This indicates the offset between the server time and the endpoint time. This indicates the validity period of the collaborative processing result, which is fixed in S4. This represents the edge time slice index corresponding to the collaborative processing result. This represents the local time slice index when the end-side receives the collaborative processing results.

[0139] when Time end side Perform the discard process and mark the discard reason as timeout failure, then continue to maintain it. Corresponding basic companionship behaviors output;

[0140] when Time end side Perform a match check, with the match check indicating the applicable status. Current local status label Given the input, give three results: consistent, partially consistent, and inconsistent, wherein the condition is met if and only if When it is determined to be consistent and enters collaborative execution, Furthermore, when both belong to the same predefined state group, they are considered partially consistent and enter conservative cooperative execution. Furthermore, if the two do not belong to the same predefined state group, they are judged to be inconsistent and [further action will be taken]. Perform drop or demotion processing, where the predefined state groups are implemented using a fixed grouping table and the grouping table includes sleep groups. Sleep state, nighttime awakening state Call group Wake word state, simple semantic state Contact group Touch event state, interaction state, user proximity state Holding Group Holding posture Emotion Group Basic emotional state And whether the status labels fall into the same group is used as a partial consistency criterion;

[0141] When a match is determined, the endpoint loads the behavior asset number from the local behavior asset cache. The corresponding behavior assets are retrieved, along with their behavior templates and default multidimensional behavior parameter sets. The multidimensional behavior parameter set generated locally in step S2 is denoted as... Incremental results of the strategy Perform additive incremental or replacement incremental updates based on field type and... Scaling all incremental fields yields a set of collaborative multidimensional behavioral parameters. Then the state correction result The correction status marked as mandatory is written to the immediate status on the terminal side. The corrective state marked as effective is written into the candidate state buffer for continuous consistency confirmation in subsequent time slices;

[0142] When partial consistency is determined, the endpoint does not change the local state label. And will The upper limit is set to 0.5 and only allows... The fields corresponding to the current state group are active. These fields are implemented using a fixed correspondence table. For the sleep group, only fields related to light brightness, sound output level, snoring rhythm, and behavioral rhythm are active. For the calling group, only sound content parameters and sound output level are active. For the contact group, only head movement amplitude, tail movement frequency, and eye animation parameters are active. For the holding group, only trunk posture adjustment parameters and limb movement parameters are active. For the emotion group, only eye emotion parameters and sound content parameters are active. When an inconsistency is detected, the terminal will... Mark as a state mismatch and Set to 0 to perform the discard process and continue to maintain. Corresponding basic companionship behaviors output;

[0143] Before collaborative execution, the edge device sets the robot's current state parameters. As a pseudo-life state, it participates in decision-making and generates parameters of willingness to respond. Interaction intensity parameters With behavioral rhythm parameters ,in Includes pseudo-emotional state field Activity status field System status fields Energy state field Security status field Interaction mode status field Hardware status field Furthermore, all fields are obtained by encoding in a fixed order from step S1, including response intention parameters. The set of values ​​is fixed as And generated according to a fixed rule table, and when the security status field... When the motor overcurrent indicator, joint limit indicator, or drop risk indicator is true, take... When the energy state field When the battery's state of charge is below 0.2% and the charging indicator is false, take... In other cases, take Interaction intensity parameters The set of values ​​is fixed as And in Time to take ,exist Time to take ,exist And the pseudo-emotional state field When drowsy, take And take in other simulated emotional states Behavioral rhythm parameters The set of values ​​is fixed as In the interaction mode status field When representing sleep patterns or quiet nighttime patterns, take And take in other interaction modes ;

[0144] The edge side will set the scene constraint parameters and Together they act on a set of collaborative multidimensional behavioral parameters To form a collaborative behavior strategy , among which when At that time, all fields with mechanical motion attributes will be prohibited from execution, and the cooperative behavior policy will be downgraded to only outputting safety prompt sounds and static eye animations, while simultaneously writing the conflict marker to the safety log. At that time, the sound output level, movement amplitude, movement frequency, and movement control parameters are downgraded, and the movement control parameters in the cooperative behavior strategy are forcibly disabled. The behavior rhythm-related fields are multiplied by 0.7 and the silent interval of each behavior template is extended to achieve a combination of delayed execution and partial execution.

[0145] When the robot's current state parameters Real-time status at the end Or cloud-side fine state When conflicts occur, the endpoint follows a fixed behavior priority rule table. The comprehensive decision-making process is based on the following behavior priority rules, ranked from highest to lowest: security state constraints, hardware state constraints, energy state constraints, interaction mode state constraints, and scenario constraint parameters. Relationship stage state and behavioral preference state, end-side instantaneous state Cloud-side fine-grained status When a higher-priority constraint is triggered, it applies constraints, downgrades, delays execution, partially executes, substitutes for execution, or disables execution on lower-priority behavior fields. The endpoint then completes the collaborative behavior strategy. After generation, it is compiled into a control sequence executable by the main controller and sent to the actuator, while simultaneously transmitting the verification results. The adopted behavior asset number The final control sequence summary and execution feedback results are written to the endpoint history cache for use in the historical policy effect score in step S3. Update and synchronize with the offline cache in S6.

[0146] In this specific embodiment, S6 includes:

[0147] The offline continuous companionship and synchronization module is used in each time slice. Read edge-cloud collaborative control parameters The network connection parameters and upload permission parameters in S3, combined with the selection results of the cloud-side enhanced path in S3 and the return status of S4, determine whether the collaborative processing result is usable;

[0148] The network connection parameters include connection status indicators. Recent heart rate round-trip time average With packet loss rate The upload license parameter is the upload license identifier. If and only if satisfying or Or the end side has sent the data packet to be uploaded. The waiting time has reached the expiration date. No corresponding collaborative processing result has been received yet. At this time, the offline continuous companionship and synchronization module enters offline mode and sets the offline flag. At the same time, maintain the basic companion behavior output generated in step S2 and prohibit entry into the cloud-side enhancement path;

[0149] After entering offline mode, the module reads the most recent collaborative behavior strategy that simultaneously passed both validity period verification and matching verification from the client-side historical cache. and its applicable status indicators ,in This indicates the time slice index of the most recent successful verification, and the local rule table is read. Cached user preference records Behavioral assets with local caching Generate offline behavior queue ,in It is a collection of behavioral assets that can be directly invoked on the client side, and each behavioral asset contains a behavioral template number, a default multi-dimensional behavioral parameter set, an adjustable range, and a cooldown time. This is a persistent preference vector on the client side, and the meanings of its fields are consistent with the behavioral preference states on the cloud side. The set of edge safety and time-based rules includes at least nighttime noise reduction rules, low battery degradation rules, overcurrent or limit prohibition rules, and drop risk prohibition rules.

[0150] Offline behavior queue A defined data structure is used, and storage is keyed according to local state tags. The queue entry structure is fixed. ,in This indicates the local status label that can trigger this entry. Indicates the template number of the invoked behavior and must belong to This indicates that the template corresponds to a set of multidimensional behavioral parameters and that the set of fields is consistent with step S2. Indicates the cooldown time and is given by the cooldown time field of the behavior asset. Indicates the time when the entry was generated;

[0151] generate Shi Xianjiang The last successfully executed behavior template number and parameter writing with positive user feedback in the middle. For the corresponding queue entries, write an offline basic companionship entry for each of the following: pre-sleep state, nighttime awakening state, wake word state, touch event state, and holding posture state, with the behavior template number starting from [previous entry]. The risk level is determined according to the rule of "the applicable status identifier contains the label and has the lowest risk level", and each entry is further classified according to this rule. Apply in sequence With the behavioral constraint parameters generated in step S2 Hard constraint pruning is implemented, and the pruning rules are fixed to limit the sound output level, light brightness, movement amplitude, movement frequency and movement control parameters to the corresponding upper limit or disabled range, thereby ensuring that offline behavior will not bypass the safety constraints.

[0152] In offline mode, the module continuously writes offline cache records. The offline cache record Store in append mode and indexed by time slice The sorting is done by a fixed set of fields for each record, including the start time of the time slice. Local status tags End-side instantaneous status Robot current state parameters Actual execution behavior template number The set of multidimensional behavioral parameters actually executed Execution return result identifier and user feedback records ,in The data is returned by the main controller and is used to indicate whether the action was completed and whether a protection shutdown was triggered. The results are determined by the interaction data collected from the device side. When the user touches the device again to soothe the user, the user continues to interact, or the user actively wakes up, it is recorded as positive. When the user explicitly refuses, continues to move away, or triggers the security protection, it is recorded as negative. All other results are recorded as neutral.

[0153] When a wake word event, touch-to-soothe event, holding event, or sleep-related event is detected, the offline continuous companionship and synchronization module preemptively interrupts non-critical actions being executed and prioritizes actions from the offline behavior queue. Calling the current local state label Consistent queue entries, where wake word events are triggered by the wake word event identifier in step S1, touch soothing events are triggered by the touch duration and touch rhythm satisfying the end-side threshold in step S1, holding events are triggered by the pick-up identifier in step S1, and sleep-related events are triggered by the pre-sleep state, sleep onset event, night wake-up state, morning wake-up event in step S2, or by the robot's current state parameters. The sleep mode switching event triggered by the interaction mode status field in the system is determined;

[0154] when There is no such thing as in When a consistent item or an item is within its cooldown period, the module rolls back to call the basic companion behavior template corresponding to the basic behavior mapping table and sets the multi-dimensional behavior parameters. Set to conservative values, where the conservative values ​​are fixed at setting the sound output level to low and the light brightness to no higher than [the specified value]. Set the movement control parameters to disabled, set the movement amplitude to the minimum non-zero level, and set the kneading frequency to 0.

[0155] When network recovery is detected and the conditions are met and And within 3 consecutive time slices and When the module exits offline mode, it sets the offline flag. Then Encapsulate them into synchronous data packets in batches according to time sequence. And send it to the cloud side, where Includes user identifier The start and end time slice index, record entry set, and deduplication sequence number are generated by a monotonically increasing counter on the end side to support idempotent writing on the cloud side.

[0156] Cloud side for each batch The backend deletes confirmed batches and retains unconfirmed batches for retransmission, thus enabling the status judgments, interaction records, behavior execution results and user feedback records generated during offline periods to be fully synchronized to the cloud and used to update the long-term user status model and relationship memory.

[0157] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

[0158] This invention establishes a continuous technical chain between real-time judgment on the edge, fine understanding on the cloud, and continuous offline support by extracting multimodal features on the edge, recognizing user status, gating the flow between the edge and cloud, modeling long-term state on the cloud and recalling relational memories, and co-executing on the edge. This addresses the problem of discontinuous support caused by limited computing power on the edge, delayed response on the cloud, and weak network interruptions.

[0159] This invention employs multi-factor edge-cloud diversion gating, edge-cloud dual-layer state modeling, distribution of state correction results or policy increment results, timeliness verification and matching verification of cloud results, and behavioral constraints based on mimicry of life states. This ensures that cloud-enhanced results do not simply replace real-time feedback from the edge, but participate in the generation of collaborative behavior strategies under the conditions of effectiveness, matching, and no conflict with the robot's current state. This better guarantees the realization of real-time life sense, understanding of complex states, and smoothness of behavior output.

Claims

1. A user state recognition and behavior decision-making method based on edge-cloud collaboration, characterized in that, include: S1. Collect multimodal data of user interaction with companion robot and robot current state data, and obtain end-side multimodal features, robot current state parameters and end-cloud collaborative control parameters through time alignment, normalization and feature extraction; S2. Perform coarse-grained user state recognition based on the multimodal features of the device side, generate the device side real-time state, local state label, state evaluation parameters and basic behavior strategy, and output basic companion behavior; S3. When the edge-cloud collaborative control parameters meet the cloud-side processing conditions and the state evaluation parameters indicate that cloud-side processing needs to be called, generate the data packet to be uploaded through the edge-cloud diversion gating and send it to the cloud side. S4. Based on the data packet to be uploaded, historical interaction records, long-term user state model, relational memory and robot current state parameters, the cloud side generates the cloud side's fine state and collaborative processing results. S5. The edge performs validity period verification and matching verification on the collaborative processing results, and generates or adjusts the collaborative behavior strategy based on the verification results, combined with the edge's real-time status, the robot's current state parameters, and the collaborative processing results. S6. When the edge-cloud collaborative control parameters indicate network anomalies, upload restrictions, or unavailability of collaborative processing results, the edge maintains basic companion behavior and synchronously caches records to update the long-term user state model and relation memory after network recovery.

2. The user state recognition and behavior decision-making method based on end-to-cloud collaboration according to claim 1, characterized in that, S1 includes: acquiring facial expressions, gaze direction, and approach / departure trajectory images via a camera; acquiring wake-up words, voice text, and acoustic features via a microphone; acquiring touch area, touch duration, touch rhythm, and touch force via a touch sensor; acquiring pick-up, put-down, rocking, inverting, and rolling states via a posture sensor and inertial measurement unit; acquiring breathing, heart rate, presence, and distance change characteristics via millimeter-wave radar; supplementing time period, device mode, and scene context based on application interaction data and environmental data; and acquiring the robot's current state data via a robot body controller, actuator feedback unit, and energy management unit.

3. The user state recognition and behavior decision-making method based on end-to-cloud collaboration according to claim 1, characterized in that, S2 includes: The edge-side multimodal features are input into the edge-side state recognition model and rule engine. At least one of the following methods is used: sequential recognition, parallel recognition, asynchronous recognition, cascaded recognition, single-modal passthrough, confidence gating, or multimodal weighted fusion, to identify at least one coarse-grained user state among the following: user proximity state, interaction state, wake word state, simple semantic state, touch event state, holding posture state, pre-sleep state, nighttime wake-up state, and basic emotional state; and the local state label is generated based on the coarse-grained user state. Based on model output probability, multimodal consistency, sensor integrity, scene risk level, and rule consistency, edge-side identification confidence parameters and state uncertainty parameters are generated, and the state evaluation parameters are constituted by the edge-side identification confidence parameters and the state uncertainty parameters. A multidimensional behavior parameter set is generated based on the coarse-grained user state, the local state label, the robot's current state parameters, the locally cached relation memory digest, and behavior constraint parameters, and the basic behavior strategy is generated based on the multidimensional behavior parameter set. The locally cached relation memory digest is obtained by updating one or more of the following: the edge-side historical cached relation memory digest, the relation memory digest that has recently passed the validity period verification and matching verification, or the synchronous record after network recovery. The behavior constraint parameters are generated based on the robot's current state parameters, scene context, and local safety rules.

4. The user state recognition and behavior decision-making method based on end-to-cloud collaboration according to claim 3, characterized in that, The edge-cloud collaborative control parameters include network connection parameters, power supply status parameters, upload permission parameters, and scene task identifiers. S3 includes: inputting the status evaluation parameters, network connection parameters, power supply status parameters, upload permission parameters, and scene task identifiers into the edge-cloud traffic diversion gating module. The status uncertainty parameter is calculated based on multimodal consistency, model output confidence, sensor integrity, scene risk level, historical strategy effectiveness, and network resource status, and a preset diversion threshold is greater than a preset local execution threshold. When the status uncertainty parameter does not exceed the preset local execution threshold and the local status label directly matches the basic behavior mapping table, an immediate execution path is adopted on the edge side. When the deterministic parameter does not exceed the preset local execution threshold but the local state label cannot be directly matched with the basic behavior mapping table, or when the state uncertainty parameter is greater than the preset local execution threshold but does not exceed the preset diversion threshold, no multimodal conflict occurs, and the current interaction request does not need to call the long-term user state model or the relational memory, a conservative execution path is adopted on the client side; when the state uncertainty parameter exceeds the preset diversion threshold, a multimodal conflict occurs, or the current interaction request needs to call the long-term user state model or the relational memory, the data packet to be uploaded is generated; and the subsequent upload strategy is dynamically adjusted based on historical upload results, model inference effects, user feedback results, and runtime resource status.

5. The user state recognition and behavior decision-making method based on end-to-cloud collaboration according to claim 4, characterized in that, S4 includes: inputting the data packet to be uploaded into a multimodal emotion understanding model and a language understanding model on the cloud side, wherein the data packet to be uploaded includes the scene task identifier; combining the long-term user state model associated with the user identifier, the relational memory, and the robot's current state parameters to generate a cloud-side refined state including emotional state, interaction willingness state, relational stage state, and behavioral preference state; correcting, supplementing, or refining the terminal-side real-time state based on the difference between the cloud-side refined state and the terminal-side real-time state to generate the state correction result; forming a behavior asset library based on the historical interaction records, user feedback records, and effect data after the execution of companionship behavior, and... Generate the policy increment result for adjusting the basic behavior strategy; the behavior asset library includes one or more of the following: verified companion behavior templates, behavior parameter combinations, behavior rhythm templates, state mapping templates, or behavior combination templates; extract the relationship memory summary applicable to the current interaction task based on the relationship memory, and generate the scene constraint parameters based on the scene task identifier and the cloud-side fine-grained state; combine the cloud-side fine-grained state, the state correction result, the policy increment result, the relationship memory summary, the scene constraint parameters, the return timestamp, the validity period identifier, and the applicable state identifier to form the collaborative processing result, and send the collaborative processing result to the edge.

6. The user state recognition and behavior decision-making method based on end-to-cloud collaboration according to claim 5, characterized in that, S5 includes: determining whether the collaborative processing result is within a preset validity period based on the return timestamp and validity period identifier in the collaborative processing result; when the collaborative processing result is not within the preset validity period, discarding or downgrading the collaborative processing result; when the collaborative processing result is within the preset validity period, matching and verifying the applicable state identifier in the collaborative processing result with the local state label at the current time on the edge; when the applicable state identifier matches the local state label, executing the collaborative behavior strategy based on the edge's real-time state, the robot's current state parameters, the relational memory summary, and the scene constraint parameters; when the applicable state identifier matches the local state label, partially matching the local state label, the collaborative behavior strategy is executed. When the collaborative processing results are inconsistent, only the preset conservative parameters corresponding to the partially consistent states are invoked. When the applicable state identifier is inconsistent with the local state label, the collaborative processing results are discarded or downgraded. When generating or adjusting the collaborative behavior strategy, the robot's current state parameters are used as a mimicry life state in the decision-making process. When the robot's current state parameters conflict with the edge-side instantaneous state or the cloud-side fine-grained state, a comprehensive decision is made based on the robot's current state parameters, the edge-side instantaneous state, the cloud-side fine-grained state, the relationship stage state, the scene constraint parameters, and the behavior priority rules. One or more of the following processing methods are then executed: constraint, downgrade, delayed execution, partial execution, alternative execution, or prohibition of execution.

7. The user state recognition and behavior decision-making method based on end-to-cloud collaboration according to claim 6, characterized in that, The robot's current state parameters include one or more of the following: mimicking emotional state, activity state, system state, energy state, safety state, interaction mode state, and hardware state; the mimicking life state is used to determine response willingness parameters, interaction intensity parameters, and behavioral rhythm parameters; the response willingness parameters, interaction intensity parameters, and behavioral rhythm parameters are used to constrain, downgrade, enhance, delay execution, partially execute, substitute execution, or prohibit execution of the collaborative behavior strategy or the basic behavior strategy.

8. The user state recognition and behavior decision-making method based on end-to-cloud collaboration according to claim 5, characterized in that, The incremental strategy result is used to adjust the multidimensional behavioral parameter set based on the real-time state of the endpoint, the current state parameters of the robot, the locally cached relational memory digest, and behavioral constraint parameters, and in combination with the relational memory digest. The multidimensional behavioral parameter set includes one or more of the following: eye gaze state parameters, eye gaze animation parameters, eye gaze emotion parameters, voice content parameters, voice output level, purring content parameters, purring rhythm, respiratory fluctuation amplitude, heart rate rhythm, kneading frequency, kneading duration, kneading force, head movement amplitude, tail movement frequency, tail movement amplitude, limb movement parameters, torso posture adjustment parameters, movement control parameters, and light brightness or system mode switching parameters. The endpoint calls the corresponding behavioral asset in the behavioral asset library based on the incremental strategy result and performs dynamic adjustments on the called behavioral asset in combination with the real-time state of the endpoint and the current state parameters of the robot.

9. The user state recognition and behavior decision-making method based on end-to-cloud collaboration according to claim 6, characterized in that, S6 include: When the connection status of the network connection parameter in the cloud-edge collaborative control parameters is disconnected, the upload permission parameter indicates that uploading is prohibited, or the collaborative processing result is not returned within the preset validity period, an offline behavior queue is generated based on the most recent collaborative behavior strategy that passed the validity period verification and status matching verification, the local rule table, the cached user preference records, and the locally cached behavior assets. When a wake word event, touch soothing event, hugging event, or sleep-related event is detected, the basic companionship behavior corresponding to the local status tag is preferentially called from the offline behavior queue. The sleep-related events include at least one of the following: bedtime event, falling asleep event, nighttime wake-up event, morning wake-up event, or sleep mode switching event. After the network connection is restored, the status judgment, interaction record, behavior execution result, and user feedback record generated during the offline period are synchronized to the cloud side in chronological order.

10. A user state recognition and behavior decision-making system based on edge-cloud collaboration, used to execute the user state recognition and behavior decision-making method based on edge-cloud collaboration as described in any one of claims 1 to 9, characterized in that, include: The multimodal and robot state acquisition module is used to collect multimodal data of user interaction with the companion robot and the robot's current state data, and generate end-side multimodal features, robot current state parameters, and end-to-cloud collaborative control parameters; the end-side coarse-grained recognition module is used to generate end-side instantaneous state, local state label, end-side recognition confidence parameter, state uncertainty parameter, multidimensional behavior parameter set, and basic behavior strategy based on the end-side multimodal features, the robot's current state parameters, locally cached relational memory summary, and behavior constraint parameters; The edge-cloud traffic gating module is used to generate a data packet to be uploaded based on the state uncertainty parameters, network connection parameters, power supply status parameters, upload permission parameters, scene task identifier, historical upload results, model inference effect, user feedback results, and running resource status, and select the edge-side immediate execution path, the edge-side conservative execution path, or the cloud-side enhanced path; the cloud-side fine understanding and strategy optimization module is used to generate a cloud-side fine state, behavior asset library, and collaborative processing results based on the data packet to be uploaded, historical interaction records, long-term user state model, relation memory, and the robot's current state parameters. The edge-side collaborative execution module is used to generate edge-side execution control results based on the edge-side real-time state, the robot's current state parameters, and the basic behavior strategy when selecting the edge-side immediate execution path or the edge-side conservative execution path. When selecting the cloud-side enhanced path, it performs validity period verification and matching verification on the collaborative processing results and generates collaborative behavior strategies based on the edge-side real-time state, the robot's current state parameters, and the collaborative processing results. The offline continuous companionship and synchronization module is used to maintain basic companionship behavior when the network connection is lost, upload is restricted, or the collaborative processing results are unavailable. It calls the offline behavior queue and synchronizes cached records after the network connection is restored.

Images