Toy interaction adaptive recognition and feedback method based on multi-modal perception and reinforcement learning

Abstract

This invention discloses a multimodal perception and reinforcement learning-based adaptive recognition and feedback method for toy interactions. The method includes acquiring and preprocessing multimodal temporal signals, inputting them into a lightweight behavior classification network to output category labels and confidence scores; constructing a low-dimensional state vector based on labels, confidence scores, user identity, and historical interaction codes; inputting this vector into a decision network employing a proximal policy optimization algorithm to generate feedback actions, wherein the confidence score dynamically adjusts the entropy coefficient to enhance exploratory behavior; executing instructions and calculating reward values, updating the decision network parameters locally using a double-buffering mechanism; and uploading the parameters, after differential privacy noise enhancement and homomorphic encryption, to the cloud for federated aggregation when conditions are met. This invention solves the problems of information gaps between recognition and decision-making, as well as long-term interaction degradation, enabling continuous evolution and privacy protection of intelligent toy interaction strategies.

Description

Technical Field

[0001] This invention relates to the field of intelligent human-computer interaction and artificial intelligence application technology, specifically to a method for the continuous evolution of intelligent toy interaction strategies based on a two-level decision-making architecture and a confidence transfer mechanism. Background Technology

[0002] With the rapid development of artificial intelligence and sensor technology, smart toys have evolved from simple sound and light-responsive products into interactive devices with perception, cognition, and decision-making capabilities. Currently, interactive control technology based on multimodal perception and behavior recognition has become mainstream. By integrating technologies such as computer vision, speech recognition, and tactile sensing to analyze user status, it aims to improve the accuracy and fun of interaction and meet children's needs for personalized companionship and education during their growth.

[0003] However, existing technologies still have significant shortcomings in interactive control: First, the behavior recognition module and the decision-making module are independent of each other, and the confidence score generated by recognition is not transmitted to the decision-making end. This leads to toys being unable to perceive the reliability of input and frequently responding incorrectly in environments with ambiguous behavior boundaries or high noise. Second, the feedback strategy mainly relies on preset rules or finite sets, lacking the long-term adaptive evolutionary ability based on reinforcement learning, making it difficult to cope with changes in user behavior patterns over time and the problem of "long-term interaction degradation". In addition, cloud-based inference has the risk of privacy leakage and latency, while local lightweight models are often insufficient in accuracy. Furthermore, existing technologies have failed to effectively resolve the contradiction between real-time performance, accuracy, and privacy protection on embedded devices, resulting in a decline in user experience after long-term use. Summary of the Invention

[0004] This invention aims to solve the technical problems in existing intelligent toy interactive control technologies, such as the information gap between recognition and decision-making, the lack of long-term adaptive evolution capability of feedback strategies, and the difficulty in balancing real-time performance, accuracy, and privacy protection between cloud inference and local deployment. It provides a multimodal perception and reinforcement learning-based adaptive recognition and feedback method for toy interaction.

[0005] To address the aforementioned technical problems, this invention provides a multimodal perception and reinforcement learning-based adaptive recognition and feedback method for toy interactions, comprising the following steps:

[0006] Acquire multimodal time-series signals collected by the intelligent toy sensor, and preprocess the multimodal time-series signals;

[0007] The preprocessed multimodal time-series signal is input into a lightweight behavior classification network, which outputs the category label and corresponding confidence score of the current interaction behavior; wherein, the lightweight behavior classification network is obtained from a cloud-based teacher model through knowledge distillation technology;

[0008] A low-dimensional state vector is constructed based on the category label, the confidence score, the user identity identifier, and the historical interaction code; wherein, the user identity identifier is obtained by long-term statistical implicit recognition of multimodal identity features, and the historical interaction code is obtained by temporal encoding of the most recent interactions through a lightweight recurrent neural network.

[0009] The low-dimensional state vector is input into a reinforcement learning decision network employing a proximal policy optimization algorithm to generate feedback action instructions. The objective function of the decision network includes an entropy regularization term and an auxiliary loss term. The confidence score serves as a dynamic adjustment factor for the entropy coefficient. When the confidence score is lower than a preset threshold, the entropy coefficient is increased to enhance the exploratory nature of the action. The auxiliary loss term is used to force a negative correlation between the action entropy output by the policy network and the input confidence score.

[0010] Execute the feedback action instruction and monitor the user's immediate response to the feedback action instruction to calculate the reward value;

[0011] The parameters of the decision network are updated locally based on the reward value. During the update process, a double buffering mechanism is used to maintain the current policy network and the candidate policy network. The candidate policy network replaces the current policy network after offline verification that its performance has not degraded.

[0012] When the preset conditions are met, the local model update parameters are uploaded to the cloud server for federated aggregation using a combination of differential privacy noise addition and lightweight homomorphic encryption, and the updated global model parameters are received.

[0013] Furthermore, the preset threshold is dynamically determined through the following steps:

[0014] Maintain a sliding window that includes the most recent classification results, and record the average confidence score and standard deviation of the confidence score within the sliding window;

[0015] The dynamic threshold is calculated in real time based on a linear combination of the average confidence level and the standard deviation of the confidence level;

[0016] The ambient noise level is obtained, and when the ambient noise level exceeds the historical mean plus twice the standard deviation, the dynamic threshold is increased.

[0017] Furthermore, it also includes long-term interaction degradation monitoring and intervention steps:

[0018] Continuously monitor the trend of average reward value changes per unit time period;

[0019] A degradation warning is triggered when the average number of daily interactions reaches a preset minimum threshold and the degradation ratio of the current reward relative to the highest average reward value in the most recent preset time period reaches a preset percentage.

[0020] In response to the degradation warning, a probe test is performed, and the smart toy is controlled to perform known historical high-reward actions;

[0021] If the reward value obtained by the probe test is lower than the preset normal level, the model is confirmed to be degraded and the model parameters are rolled back to the model version with the highest average reward in the most recent preset time period; if the reward value obtained by the probe test is not lower than the preset normal level, the rollback is canceled.

[0022] Furthermore, before inputting the preprocessed multimodal temporal signal into the lightweight behavior classification network, the process further includes:

[0023] Train teacher models using large-scale labeled interactive data in the cloud;

[0024] The knowledge of the teacher model is transferred to the student network through knowledge distillation to obtain the lightweight behavior classification network.

[0025] In particular, a temporal alignment constraint is added during the distillation process to align the feature sequence of the teacher model in the temporal dimension with the output of the student network at the corresponding temporal position.

[0026] Furthermore, the step of constructing a low-dimensional state vector based on the category label, the confidence score, the user identity identifier, and the historical interaction code also includes:

[0027] Multimodal identity features are processed using a lightweight identity classifier to output the probability distribution of user identities.

[0028] When the identity confidence level is higher than the preset identity threshold, the historical interaction code of the corresponding user is used;

[0029] When the identity confidence level is lower than the preset identity threshold, a hybrid code is obtained by weighting the historical interaction codes of each user according to the probability distribution of the user identity.

[0030] When an identity switching signal is detected, the update of the historical interaction code is frozen until identity re-identification is completed.

[0031] Furthermore, the process of uploading local model update parameters to a cloud server for federated aggregation using a combination of differential privacy noise addition and lightweight homomorphic encryption includes:

[0032] Differential privacy noise reduction is applied to the parameters of the local model update.

[0033] A lightweight homomorphic encryption scheme is used to encrypt the noisy parameters;

[0034] The encrypted parameters are uploaded to the cloud server in fragments. If the upload is interrupted, the transmission will resume from the breakpoint.

[0035] The cloud server completes the aggregation operation in encrypted form, generates new global basic model parameters, and distributes them.

[0036] Furthermore, the reward value is a multimodal composite reward, including a weighted combination of at least two of the following: a behavioral response reward based on user behavior response, an emotional tendency reward based on user voice fundamental frequency and energy characteristics, an interaction duration reward based on interaction duration, and a consistency reward based on the correlation between user reaction and feedback action.

[0037] Furthermore, it also includes dynamic precision scheduling steps:

[0038] Obtain the current battery level and processor load of the smart toy;

[0039] The inference frequency and / or feature extraction dimension of the lightweight behavior classification network are adjusted based on the current battery level and processor load.

[0040] In low-power mode, low-precision samples are given a higher priority weight than standard samples for empirical playback, and the compressed feature dimensions are interpolated and completed using historical statistical values.

[0041] Furthermore, the lightweight behavior classification network uses a depthwise separable convolutional network as its backbone network, with fewer than 50,000 parameters, and runs on the embedded processor of the smart toy.

[0042] Furthermore, the policy network of the reinforcement learning decision network employing the proximal policy optimization algorithm is a three-layer fully connected network, and the dimension of the low-dimensional state vector is between thirty and fifty dimensions.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] By outputting category labels and confidence scores through a lightweight behavior classification network, and using the confidence scores as the state vector dimension to input reinforcement learning decision network, and introducing an auxiliary loss term into the objective function to force the action entropy to be negatively correlated with the confidence, the problem of information gap caused by the independence of recognition and decision can be solved. This enables the decision network to perceive the reliability of the input and automatically enhance exploratory behavior or perform interrogative behavior when the confidence is low, thereby significantly reducing the false response rate in fuzzy interaction scenarios.

[0045] By using the confidence score as a dynamic adjustment factor for the entropy coefficient and dynamically adjusting the preset threshold based on sliding window statistics and environmental noise levels, the problem that fixed thresholds cannot adapt to different user habits and environmental changes can be solved, realizing an adaptive decision-making strategy of "high exploration with low confidence and high utilization with high confidence", further improving the naturalness and accuracy of the interaction.

[0046] By employing a double-buffering mechanism to maintain the current policy network and candidate policy network, and combining differential privacy noise addition and lightweight homomorphic encryption for federated aggregation, it is possible to achieve continuous online evolution of the local model and the introduction of collective intelligence while protecting children's data privacy. This solves the problems of feedback policy solidification and long-term interaction degradation, enabling toys to have the ability to continuously grow and optimize as they are used.

[0047] By constructing a three-layer composite degradation monitoring and intervention mechanism that includes interaction frequency filtering, relative degradation rate detection, and probe testing, and rolling back to the historical best model version after degradation is confirmed, the performance degradation problem of smart toys caused by changes in user behavior or model drift during long-term use can be systematically solved, ensuring the long-term stability of the interactive experience.

[0048] By distilling knowledge from a cloud-based teacher model to a student network, and adding time-sequence alignment constraints during the distillation process, as well as dynamic precision scheduling based on battery power and processor load, it is possible to maximize inference accuracy and response speed under the constraints of computing power and power consumption in embedded devices. This resolves the contradiction between high latency and privacy risks in cloud-based inference and low accuracy in local deployment. Attached Figure Description

[0049] Figure 1 This is a flowchart illustrating the toy interaction adaptive recognition and feedback method based on multimodal perception and reinforcement learning of the present invention.

[0050] Figure 2 This is a system principle block diagram of the present invention.

[0051] Figure 3 This is a schematic diagram of knowledge distillation and timing alignment constraints in this invention.

[0052] Figure 4 This is a schematic diagram of the low-dimensional state vector construction in this invention.

[0053] Figure 5 This is a flowchart of the confidence gating mechanism and dynamic threshold adaptive process in this invention.

[0054] Figure 6 This is a flowchart of the double buffering mechanism and offline verification process in this invention.

[0055] Figure 7 This is a flowchart of the long-term interactive degradation monitoring and intervention process in this invention.

[0056] Figure 8 This is a flowchart of federated aggregation and privacy protection in this invention.

[0057] Figure 9 This is a schematic diagram of the dynamic precision scheduling and low power consumption compensation mechanism in this invention. Detailed Implementation

[0058] The technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0059] This invention provides a toy interaction adaptive recognition and feedback method based on multimodal perception and reinforcement learning. The method can be applied to various types of intelligent interactive devices, such as intelligent plush toys, intelligent educational robots, electronic pets, or interactive dolls with multimodal perception capabilities. The following uses intelligent plush toys as an example to describe the technical solution of this invention in detail. However, it should be understood that this invention is not limited to this specific application scenario. Any intelligent terminal with data acquisition, local computing, and communication capabilities can be used with this solution.

[0060] like Figure 2 As shown, the present invention includes a perception module, a processing module, an execution module, and a communication module at the hardware level. The perception module consists of multiple types of sensors used to capture user physical operations and environmental information in real time. The processing module is the core computing unit, deployed with a behavior classification network and a reinforcement learning decision network, responsible for signal processing, feature extraction, state construction, and policy generation in the two-level decision architecture. The execution module includes an audio-visual device, a motion-driven mechanism, or a display unit, used to present feedback actions to the user. The communication module is responsible for establishing a secure connection with the cloud server when preset conditions are met, performing encrypted uploading of model parameters and downloading of global parameters.

[0061] See Figure 1 Step S1: Acquire the multimodal time-series signal collected by the intelligent toy sensor, and preprocess the multimodal time-series signal.

[0062] Preferably, the hardware configuration of the intelligent toy can be adjusted according to the specific product form. In the intelligent plush toy embodiment, the hardware platform uses an ESP32-S3 dual-core processor as the main control chip. The processor has a built-in AI acceleration instruction set, which can meet the real-time inference requirements of lightweight neural networks. The perception module specifically includes: a six-axis inertial measurement unit (IMU) for collecting three-axis acceleration data and three-axis angular velocity data to identify limb movements such as patting and shaking; a single microphone for collecting voice commands and environmental noise; a capacitive touch array for detecting the user's touch position and force; and an RGB LED light ring as the main visual feedback output interface. In the intelligent educational robot embodiment, the hardware configuration can be further expanded to include a high-definition camera for facial expression recognition and a servo motor for realizing complex movements such as head rotation or arm waving.

[0063] Step S1 specifically includes the following sub-steps.

[0064] S11: Multimodal signal acquisition; Real-time acquisition of raw signals during user interaction through multiple types of sensors built into the smart toy, including a six-axis inertial measurement unit, a microphone, and a capacitive touch sensor; wherein, the six-axis inertial measurement unit is used to acquire three-axis acceleration data and three-axis angular velocity data to capture the amplitude, frequency, and direction of the user's limb movements; the microphone is used to acquire ambient audio and user voice commands to obtain acoustic characteristics; the capacitive touch sensor is used to detect the contact position, contact area, and pressure changes between the user and the toy surface.

[0065] S12: Filtering and denoising; The acquired raw multimodal signals are filtered and denoised. For signals acquired by the inertial measurement unit, Kalman filtering or complementary filtering algorithms are used to remove high-frequency noise and gravity component interference, and to extract the true motion acceleration and angular velocity. For audio signals acquired by the microphone, a bandpass filter is used to remove low-frequency environmental noise and high-frequency electronic interference, and spectral subtraction is used to further suppress background noise. For capacitive touch signals, a moving average filter is used to eliminate signal jitter and ensure stable detection of touch events.

[0066] S13: Time synchronization; Time synchronization is performed on the filtered multimodal signals. Since the sampling rates of different sensors may differ (for example, the IMU sampling rate is 100Hz, while the audio sampling rate is 16kHz), the system maps each modal data to a unified time axis through hardware interrupts or software timestamp alignment mechanisms. Using the clock of the sensor with the highest sampling rate as a reference, linear interpolation or nearest neighbor interpolation is performed on other low sampling rate data to generate time-aligned multimodal data frames.

[0067] S14: Sliding window segmentation; The multimodal signal after time synchronization is processed by sliding window segmentation. A fixed-length time window (e.g., 500 milliseconds or 1 second) and a sliding step size (e.g., 100 milliseconds) are set to cut the continuous data stream into overlapping or non-overlapping data segments. Each data segment is an independent interactive event sample, containing the temporal data of all modalities within that time period. The segmentation process not only reduces the input dimension of the subsequent neural network, but also helps to capture dynamic behavioral features in a short period of time, providing standardized input data for subsequent behavior classification.

[0068] Step S2: Input the preprocessed multimodal time series signal into the lightweight behavior classification network, and output the category label and corresponding confidence score of the current interaction behavior.

[0069] Preferably, the lightweight behavior classification network uses a depthwise separable convolutional network as its backbone. This network architecture is designed to adapt to the computational and memory constraints of embedded processors in smart toys, with its number of parameters strictly controlled to below 50,000 (e.g., approximately 40,000 parameters). By using depthwise separable convolutions instead of standard convolution operations, computational complexity and model size can be significantly reduced while maintaining feature extraction capabilities, thereby ensuring millisecond-level real-time inference response on mainstream microcontrollers such as ESP32-S3 and Cortex-M, or low-power AI acceleration chips.

[0070] like Figure 3 As shown, in step S2, the lightweight behavior classification network is obtained from the cloud teacher model through knowledge distillation technology. The specific construction process is as follows (this process is performed before step S2).

[0071] First, a complex and highly accurate teacher model is trained on a cloud server using a large-scale labeled interactive dataset. This teacher model incorporates more layers, a wider number of channels, and an attention mechanism to fully capture the deep semantic features in multimodal time-series signals.

[0072] Secondly, knowledge from the teacher model is transferred to the student network (i.e., the aforementioned lightweight behavior classification network) with a very small number of parameters through knowledge distillation. In the knowledge distillation process, not only is a soft-label loss function used to match the probability distribution of the output layers of the teacher model and the student network, but a temporal alignment constraint is also introduced. Since multimodal sensor data (such as accelerometer and gyroscope data) has strong temporal series dependencies, and lightweight static convolutional networks are relatively weak in long-term temporal modeling capabilities, the temporal alignment constraint forces the student network to maintain consistency with the teacher model's feature sequences at corresponding temporal positions (i.e., minimize the distance between the two). This alignment mechanism compensates for the lightweight network's lack of temporal context understanding, enabling the distilled student network to achieve temporal classification accuracy close to that of the teacher model while retaining the advantage of low computational power.

[0073] In step S2, the lightweight behavior classification network receives preprocessed multimodal temporal signals as input and outputs the category label of the current interaction behavior (such as "patting", "shaking", "stroking", "stillness" etc.) and the corresponding confidence score. The confidence score is a continuous value between 0 and 1, reflecting the degree of certainty of the classification network regarding the current recognition result. This confidence score will be used as a core transmission signal to be transmitted to the subsequent decision module to guide the generation of adaptive strategies.

[0074] Furthermore, to better adapt to the differences between different hardware platforms, the structure of the lightweight behavior classification network is replaceable. In scenarios with higher requirements for temporal dependencies and sufficient computing power, a temporal convolutional network or a lightweight Transformer architecture can be used instead of a deep separable convolutional network. In scenarios with extremely low computing power, it can be simplified to a two-layer fully connected network that only identifies the most basic behavior categories. Regardless of the specific architecture used, its core is to obtain prior knowledge through knowledge distillation and to complete high-precision behavior classification and confidence assessment locally with extremely low resource consumption.

[0075] Step S3: Construct a low-dimensional state vector based on the category label, the confidence score, the user identity identifier, and the historical interaction code.

[0076] like Figure 4 As shown, step S3 aims to transform the high-dimensional, heterogeneous raw sensor data into a compact state representation suitable for processing by reinforcement learning decision networks.

[0077] Preferably, the low-dimensional state vector is composed of four core components: one-hot encoding of behavior category, recognition confidence scalar, user identity embedding vector, and historical interaction time-series encoding. The one-hot encoding of behavior category is directly derived from the category label output by the lightweight behavior classification network in step S2. The recognition confidence scalar is a continuous value normalized to the [0, 1] interval, used to characterize the reliability of the current behavior recognition. The user identity embedding vector is generated through an implicit identity recognition module. The historical interaction time-series encoding is generated through a lightweight recurrent neural network. The total dimension of the low-dimensional state vector is controlled between thirty and fifty dimensions to adapt to the computational resource constraints of the embedded processor while ensuring sufficient information representation capability. The policy network of the reinforcement learning decision network using the near-end policy optimization algorithm is a three-layer fully connected network.

[0078] Regarding the implicit identification of user identity, this implementation adopts a non-intrusive multimodal feature fusion scheme, which does not require users to actively log in or wear specific identification devices. The system processes the collected multimodal identity features through a lightweight identity classifier. The multimodal identity features include behavioral mechanical features (such as the distribution of slapping force and shaking frequency spectrum), lightweight voiceprint features (such as voice fundamental frequency statistics and speech rate), and usage time patterns (such as the distribution of active time periods). The lightweight identity classifier outputs a user identity probability distribution vector, which represents the probability that the current interactor belongs to each user in the known user set.

[0079] To address scenarios involving multiple users alternating or ambiguous identities, this implementation introduces a confidence-based hybrid coding strategy. The system presets an identity confidence threshold. When the maximum probability value in the user identity probability distribution is higher than the preset threshold, the identity recognition is deemed reliable, and the historical interaction code of the user with the highest probability is directly selected as the identity component in the state vector. When the maximum probability value is lower than the preset threshold, the identity is deemed uncertain. In this case, instead of relying solely on the historical data of a single user, a hybrid code is calculated by weighting and summing the historical interaction codes of each known user according to the user identity probability distribution. This hybrid code is then used as the identity component in the state vector. This dynamic switching mechanism ensures that personalized historical preferences can be utilized when the identity is clear, while maintaining robustness in decision-making when the identity is ambiguous.

[0080] To address the common scenario of rapid rotation or temporary intervention by multiple people in children's toy scenarios, this implementation design incorporates an identity switching detection and freezing mechanism. The system continuously monitors abrupt changes in input signals (such as sudden jumps in voiceprint features or significant deviations in behavioral patterns) as identity switching signals. When an identity switching signal is detected, the system immediately initiates the identity re-identification process. Before the re-identification is completed and a new stable identity is established, the system freezes the update operations of historical interaction codes. At this time, the state vector written into the experience replay pool remains at the value at the last moment before the switch or is reset to a zero vector, and no new interaction data is included for recursive updates. The freezing mechanism effectively prevents noise data or incorrectly assigned data generated during the identity transition period from polluting the historical memory of the decision network.

[0081] The historical interaction encoding is obtained by temporally encoding the state-action pairs of the most recent interactions using a lightweight recurrent neural network (such as a Long Short-Term Memory network LSTM or a gated recurrent unit GRU). Specifically, the input of the recurrent neural network is the sequence of category labels and corresponding confidence scores of the most recent N interactions, and the output is a fixed-length hidden state vector as the historical interaction encoding. The recurrent neural network only retains the hidden layer states as the encoding output, thereby capturing the temporal dependence of the interaction behavior with extremely low storage overhead. By concatenating the above four components in the feature dimension, a low-dimensional state vector is finally formed for input into the reinforcement learning decision network.

[0082] Step S4: Input the low-dimensional state vector into a reinforcement learning decision network using a proximal policy optimization algorithm to generate feedback action instructions.

[0083] Specifically, the reinforcement learning decision network is trained and inferred using a proximal policy optimization algorithm; the input of the decision network is the low-dimensional state vector constructed in step S3, which not only contains the behavior category label, but also incorporates the confidence score output by the behavior classification network in step S2; by using the confidence score as an explicit dimension of the state space, the policy network can perceive the reliability of the current behavior recognition.

[0084] like Figure 5 As shown, the objective function of the decision network includes an entropy regularization term and an auxiliary loss term; the confidence score, as a dynamic adjustment factor of the entropy coefficient, is used to control the balance between policy exploration and utilization; when the confidence score is lower than the confidence threshold, it indicates that there is a large uncertainty in the current behavior recognition, and the system automatically increases the entropy coefficient, making the probability distribution of the action output by the policy network more dispersed, thereby enhancing the exploratory nature of the action, manifested as outputting interrogative, confirmatory, or diversified feedback actions, to avoid incorrect fixed responses caused by misclassification; when the confidence score is higher than or equal to the confidence threshold, the entropy coefficient reverts to the baseline value or decreases, and the policy network tends to utilize existing knowledge to output feedback actions with higher certainty.

[0085] The auxiliary loss term is used to force a negative correlation between the action entropy output by the policy network and the confidence score of the input. By minimizing this auxiliary loss term, the model is prompted to maintain a high action entropy in a low confidence state and a low action entropy in a high confidence state, thereby solidifying the confidence gating mechanism at the mathematical level.

[0086] The confidence threshold is determined through the following dynamic adaptive mechanism: First, the system maintains a sliding window containing the most recent classification results (e.g., 100), and records and updates the average confidence and standard deviation of the confidence within the sliding window in real time; Second, the system calculates the basic dynamic threshold in real time based on a linear combination of the average confidence and the standard deviation of the confidence, for example, setting the basic dynamic threshold as: average confidence - k × standard deviation of confidence, where k is an adjustment coefficient; Finally, the system obtains the current ambient noise level, which is comprehensively evaluated by combining the volume decibel value collected by the microphone and the vibration intensity detected by the inertial measurement unit. When the environmental noise level exceeds the historical mean plus twice the standard deviation, it is determined that the current environment is in a high-noise interference environment. The system automatically increases the dynamic threshold to reduce false triggering or invalid exploration caused by low confidence due to sensor noise. In addition, for specific users (hereinafter referred to as "specific users") who have been identified by the system and included in the known user set through the multimodal identity implicit recognition mechanism in step S3, the system maintains independent interaction history and confidence statistics for the user. When the number of interaction samples between the specific user and the toy reaches a preset number (e.g., 200 classification results), the system automatically switches to the personalized threshold mode, that is, adjusts the threshold benchmark based on the historical average confidence distribution and personal behavior habits of the specific user to further improve the accuracy and adaptability of the interaction with the user.

[0087] Step S5: Execute the feedback action instruction and monitor the user's immediate response to the feedback action instruction to calculate the reward value.

[0088] In some implementations, step S5 controls the actuator of the smart toy to output corresponding sound, light, mechanical or voice feedback, and continuously collects multimodal response data from the user within a preset time window after the feedback is issued through a built-in sensor array.

[0089] The reward value is a multimodal composite reward, which is calculated by weighting and combining the following four sub-reward components.

[0090] The first sub-reward component: behavioral response reward; the first sub-reward component is calculated based on the user's behavioral response signal. The smart toy collects the user's limb movement data after receiving feedback through accelerometers, gyroscopes, and capacitive touch sensors. For example, when the toy performs a stroking feedback, if it detects that the user pats or continues to touch the surface of the toy again, and the force is within the preset pleasure range, it is judged as a positive behavioral response and given a higher behavioral response reward score. If it detects that the user shakes violently or does not operate for a long time, it is given a lower or negative behavioral response reward score, which is used to quantify the user's physical acceptance of the current feedback action.

[0091] The second sub-reward component is the emotional tendency reward. This component is calculated based on the user's speech fundamental frequency and energy characteristics. The smart toy collects the user's speech or non-verbal vocal signals through a microphone, extracting acoustic features such as the fundamental frequency, energy, and zero-crossing rate of the speech signals. A lightweight emotion recognition model maps these acoustic features into an emotional polarity score. For example, when a user emits a high-pitched, high-energy laugh or cheer, it is determined to be a positive emotion, and a positive emotional tendency reward is given. When a low-pitched, low-energy crying or complaining sound is detected, it is determined to be a negative emotion, and a negative emotional tendency reward is given. This second sub-reward component enables the decision network to perceive changes in the user's emotional state.

[0092] The third sub-reward component: interaction duration reward; the third sub-reward component is calculated based on the interaction duration. The system records the time interval from the start of the current feedback action to the user initiating the next interaction or the end of the interaction session. If the time interval exceeds the preset basic threshold, it indicates that the current feedback has successfully extended the user's attention dwell time. The interaction duration reward is then increased linearly according to the proportion of the excess time. The third sub-reward component encourages the decision network to generate feedback strategies that can maintain the user's long-term interest.

[0093] The fourth sub-reward component is the consistency reward. This component is calculated based on the correlation between user reactions and feedback actions. The system analyzes the matching degree between the user's current behavior pattern and expected reactions in similar scenarios in historical interaction sequences. For example, in a specific game context, if the user performs a corresponding action that conforms to the game rules after the toy emits a prompt sound, it is considered high consistency and awarded a higher consistency reward. If the user's reaction deviates from the intended feedback, it is considered low consistency and awarded a lower consistency reward. The fourth sub-reward component is used to strengthen the establishment of a stable interaction protocol between the toy and the user.

[0094] The weight coefficients of the above four types of sub-reward components can be dynamically adjusted according to the current operating mode of the smart toy. For example, in the education mode, the weight of consistency rewards and behavioral response rewards is increased, while the weight of emotional tendency rewards is decreased; in the companionship mode, the weight of emotional tendency rewards and interaction persistence rewards is increased. The final multimodal composite reward value is used as the immediate reward signal in the reinforcement learning algorithm and input to the decision network.

[0095] Step S6: Update the parameters of the decision network locally based on the reward value.

[0096] like Figure 6 As shown, step S6 employs a double-buffering mechanism to maintain the current policy network and the candidate policy network; this aims to address the non-stationarity problem in the online training process of reinforcement learning and prevent drastic fluctuations in policy performance or catastrophic forgetting caused by a single update.

[0097] Specifically, the system memory stores two sets of neural network weights with the same structure but independent parameters: one set is the current policy network, which is used to receive low-dimensional state vectors in real time and output feedback action instructions; the other set is the candidate policy network, which is used to perform gradient updates and parameter iterations in the background. At the current moment, all state, action and reward tuples generated by user interactions are first stored in the experience replay buffer, and then used to update the parameters of the candidate policy network.

[0098] During the update process, the candidate policy network calculates gradients and adjusts weights based on the objective function of the near-end policy optimization algorithm, using samples in the experience replay buffer. This is done asynchronously in the background without blocking the main interaction thread, thus ensuring the toy's real-time responsiveness to user actions.

[0099] To ensure that the performance of the updated model is not lower than that of the original model, this implementation introduces an offline verification mechanism. When the number of update steps accumulated by the candidate policy network reaches the verification trigger step threshold, or the time interval reaches the preset period, the system pauses further updates to the network and starts the offline verification process. In the offline verification stage, the system extracts a set of representative historical interaction samples from the experience replay buffer as a verification set, inputs them into the current policy network and the candidate policy network respectively, and calculates the expected cumulative reward value or average instantaneous reward value of both on the verification set.

[0100] If the candidate policy network performs better than or equal to the current policy network on the validation set, the update is deemed valid. In this case, the system completely copies the weights of the candidate policy network and overwrites them with the current policy network, completing a policy switch. After the switch is completed, the weights of the original current policy network are archived, and the new candidate policy network performs the next round of background updates based on the latest parameters.

[0101] If the performance metrics of the candidate policy network on the validation set are significantly lower than those of the current policy network, the update is deemed invalid and performance degradation occurs. In this case, the system discards the update results of the candidate policy network, retains the parameters of the current policy network unchanged, and resets the candidate policy network, allowing it to clone the parameters from the current policy network and start a new round of background learning. Through the double buffering mechanism of "background training, foreground validation, and optimal replacement", inferior policies caused by noisy data or exploration bias can be effectively filtered out.

[0102] Step S7: When the preset conditions are met, the local model update parameters are uploaded to the cloud server for federated aggregation using a combination of differential privacy noise addition and lightweight homomorphic encryption, and the updated global model parameters are received.

[0103] like Figure 8 As shown, step S7 specifically includes the following sub-steps.

[0104] S7-1: Differential Privacy Noise Addition; After the smart toy completes the parameter update of the decision network locally, it calculates the weight difference between the current policy network and the previous version of the policy network as the model update parameter to be uploaded. In order to further protect user privacy and prevent the original interaction data from being restored from the gradient information through reverse engineering, the system adds random noise that conforms to the Laplace distribution or Gaussian distribution to the model update parameter before uploading. The scale of the noise is dynamically adjusted according to the preset privacy budget to ensure that the effectiveness of the model update is preserved to the greatest extent while meeting the definition of differential privacy.

[0105] S7-2: Lightweight homomorphic encryption; Considering the computing power limitations of embedded devices in smart toys, this implementation uses a partially homomorphic encryption algorithm with lower computational overhead (such as the Paillier encryption algorithm or its variants). The system uses a preset public key to encrypt the model update parameters after adding noise, generating ciphertext parameters. Due to the characteristics of homomorphic encryption, the cloud server can directly perform addition or scalar multiplication operations on the ciphertext parameters uploaded by multiple terminals without decrypting the ciphertext, thereby realizing aggregation operations in the ciphertext state.

[0106] S7-3: Segmented Upload and Resume Transmission; To adapt to unstable wireless network environments and reduce the size of data packets in a single transmission, the system divides the encrypted ciphertext parameters into several data segments. Each segment contains a sequence number, a checksum, and a data payload. The smart toy sends these data segments to the cloud server sequentially. If a network interruption or connection timeout occurs during transmission, the smart toy records the sequence number of the last successfully uploaded segment. When the network is restored, the remaining segments are transmitted from the point of interruption (i.e., after the last successfully uploaded segment) without re-uploading the entire model parameters. The system also introduces a version number and timestamp verification mechanism. The cloud server only accepts updated parameters with a version number higher than the local record and a timestamp within the valid window. Terminals that have not participated in federated learning for a long time or expired updates are discarded or have their aggregation weight reduced.

[0107] S7-4: Cloud-based Ciphertext Aggregation and Distribution; The cloud server receives encrypted fragments from multiple smart toys, reassembles them to obtain complete ciphertext model update parameters, and utilizes the properties of homomorphic encryption to perform a weighted average or summation operation on the ciphertext parameters of all participating terminals to obtain the aggregated global ciphertext update quantity; Subsequently, the cloud server uses its private key to decrypt the aggregation result, obtain the plaintext global model update parameters, and integrates them into the global base model to generate new global base model parameters; The newly generated global base model parameters are encrypted again and distributed to each smart toy terminal; After receiving the global model parameters, each toy terminal integrates them with its locally stored personalized parameters, thereby inheriting collective intelligence while maintaining adaptability to specific users.

[0108] Preferably, to address the long-term interaction degradation of smart toys caused by changes in user behavior patterns, shifts in interests, or model solidification during extended use, this method also introduces the following long-term interaction degradation monitoring and intervention mechanism. This mechanism continuously monitors the system's operational status and interaction quality, and can automatically take remedial measures when a significant performance decline is detected, ensuring that the toy always maintains a good interactive experience.

[0109] like Figure 7 As shown, the system continuously monitors the trend of the average reward value change per unit time. In order to distinguish between the actual degradation of model capabilities and the temporary reward fluctuations caused by external environmental factors, this implementation adopts a three-layer composite condition to trigger degradation warning.

[0110] First layer: Interaction frequency filtering; The system counts the average number of interactions of the smart toy per day. Only when the average number of interactions per day reaches the frequency filtering threshold (e.g., 5 times / day) will the subsequent degradation detection logic be activated. If the interaction frequency is lower than the threshold, it is determined that there is insufficient data and the detection will not be triggered for the time being, thus avoiding false triggering caused by the user's own decreased interest (not model degradation).

[0111] The second layer: relative degradation rate detection; under the premise of meeting the frequency condition, calculate the degradation ratio of the average reward value in the current period relative to the highest average reward value in the most recent preset time period, such as the most recent 30 days. Only when the degradation ratio reaches a preset percentage (e.g., 15%), the system initially determines that there may be model degradation and triggers a degradation warning.

[0112] The third layer: probe test verification; after triggering the degradation warning, the system performs a probe test to further confirm whether it is a real degradation. Specifically, the system controls the smart toy to perform a known historical high-reward action. The probe action is a deterministic action selected from the experience playback buffer that has obtained a high reward value in similar scenarios in the past. The system monitors the user's immediate reaction to the probe action and calculates the reward value obtained from the probe test.

[0113] If the reward value obtained by the probe test is lower than the preset normal level, it is confirmed that the model has actually degraded. The system will perform a model rollback operation, rolling back the current decision network parameters to the model version with the highest average reward in the most recent preset time period. The model rollback operation is implemented through a double buffering mechanism: the system maintains two sets of parameters locally, the current policy network and the candidate policy network, and retains model snapshots of several recent historical versions. When it is confirmed that a rollback is needed, the model version parameters with the highest average reward in the most recent period are directly loaded from the storage medium into the current policy network.

[0114] If the reward value obtained by the probe test is not lower than the preset normal level, it is determined that the current performance decline is not caused by the degradation of the model itself (it may be caused by environmental noise or temporary user status), the system cancels the rollback operation, and continues to maintain the current model parameters.

[0115] Preferably, in order to balance the real-time interaction, decision-making accuracy, and device battery life in smart toys with limited embedded computing resources and battery-powered scenarios, this method also includes a dynamic accuracy scheduling step.

[0116] like Figure 9 As shown, the system obtains the current remaining battery percentage of the smart toy and the load rate of the processor or AI acceleration unit in real time. Based on the above status parameters, the system dynamically adjusts the inference frequency and / or feature extraction dimension of the lightweight behavior classification network. When the battery power is sufficient and the processor load is low, the system maintains the standard high-frequency inference mode and enables full-dimensional multimodal feature extraction to ensure the highest recognition accuracy. When the battery power is detected to be lower than the preset warning threshold (e.g., 20%) or the processor load is continuously higher than the preset high load threshold (e.g., 85%), the system automatically switches to low-power mode. In low-power mode, the system reduces the sampling rate of the sensor to reduce the input data frame rate of the behavior classification network, or uses a masking mechanism to block the feature extraction of some non-critical channels, retaining only the core feature dimensions that contribute the most to the current interaction behavior discrimination, thereby reducing the computational load and memory access overhead of a single inference.

[0117] In low-power mode, due to the compression of feature dimensions or the reduction of sampling rate, the state vector input to the subsequent reinforcement learning decision network may be sparse or missing. In order to solve the problem of decreased sample quality in the experience replay pool and unstable model training caused by this, this implementation introduces the following two compensation mechanisms.

[0118] Sample quality weighted compensation: When constructing the experience replay buffer, the system assigns higher priority weights to low-precision samples collected in low-power mode than to standard samples. Specifically, a weight coefficient is introduced into the loss function so that these low-precision samples can generate greater gradient contributions when backpropagating to update the decision network parameters, thereby offsetting the information loss caused by the reduction in data precision.

[0119] Feature interpolation completion: For compressed or discarded feature dimensions, the system uses historical statistical values ​​for interpolation completion. The system maintains a sliding window that records the mean and variance of the corresponding feature dimensions extracted in the most recent high-precision mode. When entering low-power mode, which causes some feature dimensions to be missing, the system extracts the historical statistical mean from the sliding window and combines it with the currently available low-dimensional feature context to generate an estimate of the missing dimension through linear interpolation or a prediction algorithm based on a simple regression model, and fills it into the state vector. This mechanism ensures the consistency of the state vector dimension of the input reinforcement learning decision network and avoids network structure adaptation errors caused by changes in input dimension.

[0120] Example 1

[0121] This invention is implemented in a smart plush toy; it is equipped with a six-axis inertial measurement unit, a single microphone, a capacitive touch strip and an RGB LED light ring, and the main controller is an ESP32-S3 dual-core processor with built-in AI acceleration instructions.

[0122] In step S1, the IMU sampling rate is set to 100Hz, the audio sampling rate is set to 16kHz, the sliding window length is set to 500 milliseconds, and the step size is set to 100 milliseconds.

[0123] In step S2, the classification network adopts a six-layer deep separable convolutional network with approximately 40,000 parameters. Knowledge is transferred from the cloud teacher model through knowledge distillation, and temporal alignment constraints are enabled during the distillation process.

[0124] In step S3, the low-dimensional state vector is composed of one-hot encoding of the category label, confidence score, user identity embedding encoding (8 dimensions), and LSTM encoding of historical interactions (16 dimensions), totaling more than thirty dimensions.

[0125] In step S4, the decision network adopts the proximal policy optimization algorithm. The policy network is a three-layer fully connected network (128 neurons in each hidden layer). The confidence threshold is initially set to 0.7 and dynamically calibrated through a sliding window (window size equal to 100 classification results). The ambient noise level is comprehensively evaluated by microphone volume and IMU vibration intensity. When the noise increases, the threshold is automatically raised. When the classification confidence is not lower than the threshold, the decision network outputs a normal feedback action. When the confidence is lower than the threshold, the action entropy of the policy network automatically increases and outputs an interrogative action. The coefficient λ of the auxiliary loss term in the objective function of the decision network is set to 0.1 to constrain the negative correlation between action entropy and confidence.

[0126] In step S5, the reward function is a multimodal composite reward, which includes behavioral response reward (weight 0.3), sentiment tendency reward (weight 0.3), interaction persistence reward (weight 0.2) and consistency reward (weight 0.2), and the weighted sum of each component.

[0127] In step S6, the threshold for triggering offline verification in the double buffering mechanism is 200 steps, the verification set size is 100 samples, and the candidate policy network is judged to have no performance degradation and is replaced when the average reward on the verification set is not less than 95% of that of the current policy network.

[0128] In step S7, federated learning adopts a model-independent meta-learning framework. Local fine-tuning is triggered once every 200 accumulated experiences. Federated aggregation is triggered every day at midnight during the charging state. Parameter upload uses differential privacy noise addition (ε=0.5) and Paillier homomorphic encryption. Aggregation is completed in the cloud under ciphertext.

[0129] In long-term degradation monitoring, the frequency filtering threshold is set to 5 times / day, the degradation ratio threshold is set to 15%, the preset time period is the most recent 30 days, and the probe test selects the action with the highest average reward in the most recent 7 days as the probe action. If the probe reward is lower than 70% of the historical probe average, degradation is confirmed and rolled back.

[0130] In an eight-week field test involving ten children, the false response rate of this solution was approximately 5%, significantly better than the false response rate of approximately 18% of the traditional solution. The average duration of a single user interaction increased from approximately 6 minutes in the first week to approximately 14 minutes in the eighth week, while the traditional solution decreased from approximately 6 minutes to approximately 3 minutes.

[0131] Example 2

[0132] This invention is implemented in an intelligent educational robot; it has voice interaction and simple action expression capabilities, and is used for children's early English learning.

[0133] In step S2, the classification network adds an emotional classification of children's voice input, expanding the classification categories to twelve (including happy reading aloud, hesitation, anxiety, etc.).

[0134] In step S4, a learning progress term is added to the reward function of the decision network: when the child answers correctly consecutively, an extra positive reward is given to encourage the challenge; when the child shows signs of anxiety, the decision network automatically reduces the difficulty of the questions and adds encouraging voices.

[0135] In step S7, the cloud aggregates the global model once a week. The global model contains interactive experiences from thousands of terminal devices (all of which are encrypted parameters and do not contain raw data). The test results show that children's learning efficiency is about 32% higher than that of the fixed strategy group, and the error rate under anxious emotions is reduced by about 45%.

[0136] Example 3

[0137] This invention is implemented in a low-computing-power pure motion interactive toy; it only has a six-axis inertial measurement unit and a capacitive touch sensor, no microphone or camera, the main controller is a Cortex-M0 processor, and the memory is only 64KB.

[0138] In step S2, the classification network uses a two-layer fully connected network (with only a few thousand parameters) to identify only four types of behaviors: slapping, shaking, touching, and stillness. Since there is no microphone, the reward function in step S5 only uses behavior response reward and interaction duration reward, and distinguishes between excited slapping and angry slapping by force normalization (comparing the current force with the average of the user's historical forces).

[0139] In step S4, the number of neurons in the hidden layer of the decision network is reduced to 64, and the confidence threshold is fixed at 0.65 (a dynamic threshold is not used because there is no ability to assess environmental noise).

[0140] In step S6, due to memory limitations, the capacity of the experience replay buffer is set to 500 entries, and the offline verification step threshold for the candidate policy network is reduced to 50 steps.

[0141] In step S7, federated learning only uploads the key layer parameters of the policy network (approximately 20,000 parameters), further reducing communication overhead. This embodiment demonstrates that the core mechanism of the present invention does not rely on high-end sensors and powerful computing power, and remains effective even on the lowest-configuration toys.

[0142] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multimodal perception and reinforcement learning-based adaptive recognition and feedback method for toy interactions, characterized in that, Includes the following steps: Acquire multimodal time-series signals collected by the intelligent toy sensor, and preprocess the multimodal time-series signals; The preprocessed multimodal time-series signal is input into a lightweight behavior classification network, which outputs the category label and corresponding confidence score of the current interaction behavior; wherein, the lightweight behavior classification network is obtained from a cloud-based teacher model through knowledge distillation technology; A low-dimensional state vector is constructed based on the category label, the confidence score, the user identity identifier, and the historical interaction code; wherein, the user identity identifier is obtained by long-term statistical implicit recognition of multimodal identity features, and the historical interaction code is obtained by temporal encoding of the most recent interactions through a lightweight recurrent neural network. The low-dimensional state vector is input into a reinforcement learning decision network employing a proximal policy optimization algorithm to generate feedback action instructions. The objective function of the decision network includes an entropy regularization term and an auxiliary loss term. The confidence score serves as a dynamic adjustment factor for the entropy coefficient. When the confidence score is lower than a preset threshold, the entropy coefficient is increased to enhance the exploratory nature of the action. The auxiliary loss term is used to force a negative correlation between the action entropy output by the policy network and the input confidence score. Execute the feedback action instruction and monitor the user's immediate response to the feedback action instruction to calculate the reward value; The parameters of the decision network are updated locally based on the reward value. During the update process, a double buffering mechanism is used to maintain the current policy network and the candidate policy network. The candidate policy network replaces the current policy network after offline verification that its performance has not degraded. When the preset conditions are met, the local model update parameters are uploaded to the cloud server for federated aggregation using a combination of differential privacy noise addition and lightweight homomorphic encryption, and the updated global model parameters are received.

2. The method according to claim 1, characterized in that, The preset threshold is dynamically determined through the following steps: Maintain a sliding window that includes the most recent classification results, and record the average confidence score and standard deviation of the confidence score within the sliding window; The dynamic threshold is calculated in real time based on a linear combination of the average confidence level and the standard deviation of the confidence level; The ambient noise level is obtained, and when the ambient noise level exceeds the historical mean plus twice the standard deviation, the dynamic threshold is increased.

3. The method according to claim 1, characterized in that, It also includes long-term interaction degradation monitoring and intervention steps: Continuously monitor the trend of average reward value changes per unit time period; A degradation warning is triggered when the average number of daily interactions reaches a preset minimum threshold and the degradation ratio of the current reward relative to the highest average reward value in the most recent preset time period reaches a preset percentage. In response to the degradation warning, a probe test is performed, and the smart toy is controlled to perform known historical high-reward actions; If the reward value obtained by the probe test is lower than the preset normal level, the model is confirmed to be degraded and the model parameters are rolled back to the model version with the highest average reward in the most recent preset time period. If the reward value obtained from the probe test is not lower than the preset normal level, then the rollback is cancelled.

4. The method according to claim 1, characterized in that, Before inputting the preprocessed multimodal temporal signal into the lightweight behavior classification network, the process also includes: Train teacher models using large-scale labeled interactive data in the cloud; The knowledge of the teacher model is transferred to the student network through knowledge distillation to obtain the lightweight behavior classification network. In particular, a temporal alignment constraint is added during the distillation process to align the feature sequence of the teacher model in the temporal dimension with the output of the student network at the corresponding temporal position.

5. The method according to claim 1, characterized in that, The process of constructing a low-dimensional state vector based on the category label, the confidence score, the user identity identifier, and historical interaction encoding also includes: Multimodal identity features are processed using a lightweight identity classifier to output the probability distribution of user identities. When the identity confidence level is higher than the preset identity threshold, the historical interaction code of the corresponding user is used; When the identity confidence level is lower than the preset identity threshold, a hybrid code is obtained by weighting the historical interaction codes of each user according to the probability distribution of the user identity. When an identity switching signal is detected, the update of the historical interaction code is frozen until identity re-identification is completed.

6. The method according to claim 1, characterized in that, The process of uploading local model update parameters to a cloud server for federated aggregation using a combination of differential privacy noise addition and lightweight homomorphic encryption includes: Differential privacy noise reduction is applied to the parameters of the local model update. A lightweight homomorphic encryption scheme is used to encrypt the noisy parameters; The encrypted parameters are uploaded to the cloud server in fragments. If the upload is interrupted, the transmission will resume from the breakpoint. The cloud server completes the aggregation operation in encrypted form, generates new global basic model parameters, and distributes them.

7. The method according to claim 1, characterized in that, The reward value is a multimodal composite reward, including a weighted combination of at least two of the following: behavioral response reward based on user behavior response, emotional tendency reward based on user voice fundamental frequency and energy characteristics, interaction duration reward based on interaction duration, and consistency reward based on the correlation between user reaction and feedback action.

8. The method according to claim 1, characterized in that, It also includes dynamic precision scheduling steps: Obtain the current battery level and processor load of the smart toy; The inference frequency and / or feature extraction dimension of the lightweight behavior classification network are adjusted based on the current battery level and processor load. In low-power mode, low-precision samples are given a higher priority weight than standard samples for empirical playback, and the compressed feature dimensions are interpolated and completed using historical statistical values.

9. The method according to claim 1, characterized in that, The lightweight behavior classification network uses a depthwise separable convolutional network as its backbone network, with fewer than 50,000 parameters, and runs on the embedded processor of a smart toy.

10. The method according to claim 1, characterized in that, The policy network of the reinforcement learning decision network employing the proximal policy optimization algorithm is a three-layer fully connected network, and the dimension of the low-dimensional state vector is between thirty and fifty dimensions.

Images