Steam teaching robot environment adaptation method based on deep reinforcement learning

Abstract

The application relates to a STEAM teaching robot environment adaptation method based on deep reinforcement learning, which comprises the following steps: S1: collecting classroom multi-modal environment data and preprocessing, and obtaining unified multi-modal data after preprocessing; S2: constructing a classroom semantic map and an environment change factor; S3: multi-dimensionally modeling a student state, and obtaining a student state matrix; S4: constructing a unified state representation processable by a reinforcement learning model; S5: according to the unified state representation processable by the reinforcement learning model, generating a robot teaching adaptation strategy through a hierarchical deep reinforcement learning model, and obtaining a candidate action; S6: performing safety and teaching suitability checking on the robot action, obtaining a robot finally allowed action, and sending the robot finally allowed action to a robot motion control module, a voice interaction module and a teaching content display module. The application effectively improves the classroom perception ability and autonomous adaptation ability of the STEAM teaching robot.

Description

Technical Field

[0001] This invention relates to the field of educational robot optimization, and more particularly to a method for STEAM educational robot environmental adaptation based on deep reinforcement learning. Background Technology

[0002] With the development of artificial intelligence, robotics, and intelligent education technologies, STEAM education is gradually shifting from traditional knowledge-based instruction to interdisciplinary inquiry, project-based learning, and intelligent interactive learning models. In STEAM teaching scenarios, students typically engage in activities such as observation, discussion, experimentation, design, and collaboration related to science, technology, engineering, art, and mathematics. The classroom environment is characterized by diverse task types, complex student behaviors, dynamically changing teaching pace, and frequent use of experimental equipment. Educational robots, as intelligent teaching aids, can perform functions such as knowledge explanation, interactive Q&A, experimental guidance, classroom monitoring, and learning feedback, thus possessing high application value in STEAM teaching scenarios.

[0003] However, existing educational robots still have certain shortcomings in adapting to the classroom environment. On the one hand, some educational robots mainly rely on preset teaching procedures or fixed rules to execute actions, making it difficult to adjust in real time according to changes in student positions, obstacles, noise levels, the status of teaching equipment, and student participation. On the other hand, existing methods typically model student states in a relatively simplistic way, relying more on answer results, voice interaction, or simple facial expression recognition, making it difficult to comprehensively reflect students' attention, emotions, knowledge acquisition, experimental progress, and classroom participation. Since both the classroom environment and student states are highly dynamic and uncertain, if the robot lacks a unified representation of the environment-student-task state, it is easy for teaching actions to become mismatched with the current classroom context, affecting the robot's teaching assistance effectiveness.

[0004] Furthermore, while reinforcement learning methods can be used for robot policy learning and adaptive decision-making, they still face challenges in STEAM educational robot scenarios, such as complex state spaces, multi-layered action spaces, and the coexistence of teaching objectives and safety constraints. Single-level reinforcement learning models struggle to simultaneously handle the selection of teaching objectives, the generation of specific interactive actions, and classroom safety execution constraints, and are prone to generating actions that are unsuitable for the current teaching stage or do not meet classroom safety requirements. Summary of the Invention

[0005] To address the aforementioned problems, the present invention aims to provide a method for STEAM teaching robot environmental adaptation based on deep reinforcement learning, which effectively enhances the classroom perception and autonomous adaptation capabilities of STEAM teaching robots.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: The STEAM teaching robot environmental adaptation method based on deep reinforcement learning includes the following steps: S1: Collect and preprocess classroom multimodal environment data to obtain preprocessed unified multimodal data; S2: Based on the preprocessed unified multimodal data, construct a classroom semantic map and environmental change factors; S3: Using student visual behavior data, voice interaction data, answer data, and experimental operation data as inputs, multi-dimensional modeling of student status is performed to obtain a student status matrix; S4: Based on classroom semantic map, environmental change factors, student state matrix, teaching task progress and robot's own state as input, construct a unified state representation that can be processed by reinforcement learning model; S5: Based on the unified state representation that the reinforcement learning model can handle, a hierarchical deep reinforcement learning model is used to generate robot teaching adaptation strategies and obtain candidate actions; S6: Taking candidate actions, classroom semantic map, robot status and student position as input, the robot's actions are checked for safety and teaching suitability. The robot's final actions are then sent to the robot motion control module, voice interaction module and teaching content display module.

[0007] Furthermore, classroom multimodal environment data is collected and preprocessed to obtain preprocessed unified multimodal data, as detailed below: Let the multimodal environmental data collected at time t be represented as: D t ={I t ,P t A t ,L t E t}; Among them, I t Represents classroom RGB image data, P t Let A represent depth point cloud data, and L represent audio environment data. t E represents the robot's own positioning data. t This indicates the status data of classroom equipment; The data collected by different sensors are synchronized in time, transformed in coordinates and filtered for noise, and all types of data are mapped to the global coordinate system of the classroom. Let the data collected by the i-th type of sensor at time t be... Its corresponding timestamp is Using robot master timestamp Based on this, after time synchronization, the following is obtained: ; For spatial data from different coordinate systems, coordinate transformation is performed using an extrinsic parameter matrix: ; in, This represents a spatial point in the coordinate system of the i-th type of sensor. This represents the coordinate transformation matrix from the i-th type of sensor coordinate system to the classroom global coordinate system G. This represents the transformed global coordinate point.

[0008] The preprocessed unified multimodal data is represented as follows: ; in, These represent the preprocessed classroom RGB image data, depth point cloud data, audio environment data, robot self-localization data, and classroom equipment status data, respectively.

[0009] Furthermore, based on the preprocessed unified multimodal data, a classroom semantic map and environmental change factors are constructed, as follows: Based on preprocessed image data, point cloud data, and device status data, spatial objects in the classroom are detected and identified to obtain a set of classroom objects. Let the set of classroom objects identified at time t be: ; The j-th classroom object is represented as: ; in, Indicates the object category, Represents the bounding box or spatial enclosure of an object. This indicates the position of the object in the global coordinate system of the classroom. Represents object attribute information The target detection results in the image are represented as follows:

[0010] Among them, f det (·) represents the object detection model. Indicates the detection confidence level; By combining depth data, the three-dimensional position of the target is estimated: ; Among them, Π -1 (·) represents the function that backprojects the 2D bounding box and depth data into 3D space, where K represents the camera intrinsic matrix. This represents the transformation matrix from the camera coordinate system to the classroom global coordinate system. The classroom space is divided into grids, and a semantic map of the classroom is constructed based on target recognition results, point cloud obstacle information, and device status information; let the classroom space be divided into H×W grids, and the semantic map of the classroom at time t be represented as M. t For the (u,v)th raster in the map, its semantic feature vector is represented as: ; in, This represents the probability that the route is passable. This indicates the probability of an obstacle being occupied. Represents the student distribution probability. This represents the probability distribution of teachers. This represents the probability distribution of teaching equipment. Indicates noise intensity; Indicates the safety distance constraint value; Based on the construction of a classroom semantic map, environmental change factors are extracted according to the differences in the classroom environment at continuous time points to characterize the degree of dynamic change in the classroom. These environmental change factors include obstacle change factors, student position change factors, noise change factors, equipment status change factors, and passable area change factors. The environmental change factor at time t is expressed as: ; in, Indicates the obstacle change factor. This represents the factor indicating the change in student position. Indicates the noise variation factor. Indicates the equipment status change factor. Indicates the change factor of the passable area; The obstacle change factor can be calculated from the difference in obstacle channels at adjacent time points:

[0011] The final output is a classroom semantic map M. t and environmental change factor C t .

[0012] Furthermore, using student visual behavior data, voice interaction data, answer data, and experimental operation data as inputs, a multi-dimensional model of student states is constructed to obtain a student state matrix, as follows: The inputs include student visual behavior data, voice interaction data, answer data, and experimental operation data. Visual behavior data includes student facial images, head posture, gaze direction, body movements, and classroom behavior information; voice interaction data includes student speech audio, voice activity, voice content, voice emotion, and interaction rounds; answer data includes student answer results, answer time, question knowledge points, error types, and historical answer records; experimental operation data includes student operation steps, experimental equipment usage status, operation sequence, operation duration, and operation standardization information. In the multi-dimensional modeling process, students are analyzed from the aspects of attention state, emotional state, interaction state, knowledge mastery state, and experimental operation state. For visual behavior data, analysis is performed based on student head orientation, gaze direction, facial expressions, and body movements. The system determines whether students are looking at the robot, the teaching screen, or experimental equipment based on their posture, and identifies their classroom behaviors such as focus, confusion, fatigue, raising their hands, looking down, and leaving their seats. For voice interaction data, it uses voice activity detection, sound source localization, speech recognition, and semantic analysis to determine whether students actively speak, whether their speech is relevant to the teaching topic, whether their tone is positive, and whether they ask for help or have questions. For answer data, it estimates students' mastery of different knowledge points based on answer accuracy, reaction time, knowledge point correlation, and historical answer trends. For experimental operation data, it judges students' experimental progress, operational standardization, and whether there are any errors based on the matching relationship between students' current operation steps and standard experimental procedures. The system integrates the state features of each student across various dimensions to generate a corresponding student state vector, and arranges the state vectors of all students in the classroom according to their student numbers to form a student state matrix. Each row in this student state matrix corresponds to one student, and each column corresponds to a state feature or a fused implicit feature.

[0013] Furthermore, based on the classroom semantic map, environmental change factors, student state matrix, teaching task progress, and the robot's own state as input, a unified state representation that the reinforcement learning model can process is constructed, as follows: Based on the acquired classroom semantic map M... t and environmental change factor C t And the student state matrix S output in step S2 t Simultaneously collect the current teaching task progress P t and the robot's own state R t Encoding, normalizing, and unifying the dimensions of different types of data are performed to eliminate scale and structural differences between multi-source data; for classroom semantic maps M t Spatial semantic features are extracted using convolutional neural networks or raster feature encoding networks to obtain map encoding vectors. For environmental change factor C t A multilayer perceptron is used for nonlinear mapping to obtain dynamic environmental features. For the student state matrix S t Attention pooling is used to extract group learning state features. Regarding the teaching task progress P t The teaching stage number, the percentage of knowledge points completed, and the experimental progress information are encoded as task features. Regarding the robot's own state R t Then, the robot's state features are obtained through normalization and feature mapping. Through the above processing, various heterogeneous inputs are converted into state sub-vectors in the same feature space; By fusing map encoding vectors, environmental dynamic features, student group state features, teaching task progress features, and robot state features, a unified state representation X that can be processed by reinforcement learning models is formed. t This unified state is represented as: ; Among them, F state (·) represents the state coding and fusion function.

[0014] Furthermore, the hierarchical deep reinforcement learning model comprises two layers: a high-level policy network and a low-level policy network; the high-level policy network is based on a unified state representation X. t Make macro-level teaching objective decisions and generate sub-objectives g t The low-level policy network uses sub-objective g t and current state X t As input, generate specific robot candidate actions a t The overall expression of a layered architecture is: ; Where, π H (X t ;θ H ) represents the high-level policy network, with parameter θ. H ;π L (X t ;θ L ) represents the low-level policy network, with parameter θ. L g t a represents the sub-target vector generated by the high-level strategy; t This represents the candidate actions generated by the low-level strategy.

[0015] Furthermore, the training of the hierarchical deep reinforcement learning model is as follows: The high-level policy network is trained using the Double DQN method, which combines experience replay and target network mechanisms. The loss function is:

[0016] The target value is:

[0017] in, γ is the target network parameter. H As a discount factor, It serves as a buffer for replaying high-level experiences.

[0018] The low-level policy network is trained using the proximal policy optimization method, with the objective function being: ; The importance sampling ratio is:

[0019] The generalized advantage is estimated as follows: Where, λ L For GAE parameters, γ L For lower-level discount factors, is the PPO cutoff coefficient.

[0020] Furthermore, using candidate actions, classroom semantic map, robot status, and student positions as input, the robot's actions are checked for safety and pedagogical suitability to obtain the final actions that the robot is allowed to execute. These actions are then sent to the robot motion control module, voice interaction module, and teaching content display module, as detailed below: With candidate action a t Classroom Semantic Map M t Robot's current state R t and student location set As input, safety and pedagogical suitability checks are performed on the robot's motion behavior, voice interaction behavior, and teaching content display behavior, respectively; candidate actions are represented as... ,in Indicates the robot's movement actions. Indicates voice interaction actions, This indicates the actions used to demonstrate the teaching content; According to the classroom semantic map M t Obstacle courses, passable areas, safe distance routes, and student location clusters are included. The system determines whether candidate motion actions will cause the robot to enter an obstacle area, a non-accessible area, or a student's safe distance. For candidate robot motion trajectories, the system predicts the short-term motion trajectory based on the robot's current pose, speed, and candidate motion commands, and performs collision detection between the predicted trajectory and obstacles, tables and chairs, experimental equipment, student positions, and teacher positions in the classroom semantic map. When the predicted trajectory overlaps with an obstacle area, or the distance between the robot and any student is less than the preset safe distance, the system determines that the candidate motion action does not meet the safety requirements and replaces it with actions such as stopping, decelerating, detouring, or maintaining the current position. Based on the robot's current state R t Perform feasibility checks on candidate actions; determine whether candidate voice interaction actions and teaching content display actions are suitable for the current teaching scenario based on the current teaching task stage, student location distribution, student learning status, and the location of teaching resources in the classroom semantic map; and base decisions on the student location set. With Classroom Semantic Map M t Determine whether the interaction target is clear and whether the robot has an appropriate interaction distance and orientation; The actions that pass the inspection are determined as the actions that the robot is ultimately allowed to execute. , which is represented as F check (·) represents the safety and pedagogical suitability check function.

[0021] The present invention has the following beneficial effects: 1. This invention integrates classroom multimodal environment data, student status data, teaching task progress and robot's own status into a comprehensive model, enabling STEAM teaching robots to obtain a more complete classroom environment perception capability. It can simultaneously perceive classroom space layout, student distribution, obstacle changes, noise changes, teaching equipment status and student learning behavior, thereby improving the robot's understanding of dynamic classroom environment and the accuracy of state representation. 2. This invention generates robot teaching adaptation strategies based on a hierarchical deep reinforcement learning model. It decomposes the complex teaching adaptation process into high-level teaching sub-goal decisions and low-level specific action execution decisions. High-level strategies determine the current teaching guidance direction, such as explanation and progression, interactive questioning, roving guidance, experimental instruction, or attention awakening. Low-level strategies generate corresponding movement, voice interaction, and content display actions. Therefore, the robot can dynamically adjust its teaching behavior based on the student state matrix, classroom semantic map, and teaching task progress, improving the flexibility, continuity, and personalization of teaching strategies. 3. This invention introduces a safety and teaching suitability check mechanism before the candidate action is executed, which can prevent the robot from colliding with the classroom, getting too close, obstructing the teaching line of sight, or performing actions that are not suitable for the current teaching stage. By comprehensively constraining and judging the candidate action, classroom semantic map, robot status, and student position, it not only improves the safety and reliability of robot motion execution, but also ensures that voice interaction, teaching content display, and experimental guidance behavior match the current classroom context, thereby improving the classroom adaptability and teaching application effect of STEAM teaching robots. Attached Figure Description

[0022] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0023] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments: refer to Figure 1 In this embodiment, the STEAM teaching robot environmental adaptation method based on deep reinforcement learning includes the following steps: S1: Collect and preprocess classroom multimodal environment data to obtain preprocessed unified multimodal data; S2: Based on the preprocessed unified multimodal data, construct a classroom semantic map and environmental change factors; S3: Using student visual behavior data, voice interaction data, answer data, and experimental operation data as inputs, multi-dimensional modeling of student status is performed to obtain a student status matrix; S4: Based on classroom semantic map, environmental change factors, student state matrix, teaching task progress and robot's own state as input, construct a unified state representation that can be processed by reinforcement learning model; S5: Based on the unified state representation that the reinforcement learning model can handle, a hierarchical deep reinforcement learning model is used to generate robot teaching adaptation strategies and obtain candidate actions; S6: Taking candidate actions, classroom semantic map, robot status and student position as input, the robot's actions are checked for safety and teaching suitability. The robot's final actions are then sent to the robot motion control module, voice interaction module and teaching content display module.

[0024] In this embodiment, classroom multimodal environment data is collected and preprocessed to obtain preprocessed unified multimodal data, as detailed below: The collected data includes classroom image data, depth point cloud data, sound intensity data, noise distribution data, obstacle location data, student and teacher location data, desk, chair and experimental equipment location data, and display screen or experimental equipment operation status data. Let the multimodal environmental data collected at time t be represented as: D t ={I t ,P t A t ,L t E t}; Among them, I t Represents classroom RGB image data, P t Let A represent depth point cloud data, and L represent audio environment data. t E represents the robot's own positioning data. t This indicates the status data of classroom equipment; The data collected by different sensors are synchronized in time, transformed in coordinates and filtered for noise, and all types of data are mapped to the global coordinate system of the classroom. Let the data collected by the i-th type of sensor at time t be... Its corresponding timestamp is Using robot master timestamp Based on this, after time synchronization, the following is obtained: ; For spatial data from different coordinate systems, coordinate transformation is performed using an extrinsic parameter matrix: ; in, This represents a spatial point in the coordinate system of the i-th type of sensor. This represents the coordinate transformation matrix from the i-th type of sensor coordinate system to the classroom global coordinate system G. This represents the transformed global coordinate point.

[0025] The preprocessed unified multimodal data is represented as follows: ; in, These represent the preprocessed classroom RGB image data, depth point cloud data, audio environment data, robot self-localization data, and classroom equipment status data, respectively.

[0026] In this embodiment, a classroom semantic map and environmental change factors are constructed based on the preprocessed unified multimodal data, as detailed below: Based on preprocessed image data, point cloud data, and device status data, spatial objects in the classroom are detected and identified to obtain a set of classroom objects. Classroom objects include students, teachers, desks and chairs, experimental tables, robots, displays, experimental equipment, obstacles, and passable areas. Let the set of classroom objects identified at time t be: ; The j-th classroom object is represented as: ; in, Indicates the object category, Represents the bounding box or spatial enclosure of an object. This indicates the position of the object in the global coordinate system of the classroom. This indicates object attribute information, such as whether it is movable, whether it is an obstacle, whether it is teaching equipment, whether it is a student object, etc. The target detection results in the image are represented as follows:

[0027] Among them, f det (·) represents the object detection model. Indicates the detection confidence level; By combining depth data, the three-dimensional position of the target is estimated: ; Among them, Π -1 (·) represents the function that backprojects the 2D bounding box and depth data into 3D space, where K represents the camera intrinsic matrix. This represents the transformation matrix from the camera coordinate system to the classroom global coordinate system. The classroom space is divided into grids, and a semantic map of the classroom is constructed based on target recognition results, point cloud obstacle information, and device status information; let the classroom space be divided into H×W grids, and the semantic map of the classroom at time t be represented as M. t Each semantic channel includes a passable area channel, an obstacle channel, a student location channel, a teacher location channel, a teaching equipment channel, an experimental equipment channel, a noise intensity channel, and a safe distance channel, etc. For the (u,v)th raster in the map, its semantic feature vector is represented as: ; in, This represents the probability that the route is passable. This indicates the probability of an obstacle being occupied. Represents the student distribution probability. This represents the probability distribution of teachers. This represents the probability distribution of teaching equipment. Indicates noise intensity; Indicates the safety distance constraint value; The probability of obstacle occupancy is calculated using a Bayesian update method:

[0028] Among them, z t This indicates the current sensor observation results. This indicates an event where a grid cell (u,v) is occupied by an obstacle. The student location distribution channel is constructed using a Gaussian kernel:

[0029] Among them, S t Let q represent the set of student objects. u,v This represents the spatial coordinates corresponding to the raster (u,v). Let σ represent the position of the j-th student. s Parameters representing the range of influence of student location; The noise intensity channel is represented as follows: ; in, Let w represent the audio signal collected by the r-th microphone, RMS(·) represent the root mean square sound intensity, and w represent the signal strength. r (u,v) represents the spatial weight of the r-th microphone relative to the grid (u,v); Based on the construction of a classroom semantic map, environmental change factors are extracted according to the differences in the classroom environment at continuous time points to characterize the degree of dynamic change in the classroom. These environmental change factors include obstacle change factors, student position change factors, noise change factors, equipment status change factors, and passable area change factors. The environmental change factor at time t is expressed as: ; in, Indicates the obstacle change factor. This represents the factor indicating the change in student position. Indicates the noise variation factor. Indicates the equipment status change factor. Indicates the change factor of the passable area; The obstacle change factor can be calculated from the difference in obstacle channels at adjacent time points:

[0030] The student location change factor is expressed as: ; Where, N t This represents the number of students tracked at any given time. and These represent the positions of the j-th student at the current time and the previous time, respectively. The noise variation factor is expressed as: ; in,

[0031] The equipment status change factor is expressed as: ; Where D represents the number of classroom devices, This represents the state of the d-th classroom device at time t, and I(·) represents the indicator function, which takes the value 1 when the condition is true and 0 otherwise; The change factor for traversable areas is expressed as: ; The final output is a classroom semantic map M. t and environmental change factor C t .

[0032] In this embodiment, student visual behavior data, voice interaction data, answer data, and experimental operation data are used as inputs to perform multi-dimensional modeling of student states and obtain a student state matrix. Specifically, the inputs include: visual behavior data (student facial images, head posture, gaze direction, body movements, and classroom behavior information); voice interaction data (student speaking audio, voice activity, voice content, voice emotion, and interaction rounds); answer data (student answer results, answer time, question knowledge points, error type, and historical answer records); and experimental operation data (student operation steps, experimental equipment usage status, operation sequence, operation duration, and operation standardization information). During the multi-dimensional modeling process, students are analyzed from the aspects of attention state, emotional state, interaction state, knowledge mastery state, and experimental operation state. For visual behavior data, based on the student's head orientation, gaze direction, facial expression, and body posture, it is determined whether the student is looking at the robot, teaching screen, or experimental equipment, and their focused, confused, fatigued, raised hand, lowered head, or seated classroom behaviors are identified. For voice interaction data, the system uses voice activity detection, sound source localization, speech recognition, and semantic analysis to determine whether students actively speak, whether their speech is relevant to the teaching topic, whether their tone is positive, and whether they ask for help or have questions. For answer data, the system estimates students' mastery of different knowledge points based on answer accuracy, reaction time, knowledge point correlation, and historical answer trends. For experimental operation data, the system judges students' experimental progress, operational standardization, and whether there are any errors based on the matching relationship between students' current operation steps and standard experimental procedures. The system can make a comprehensive judgment on students' learning status from multiple dimensions such as "whether they are focused," "whether they understand," "whether they participate," "whether they master," and "whether they operate correctly." The system integrates the state characteristics of each student in each dimension to generate a corresponding student state vector, and arranges the state vectors of all students in the classroom according to their student numbers to form a student state matrix. Each row in this student state matrix corresponds to one student, and each column corresponds to a state feature or a fused implicit feature, such as attention score, emotion category, voice activity, willingness to interact, probability of knowledge mastery, experimental progress, and operational standardization score.

[0033] In this embodiment, based on the classroom semantic map, environmental change factors, student state matrix, teaching task progress, and the robot's own state as inputs, a unified state representation that the reinforcement learning model can process is constructed, as follows: Based on the acquired classroom semantic map M... t and environmental change factor C t And the student state matrix S output in step S2 t Simultaneously collect the current teaching task progress P t and the robot's own state R tAmong them, the classroom semantic map Mt is used to describe semantic information such as navigable areas, obstacles, student distribution, teacher location, teaching equipment location, experimental equipment location, noise areas, and safe distance constraints in the classroom space; environmental change factor C t Used to describe the degree of change in the classroom environment at adjacent time points in terms of obstacles, student positions, noise, equipment status, and passable areas; Student state matrix S t Used to characterize each student's attention, emotion, verbal interaction, knowledge mastery, and experimental operation status; Teaching task progress P t Used to represent the current teaching stage of the course, the completion rate of knowledge points, the completion status of experimental steps, and the remaining teaching objectives; the robot's own state R t This is used to characterize the robot's current position, orientation, speed, battery level, load status, current interaction mode, and in-process teaching actions; it encodes, normalizes, and unifies the dimensions of different types of data to eliminate scale and structural differences between multi-source data; and it is used for classroom semantic maps M. t Spatial semantic features are extracted using convolutional neural networks or raster feature encoding networks to obtain map encoding vectors. For environmental change factor C t A multilayer perceptron is used for nonlinear mapping to obtain dynamic environmental features. For the student state matrix S t Attention pooling is used to extract group learning state features. Regarding the teaching task progress P t The teaching stage number, the percentage of knowledge points completed, and the experimental progress information are encoded as task features. Regarding the robot's own state R t Then, the robot's state features are obtained through normalization and feature mapping. Through the above processing, various heterogeneous inputs are converted into state sub-vectors in the same feature space; By fusing map encoding vectors, environmental dynamic features, student group state features, teaching task progress features, and robot state features, a unified state representation X that can be processed by reinforcement learning models is formed. t This unified state is represented as: ; Among them, F state(·) represents the state encoding and fusion function. The fusion function uses weighted fusion, enabling the reinforcement learning model to simultaneously perceive key information such as "whether the classroom environment has changed," "whether students currently understand and participate," "whether the teaching task needs to be advanced or adjusted," and "whether the robot currently has the conditions to execute." The resulting unified state representation not only includes static spatial semantic information but also dynamic environmental changes, student learning status, teaching progress, and robot operation constraints. This effectively reduces the complexity of the original multimodal data and improves the accuracy, stability, and interpretability of the subsequent hierarchical deep reinforcement learning model in generating instructional adaptation strategies.

[0034] In this embodiment, the hierarchical deep reinforcement learning model comprises two layers: a high-level policy network and a low-level policy network; the high-level policy network is based on a unified state representation X. t Make macro-level teaching objective decisions and generate sub-objectives g t The low-level policy network uses sub-objective g t and current state X t As input, generate specific robot candidate actions a t The overall expression of a layered architecture is: ; Where, π H (X t ;θ H ) represents the high-level policy network, with parameter θ. H ;π L (X t ;θ L ) represents the low-level policy network, with parameter θ. L g t a represents the sub-target vector generated by the high-level strategy; t This represents the candidate actions generated by the low-level strategy.

[0035] In this embodiment, the unified state representation X obtained in step S3 is... t As input to the hierarchical model, deep features are further extracted through a shared encoding network: ; Among them, W enc and b enc h is a learnable parameter t To share hidden layer feature vectors; Based on shared encoding, a Long Short-Term Memory (LSTM) network is introduced to model the historical state sequence: ; in, c represents the timing hidden state of the LSTM output. tIndicates cell state; The high-level strategy network is responsible for determining the macro-level teaching objectives based on the current overall classroom situation, and its decision-making cycle is T. H Each time step, that is, every T H Update the sub-target once per step.

[0036] (1) Definition of sub-target space Sub-target g t From a predefined set of discrete sub-objectives The selection of sub-targets includes, but is not limited to: ={g1: Explanation and Progression, g2: Interactive Questioning, g3: Supervision and Guidance, g4: Experimental Guidance, g5: Attention Awakening, g6: Adjustment of Teaching Pace}G={g1: Explanation and Progression, g2: Interactive Questioning, g3: Supervision and Guidance, g4: Experimental Guidance, g5: Attention Awakening, g6: Adjustment of Teaching Pace}; (2) High-level strategy network structure The high-level policy network adopts a Dueling DQN structure, which estimates the state value function and action advantage function separately: ; Among them, V H (·) is the state value function, A H (·) represents the action advantage function, and both are implemented by independent fully connected networks; The sub-objectives of the high-level strategy are selected using an ε-greedy strategy: ; (3) High-level reward function The high-level reward function comprehensively reflects the achievement of teaching objectives and is defined as follows: ; in, This indicates the increment in the progress of the teaching task. This represents the average attention score of all students in the class. This indicates the average level of knowledge mastery across the entire class. The sub-objective switching penalty term is used to avoid frequent policy oscillations, and ω1, ω2, ω3, and ω4 are adjustable weight coefficients. The low-level policy network, given the sub-objective g by the high-level policy, t Guided by this, specific candidate robot actions are generated at each time step; (1) Definition of candidate action space Candidate action a t It consists of three types of motion components, each corresponding to a robot's movement. Voice interaction actions Demonstration of teaching content and actions The three types of actions together constitute the joint action space: ; in: This includes forward, backward, left turn, right turn, rotating in place, and stopping; This includes the selection of content to be explained, types of questions to ask, encouragement of feedback, guidance on error correction, and instructions to remain silent. This includes displaying knowledge graphs, playing experimental videos, showing quiz results, and displaying experimental step prompts; (2) Sub-target conditional input Subtarget g t The embedded encoding is performed and concatenated with the temporal hidden state, serving as the input to the low-level policy network: ; Among them, Embed(g t ) represents the embedding vector of the sub-target. This is for conditionalizing input features.

[0037] (3) Low-level strategy network structure The low-level policy network adopts an Actor-Critic structure to estimate the policy and value function, respectively.

[0038] The probability distribution of each action component output by the Actor network: ; The sampling method for joint candidate actions is as follows: ; The Critic network estimates the value function in the current state: ; (4) Low-level reward function The low-level reward function is used to evaluate the degree of completion of a single action with respect to the current sub-goal: ; Among them, R goal (·) indicates the contribution of the action to the achievement of the sub-goal, R interact (·) indicates the degree of matching between the voice interaction and the student's state, R nav (·) indicates the navigational rationality of the movement action in the classroom semantic map, R penalty (·) represents the action penalty (e.g., repetitive action or invalid interaction), and φ, γ1, γ2, γ3 are adjustable weight coefficients.

[0039] In this embodiment, the training of the hierarchical deep reinforcement learning model is as follows: The high-level policy network is trained using the Double DQN method, which combines experience replay and target network mechanisms. The loss function is:

[0040] The target value is:

[0041] in, γ is the target network parameter. H As a discount factor, It serves as a buffer for replaying high-level experiences.

[0042] The low-level policy network is trained using the proximal policy optimization method, with the objective function being: ; The importance sampling ratio is:

[0043] The generalized advantage is estimated as follows: Where, λ L For GAE parameters, γ L For lower-level discount factors, is the PPO cutoff coefficient.

[0044] In this embodiment, candidate actions, a classroom semantic map, robot status, and student positions are used as inputs to perform safety and pedagogical suitability checks on robot actions. The final actions permitted for robot execution are then obtained and sent to the robot motion control module, voice interaction module, and teaching content display module, as detailed below: With candidate action a t Classroom Semantic Map M t Robot's current state R t and student location set As input, safety and pedagogical suitability checks are performed on the robot's motion behavior, voice interaction behavior, and teaching content display behavior, respectively; candidate actions are represented as... ,in Indicates the robot's movement actions. Indicates voice interaction actions, This indicates the actions used to demonstrate the teaching content; According to the classroom semantic map M t Obstacle courses, passable areas, safe distance routes, and student location clusters are included. The system determines whether candidate motion actions will cause the robot to enter an obstacle area, a non-accessible area, or a student's safe distance. For candidate robot motion trajectories, the system predicts the short-term motion trajectory based on the robot's current pose, speed, and candidate motion commands, and performs collision detection on the predicted trajectory against obstacles, tables and chairs, experimental equipment, student positions, and teacher positions in the classroom semantic map. When the predicted trajectory overlaps with an obstacle area, or the distance between the robot and any student is less than the preset safe distance, the system determines that the candidate motion action does not meet the safety requirements and replaces it with actions such as stopping, decelerating, detouring, or maintaining the current position. Based on the robot's current state R t The system performs an executability check on candidate actions. The robot's current status can include battery level, current position, orientation, speed, mechanical structure status, communication status, load status, and current module occupancy status. If the robot's battery level is below a threshold, the motion control module is malfunctioning, the voice module is occupied, the display module is unavailable, or the robot's current pose does not support a particular action, the system will determine that the corresponding candidate action is unexecutable. For example, when the robot's battery is low, long-distance patrol actions are prohibited; when the robot is near an experimental platform or a densely populated area with many students, its rotation range and movement speed are limited; when the display module is faulty or occupied, candidate teaching demonstration actions are delayed or replaced with voice explanation actions. By incorporating the robot's own operating status into the action selection process, the stability and reliability of the system's execution can be improved. Based on the current teaching task stage, student location distribution, student learning status, and the location of teaching resources in the classroom semantic map, the system determines whether candidate voice interaction actions and teaching content display actions are suitable for the current teaching scenario. If the current teaching stage is a concept explanation stage, actions such as knowledge explanation, analogy explanation, and interactive questioning are given priority. If the current teaching stage is an experimental operation stage, actions such as experimental step prompts, safety precaution reminders, error correction, and supervision are given priority. For voice interaction actions, the system also considers student location and classroom noise area to determine whether the voice output volume, direction, and target student are appropriate. For teaching content display actions, the system determines whether the currently displayed content is consistent with the teaching task progress, whether it is suitable for students' current knowledge level, and whether it will cause information overload or repetitive interference. This ensures that the teaching behaviors output by the robot are not only executable but also conform to the teaching objectives and classroom pace. (Based on student location set) With Classroom Semantic Map M tThe system determines whether the interaction target is clear and whether the robot has an appropriate interaction distance and orientation. When the candidate action is "asking a student a question" or "approaching a group to provide experimental guidance," the system needs to confirm the current location of the target student or group, whether there are obstacles around, and whether the robot can reach the appropriate location without affecting other students. If the target student is too far away, in a noisy area, obstructed by obstacles, or if the robot would have to cross a non-travelable area to approach the target student, the system can adjust the candidate action to a remote voice prompt, a screen display prompt, or select a closer and safer interaction location. This approach enhances the relevance of robot-assisted teaching interaction while avoiding ineffective or abrupt classroom behaviors. When a candidate action fails the safety or pedagogical suitability check, the system can use action correction, action replacement, or action blocking mechanisms to generate the final allowed action. Action correction refers to adjusting action parameters while retaining the original teaching intent, such as reducing movement speed, shortening movement distance, changing movement direction, adjusting voice volume, or changing the display page; action replacement refers to replacing unsafe or inappropriate actions with preset safe actions or pedagogically equivalent actions, such as replacing "approaching students to explain" with "standing still to explain and display prompts"; action blocking refers to directly prohibiting the current candidate action and selecting the next action that meets the constraints from the candidate action set.

[0045] The actions that pass the inspection are determined as the actions that the robot is ultimately allowed to execute. , which is represented as F check The parentheses (·) represent the safety and instructional suitability check function. If the final permitted action is a compound action, the system breaks it down according to the action type and sends it separately to the robot motion control module, the voice interaction module, and the instructional content display module. The motion control module is responsible for executing robot movement, turning, stopping, obstacle avoidance, and pose adjustment; the voice interaction module is responsible for explanation, questioning, feedback, reminders, and error correction; and the instructional content display module is responsible for courseware switching, experimental step prompts, knowledge graph display, quiz result feedback, and multimedia content playback.

[0046] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0047] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0048] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0049] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0050] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any other way. Any person skilled in the art may make changes or modifications to the disclosed technical content to create equivalent embodiments. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A method for environmental adaptation of STEAM teaching robots based on deep reinforcement learning, characterized in that, Includes the following steps: S1: Collect and preprocess classroom multimodal environment data to obtain preprocessed unified multimodal data; S2: Based on the preprocessed unified multimodal data, construct a classroom semantic map and environmental change factors; S3: Using student visual behavior data, voice interaction data, answer data, and experimental operation data as inputs, multi-dimensional modeling of student status is performed to obtain a student status matrix; S4: Based on classroom semantic map, environmental change factors, student state matrix, teaching task progress and robot's own state as input, construct a unified state representation that can be processed by reinforcement learning model; S5: Based on the unified state representation that the reinforcement learning model can handle, a hierarchical deep reinforcement learning model is used to generate robot teaching adaptation strategies and obtain candidate actions; S6: Taking candidate actions, classroom semantic map, robot status and student position as input, the robot's actions are checked for safety and teaching suitability. The robot's final actions are then sent to the robot motion control module, voice interaction module and teaching content display module.

2. The STEAM teaching robot environmental adaptation method based on deep reinforcement learning according to claim 1, characterized in that, The process of collecting and preprocessing classroom multimodal environment data to obtain preprocessed unified multimodal data is as follows: Let the multimodal environmental data collected at time t be represented as: D t ={I t ,P t ,A t ,L t ,E t }; Among them, I t Represents classroom RGB image data, P t Let A represent depth point cloud data, and L represent audio environment data. t E represents the robot's own positioning data. t This indicates the status data of classroom equipment; The data collected by different sensors are synchronized in time, transformed in coordinates and filtered for noise, and all types of data are mapped to the global coordinate system of the classroom. Let the data collected by the i-th type of sensor at time t be... Its corresponding timestamp is Using robot master timestamp Based on this, after time synchronization, the following is obtained: ； For spatial data from different coordinate systems, coordinate transformation is performed using an extrinsic parameter matrix: ； in, This represents a spatial point in the coordinate system of the i-th type of sensor. This represents the coordinate transformation matrix from the i-th type of sensor coordinate system to the classroom global coordinate system G. This represents the transformed global coordinate point; The preprocessed unified multimodal data is represented as follows: ； in, These represent the preprocessed classroom RGB image data, depth point cloud data, audio environment data, robot self-localization data, and classroom equipment status data, respectively.

3. The STEAM teaching robot environmental adaptation method based on deep reinforcement learning according to claim 2, characterized in that, Based on the preprocessed unified multimodal data, a classroom semantic map and environmental change factors are constructed as follows: Based on preprocessed image data, point cloud data, and device status data, spatial objects in the classroom are detected and identified to obtain a set of classroom objects. Let the set of classroom objects identified at time t be: ; The j-th classroom object is represented as: ; in, Indicates the object category, Represents the bounding box or spatial enclosure of an object. This indicates the position of the object in the global coordinate system of the classroom. Represents object attribute information The target detection results in the image are represented as follows: ； Among them, f det (·) represents the object detection model. Indicates the detection confidence level; By combining depth data, the three-dimensional position of the target is estimated: ; Among them, Π -1 (·) represents the function that backprojects the 2D bounding box and depth data into 3D space, where K represents the camera intrinsic matrix. This represents the transformation matrix from the camera coordinate system to the classroom global coordinate system. The classroom space is divided into grids, and a semantic map of the classroom is constructed based on target recognition results, point cloud obstacle information, and device status information; let the classroom space be divided into H×W grids, and the semantic map of the classroom at time t be represented as M. t For the (u,v)th raster in the map, its semantic feature vector is represented as: ; in, This represents the probability that the route is passable. This indicates the probability of an obstacle being occupied. Represents the student distribution probability. This represents the probability distribution of teachers. This represents the probability distribution of teaching equipment. Indicates noise intensity; Indicates the safety distance constraint value; Based on the construction of a classroom semantic map, environmental change factors are extracted according to the differences in the classroom environment at continuous time points to characterize the degree of dynamic change in the classroom. These environmental change factors include obstacle change factors, student position change factors, noise change factors, equipment status change factors, and passable area change factors. The environmental change factor at time t is expressed as: ; in, Indicates the obstacle change factor. This represents the factor indicating the change in student position. Indicates the noise variation factor. Indicates the equipment status change factor. Indicates the change factor of the passable area; The obstacle change factor can be calculated from the difference in obstacle channels at adjacent time points: ； The final output is a classroom semantic map M. t and environmental change factor C t .

4. The STEAM teaching robot environmental adaptation method based on deep reinforcement learning according to claim 1, characterized in that, The method uses student visual behavior data, voice interaction data, answer data, and experimental operation data as inputs to perform multi-dimensional modeling of student states and obtain a student state matrix. Specifically, the inputs include: visual behavior data (student facial images, head posture, gaze direction, body movements, and classroom behavior information); voice interaction data (student speech audio, voice activity, voice content, voice emotion, and interaction rounds); answer data (student answer results, answer time, question knowledge points, error types, and historical answer records); and experimental operation data (student operation steps, experimental equipment usage status, operation sequence, operation duration, and operation standardization information). In the multi-dimensional modeling process, students are analyzed from the aspects of attention state, emotional state, interaction state, knowledge mastery state, and experimental operation state. For visual behavior data, analysis is performed based on student head orientation, gaze direction, facial expression, and body posture. The system determines whether students are looking at robots, teaching screens, or experimental equipment, and identifies their classroom behaviors such as focus, confusion, fatigue, raising hands, looking down, and leaving their seats. For voice interaction data, it uses voice activity detection, sound source localization, speech recognition, and semantic analysis to determine whether students actively speak, whether their speech is relevant to the teaching topic, whether their tone is positive, and whether they ask for help or have questions. For answer data, it estimates students' mastery of different knowledge points based on answer accuracy, reaction time, knowledge point correlation, and historical answer trends. For experimental operation data, it judges students' experimental progress, operational standardization, and whether there are any errors based on the matching relationship between students' current operation steps and standard experimental procedures. The system integrates the state characteristics of each student across various dimensions to generate a corresponding student state vector, and arranges the state vectors of all students in the classroom according to their student numbers to form a student state matrix. Each row in this student state matrix corresponds to one student, and each column corresponds to a state feature or a fused implicit feature.

5. The STEAM teaching robot environmental adaptation method based on deep reinforcement learning according to claim 1, characterized in that, The unified state representation that the reinforcement learning model can process is constructed based on the classroom semantic map, environmental change factors, student state matrix, teaching task progress, and the robot's own state as inputs, as follows: Based on the acquired classroom semantic map M... t and environmental change factor C t And the student state matrix S output in step S2 t Simultaneously collect the current teaching task progress P t and the robot's own state R t Encoding, normalizing, and unifying the dimensions of different types of data are performed to eliminate scale and structural differences between multi-source data; for classroom semantic maps M t Spatial semantic features are extracted using convolutional neural networks or raster feature encoding networks to obtain map encoding vectors. For environmental change factor C t A multilayer perceptron is used for nonlinear mapping to obtain dynamic environmental features. For the student state matrix S t Attention pooling is used to extract group learning state features. Regarding the teaching task progress P t The teaching stage number, the percentage of knowledge points completed, and the experimental progress information are encoded as task features. Regarding the robot's own state R t Then, the robot's state features are obtained through normalization and feature mapping. Through the above processing, various heterogeneous inputs are converted into state sub-vectors in the same feature space; By fusing map encoding vectors, environmental dynamic features, student group state features, teaching task progress features, and robot state features, a unified state representation X that can be processed by reinforcement learning models is formed. t This unified state is represented as: ； Among them, F state (·) represents the state coding and fusion function.

6. The STEAM teaching robot environmental adaptation method based on deep reinforcement learning according to claim 5, characterized in that, The hierarchical deep reinforcement learning model comprises two layers: a high-level policy network and a low-level policy network; the high-level policy network is based on a unified state representation X. t Make macro-level teaching objective decisions and generate sub-objectives g t The low-level policy network uses sub-objective g t and current state X t As input, generate specific robot candidate actions a t The overall expression of a layered architecture is: ； Where, π H (X t ;θ H ) represents the high-level policy network, with parameter θ. H ;π L (X t ;θ L ) represents the low-level policy network, with parameter θ. L g t a represents the sub-target vector generated by the high-level strategy; t This represents the candidate actions generated by the low-level strategy.

7. The STEAM teaching robot environmental adaptation method based on deep reinforcement learning according to claim 6, characterized in that, The training of the hierarchical deep reinforcement learning model is as follows: The high-level policy network is trained using the Double DQN method, which combines experience replay and target network mechanisms. The loss function is: ； The target value is: ； in, γ is the target network parameter. H As a discount factor, This serves as a buffer for replaying high-level experiences. The low-level policy network is trained using the proximal policy optimization method, with the objective function being: ； The importance sampling ratio is: ； The generalized advantage is estimated as follows: Where, λ L For GAE parameters, γ L For lower-level discount factors, is the PPO cutoff coefficient.

8. The STEAM teaching robot environmental adaptation method based on deep reinforcement learning according to claim 1, characterized in that, The system takes candidate actions, a classroom semantic map, robot status, and student positions as inputs to perform safety and pedagogical suitability checks on robot actions, obtaining the final actions that the robot is allowed to execute. These actions are then sent to the robot motion control module, voice interaction module, and teaching content display module, as detailed below: With candidate action a t Classroom Semantic Map M t Robot's current state R t and student location set As input, safety and pedagogical suitability checks are performed on the robot's motion behavior, voice interaction behavior, and teaching content display behavior, respectively; candidate actions are represented as... ,in Indicates the robot's movement actions. Indicates voice interaction actions, This indicates the actions used to demonstrate the teaching content; According to the classroom semantic map M t Obstacle courses, passable areas, safe distance routes, and student location clusters are included. The system determines whether candidate motion actions will cause the robot to enter an obstacle area, a non-accessible area, or a student's safe distance. For candidate robot motion trajectories, the system predicts the short-term motion trajectory based on the robot's current pose, speed, and candidate motion commands, and performs collision detection between the predicted trajectory and obstacles, tables and chairs, experimental equipment, student positions, and teacher positions in the classroom semantic map. When the predicted trajectory overlaps with an obstacle area, or the distance between the robot and any student is less than the preset safe distance, the system determines that the candidate motion action does not meet the safety requirements and replaces it with actions such as stopping, decelerating, detouring, or maintaining the current position. Based on the robot's current state R t Perform feasibility checks on candidate actions; determine whether candidate voice interaction actions and teaching content display actions are suitable for the current teaching scenario based on the current teaching task stage, student location distribution, student learning status, and the location of teaching resources in the classroom semantic map; and base decisions on the student location set. With Classroom Semantic Map M t Determine whether the interaction target is clear and whether the robot has an appropriate interaction distance and orientation; The actions that pass the inspection are determined as the actions that the robot is ultimately allowed to execute. , which is represented as F check (·) represents the safety and pedagogical suitability check function.

Images