A reinforcement learning supported collaborative learning intervention method and system

Abstract

The application provides a collaborative learning intervention method and system under reinforcement learning, comprising the following steps: selecting an intervention action from an intervention action space based on a perceived state of a collaborative discussion environment; obtaining a reward and a next moment state fed back by the collaborative discussion environment after executing the intervention action, and then obtaining transition data; the state comprises collaborative sessions of all learners in the collaborative discussion environment; repeating the above steps until a preset number of transition data are obtained; training a Q network based on the preset number of transition data; in the training process, an improved Q learning method is used to update a Q value function; using the trained Q network to determine a current intervention action corresponding to a maximum Q value in a current state; and providing an intervention suggestion to the learner based on the current intervention action. The application can provide learning support for learners who do not fully master the meta-cognitive strategies of collaborative learning, and promote the collaborative activities among learners and the development of individual meta-cognitive ability.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent education technology, and more specifically, relates to a collaborative learning intervention method and system supported by reinforcement learning. Background Art

[0002] In reading, social collaborative annotation is a specific form of collaborative learning. It requires students to work in groups to read, annotate, and exchange ideas on the same material, thereby promoting a deep understanding of the relevant knowledge and the development of collaborative communication skills. Collaborative learning itself also requires support. When it comes to tracking each student's learning information and learning needs, adding technical support can reduce the burden on teachers.

[0003] Collaborative learning intervention methods and systems currently focus primarily on helping students reach consensus and advance collaborative tasks, but lack support for metacognitive processes throughout the entire learning process. Metacognition is crucial for maintaining learning motivation, coordinating collaborative tasks, developing learning strategies, and promoting knowledge construction, yet research and design efforts in this area are limited. Therefore, providing metacognitive interventions within the collaborative learning process to promote collaborative activities and individual metacognitive development has become a pressing technical challenge in the field of intelligent educational technology. Summary of the Invention

[0004] In view of the shortcomings of the existing technology, the purpose of the present invention is to provide a collaborative learning intervention method and system supported by reinforcement learning, aiming to solve the problem of how to provide learning intervention for learners at the metacognitive level of the collaborative learning process.

[0005] To achieve the above objectives, in a first aspect, the present invention provides a collaborative learning intervention method supported by reinforcement learning, comprising:

[0006] S101 is based on the perceived state of the collaborative discussion environment t Select an intervention action a from the intervention action space t , when performing intervention action a t Afterwards, the reward r is given by the collaborative discussion environment t and the state s at the next moment t+1 , and then obtain the transfer data (s t ,a t ,r t ,s t+1 ); the state s t A collaborative session including all learners in the collaborative discussion environment;

[0007] S102 repeats step S101 until a preset amount of transfer data is obtained;

[0008] S103 trains the Q network based on the preset amount of transfer data, and during the training process, uses an improved Q learning method to update the Q value function;

[0009] S104 uses the trained Q network to determine the current intervention action corresponding to the maximum Q value in the current state from the intervention action space;

[0010] S105 provides intervention suggestions to the learner based on the current intervention action.

[0011] In an optional example, the improved Q-learning method specifically updates the Q-value function based on the following formula:

[0012] Q(s t ,a t )←Q(s t ,a t )+α·(R t +γ·max(Q(s t+1 ,·))-Q(s t ,a t ))

[0013] Where, Q(s t ,a t ) is the Q value of step t, max(Q(s t+1 ,·)) is the state s t+1 The maximum Q value under s t is the state at step t, a t is the intervention action at step t, s t+1 is the state at step t+1, R t is the reward at step t, α is the learning rate, and γ is the discount rate.

[0014] In an optional example, the intervention action space includes: individual metacognition suggestions, group progress coordination suggestions, group communication suggestions at the beginning of the task, group communication suggestions throughout the learning process, group knowledge integration suggestions at the example stage, and group knowledge integration suggestions at the argumentation stage.

[0015] In an optional example, S105 specifically includes:

[0016] Based on the current intervention action, randomly select a corresponding gain frame or loss frame;

[0017] The selected gain or loss frames are presented in a collaborative discussion environment to provide intervention suggestions to learners.

[0018] In an optional example, before S104, the following steps are further included:

[0019] Based on the historical intervention actions performed and the historical collaboration sessions of all learners, it is determined whether to intervene in the learner.

[0020] In a second aspect, the present invention provides a collaborative learning intervention system supported by reinforcement learning, comprising:

[0021] Data acquisition module, used to perceive the state of the collaborative discussion environment t Select an intervention action a from the intervention action space t , when performing intervention action a t Afterwards, the reward r is given by the collaborative discussion environment t and the state s at the next moment t+1 , and then obtain the transfer data (s t, a t ,r t ,s t+1 ); the state s t A collaborative session including all learners in the collaborative discussion environment;

[0022] A data accumulation module, configured to repeatedly call the data accumulation module until a preset amount of transfer data is obtained;

[0023] A network training module, configured to train the Q network based on the preset amount of transfer data, and during the training process, update the Q value function using an improved Q learning method;

[0024] an action decision module, configured to determine, from the intervention action space, a current intervention action corresponding to the maximum Q value in the current state using the trained Q network;

[0025] The intervention suggestion module is used to make intervention suggestions to the learner based on the current intervention action.

[0026] In an optional example, the improved Q-learning method in the network training module specifically updates the Q-value function based on the following formula:

[0027] Q(s t ,a t )←Q(s t ,a t )+α·(R t +γ·max(Q(s t+1 ,·))-Q(s t ,a t ))

[0028] Where, Q(s t ,a t ) is the Q value of step t, max(Q(s t+1 ,·)) is the state st+1 The maximum Q value under s t is the state at step t, a t is the intervention action at step t, s t+1 is the state at step t+1, R t is the reward at step t, α is the learning rate, and γ is the discount rate.

[0029] In an optional example, the intervention action space in the data acquisition module includes: personal metacognitive suggestions, group progress coordination suggestions, group communication suggestions at the beginning of the task, group communication suggestions throughout the learning process, group knowledge integration suggestions in the example stage, and group knowledge integration suggestions in the argumentation stage.

[0030] In an optional example, the intervention suggestion module is specifically configured to:

[0031] Based on the current intervention action, randomly select a corresponding gain frame or loss frame;

[0032] The selected gain or loss frames are presented in a collaborative discussion environment to provide intervention suggestions to learners.

[0033] In an optional example, an intervention judgment module is further included, configured to:

[0034] Based on the historical intervention actions performed and the historical collaboration sessions of all learners, it is determined whether to intervene in the learner.

[0035] In general, the above technical solutions conceived by the present invention have the following beneficial effects compared with the prior art:

[0036] The present invention provides a collaborative learning intervention method and system supported by reinforcement learning. By applying reinforcement learning to collaborative learning discussion scenarios, learners are provided with motivational intervention information based on previous context. The previous context includes both the intervention discourse information previously implemented by the intelligent agent and the previous collaborative conversation information of all learners. This can provide learning support for learners who do not fully master the metacognitive strategies of collaborative learning, promote collaborative activities among learners and the development of individual metacognitive abilities, and avoid duplication and loss of control of intervention, which would cause cognitive stress for learners. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a flowchart of the collaborative learning intervention method supported by reinforcement learning provided by the present invention;

[0038] Figure 2 is a flow chart of the intervention formation provided by the present invention;

[0039] Figure 3It is a technical structure diagram of the intervention timing and intervention content provided by the present invention;

[0040] Figure 4 is a schematic diagram of the reinforcement learning model provided by the present invention;

[0041] Figure 5 This is an architectural diagram of the collaborative learning intervention system supported by reinforcement learning provided by the present invention. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0043] Current collaborative annotation reading platforms generally offer features such as annotation, tagging, replying to annotations, likes, and networked analysis of opinions and interactions. They lack AI-powered analysis and automated intervention for the process of constructing opinions through real-time collaborative communication, and in particular, lack intervention focused on learners' metacognition. Few learning support services offer comprehensive metacognitive interventions to help learners master learning strategies, thereby promoting collaborative activities and individual metacognitive development.

[0044] Reinforcement learning is a general learning method in which an intelligent agent perceives state information from the environment and selects appropriate actions based on its own strategy, thereby changing the environmental state and receiving rewards or incentives, which represent the utility function of the new state for the agent. The agent's goal is to find a strategy for selecting actions that maximizes the potential reward, based on exploring the state space of the environment.

[0045] In summary, the present invention has designed a human-computer collaborative intervention mechanism based on reading and collaborative discussion contexts: the intelligent agent in the system and method, as an outsourced form of social intelligence, monitors learners' learning problems in real time and then provides metacognitive learning strategy intervention information. This information generation is based on a reinforcement learning algorithm. This reinforcement learning algorithm can accumulate learners' learning trajectory information and provide incentive responses to learners' behaviors, thereby achieving learning support and helping group members to effectively share, interact, construct, and collaborate. Here, the learning trajectory specifically refers to the learner's collaborative conversation trajectory.

[0046] Previous learning support mainly focused on cognitive intervention, lacking detection and intervention at the learner's metacognitive level. In this regard, the present invention provides a collaborative learning intervention method under reinforcement learning support, the execution subject of which is an intelligent agent. Figure 1This is a flow chart of the collaborative learning intervention method supported by reinforcement learning provided by the present invention. Figure 1 As shown, the method includes:

[0047] Step S101, based on the perceived state s of the collaborative discussion environment t Select an intervention action a from the intervention action space t , when performing intervention action a t Afterwards, the reward r is given by the collaborative discussion environment t and the state s at the next moment t+1 , and then obtain the transfer data ( t, a t ,t,t+1); state s t Include collaborative sessions with all learners in a collaborative discussion environment;

[0048] Step S102, repeating step S101 until a preset amount of transfer data is obtained;

[0049] Step S103, training the Q network based on a preset amount of transfer data, and during the training process, updating the Q value function using an improved Q learning method;

[0050] Step S104, using the trained Q network, determine the current intervention action corresponding to the maximum Q value in the current state from the intervention action space;

[0051] Step S105: Provide intervention suggestions to the learner based on the current intervention action.

[0052] Taking the collaborative reading scenario as an example, the automatic intervention achieved by the present invention is that during the task discussion stage after the learners have conducted collaborative reading, the system automatically inserts teaching intervention information in the form of annotations or message pop-ups. Specifically, in the group's online collaborative reading seminar, each speech of the learner is regarded as the minimum intervention detection unit. The system identifies the learner's collaborative discussion status, and then analyzes whether the learner has a problem and determines whether automatic intervention should be implemented. The timing of intervention is determined by the reinforcement learning algorithm. If intervention is required, appropriate intervention discourse is selected, and the available intervention discourse categories are manually formulated and discussed based on teaching theory.

[0053] The application of reinforcement learning algorithms requires defining the state of the system, the set of actions that can be implemented, and the incentive mechanism, namely<S,A,R> The state of the learner in the collaborative reading discussion activity determines the system state in the reinforcement learning, denoted as s t , the intervention discourse that the system can select is the intervention action that can be implemented in the current state of the system, denoted as a t . Accordingly, s t+1 and a t+1It is the state of the system in the next step and the intervention action that the system can implement. S and A are all s t and a t are the state space and action space respectively.

[0054] Once a certain amount of data has been accumulated, the Q-network can be trained to fit the Q-value function. During the training process, an improved Q-learning method is used to iteratively update the Q-value function. Based on this, the trained Q-network is used to determine the current intervention action corresponding to the maximum Q-value in the current state from the intervention action space. This allows for intervention recommendations to be made to learners based on the current intervention action, helping them master collaborative learning strategies, promoting collaborative activities among learners, and developing individual metacognitive abilities. The intervention action can target individual learners or all learners in a group, depending on the type of intervention action determined.

[0055] The method provided by the embodiment of the present invention provides learners with motivational intervention information based on previous context by applying reinforcement learning to collaborative learning discussion scenarios. The previous context includes both the intervention discourse information previously implemented by the intelligent agent and the previous collaborative conversation information of all learners. This can provide learning support for learners who do not fully master the metacognitive strategies of collaborative learning, promote collaborative activities among learners and the development of individual metacognitive abilities, and avoid repetition and loss of control of intervention, which would cause cognitive stress for learners.

[0056] Based on the above embodiment, the improved Q-learning method specifically updates the Q-value function based on the following formula:

[0057] Q(s t ,a t )←Q(s t ,a t )+α·(R t +γ·max(Q(s t+1 ,·))-Q(s t ,a t ))

[0058] Where, Q(s t ,a t ) is the Q value of step t, max(Q(s t+1 ,·)) is the state s t+1 The maximum Q value under s t is the state at step t, a t is the intervention action at step t, s t+1 is the state at step t+1, R t is the reward at step t, α is the learning rate, and γ is the discount rate. The value obtained by iterating the right formula is used to update the original Q value.

[0059] It should be noted that the intelligent agent not only considers the incentive index, but also needs to consider the context and the action evaluation issued by the learner (represented by their respective Q functions) to implement the corresponding intervention at step t+1. According to the situational considerations, the learner's incentive cannot be too large, otherwise the learner will get used to being encouraged, which may easily lead to negative effects, such as blind self-confidence. Therefore, it is necessary to weaken it in γ·max(Q(s t+1 ,·)) minus Q(s t ,a t ).

[0060] Furthermore, the ultimate goal of the agent is to maximize the cumulative instantaneous reward value. In order to be able to relate to the previous and next contexts, R t It is cumulative:

[0061] R t =r t +γr t+1 +γ 2 r t+1 +…+γ t-1 r t+1 ,0<γ≤1

[0062] At each step t, t = 1, 2, ..., of interacting with the environment, the agent perceives the state of the environment s t ∈S, then choose an action After executing the action, the environment gives the intelligent agent a numerical reward r t and the next environmental state.

[0063] Considering that the collaborative conversations of all learners included in the state and the intervention actions of the intelligent agent contain text information, we can first perform text semantic extraction on them, and then input the feature vector after semantic extraction into the reinforcement learning Q network for application.

[0064] The present invention performs incentive intervention based on the preceding and following contexts, and the intelligent agent not only considers the incentive value of the intelligent agent but also the preceding and following contexts to avoid possible repetitive content, thereby avoiding cognitive pressure on learners.

[0065] Based on any of the above embodiments, and considering that the collaborative reading discussion process includes different learning stages, in order to improve the targeted effect of intervention, the embodiment of the present invention combines the two factors of intervention timing and intervention strategy to design an intervention action space. The intervention action space includes: personal metacognitive suggestions, group progress coordination suggestions, group communication suggestions at the beginning of the task, group communication suggestions throughout the learning process, group knowledge integration suggestions in the example stage, and group knowledge integration suggestions in the demonstration stage. The intervention actions performed by the intelligent agent can be selected from this intervention action space.

[0066] Based on any of the above embodiments, S105 specifically includes:

[0067] Based on the current intervention action, randomly select the corresponding gain frame or loss frame;

[0068] The selected gain or loss frames are presented in a collaborative discussion environment to provide intervention suggestions to learners.

[0069] For the same intervention action, the present invention introduces two corresponding frameworks. One is a framework that emphasizes the benefits of engaging in a certain behavior, namely the benefit framework, and the other is a framework that emphasizes the consequences of not engaging in a certain behavior, namely the loss framework. For example, the current intervention action is personal metacognitive suggestions, and the corresponding benefit framework can be "If you have questions, you can speak with your groupmates and ask questions. This may be one of the achievements of your group." The loss framework can be "If you have questions and do not speak with your groupmates and ask questions, you may miss one of the group's achievements."

[0070] Based on any of the foregoing embodiments, before S104, the method further includes:

[0071] Based on the historical intervention actions performed and the historical collaboration sessions of all learners, it is determined whether to intervene in the learner.

[0072] Furthermore, Figure 2 It is a flow chart of the intervention provided by the present invention, such as Figure 2 As shown in the figure, learners engage in collaborative conversations under the supervision of an intelligent agent. The agent decides when to intervene and what to intervene. Both learner and agent conversation information is stored on the backend, and this information influences the agent's next intervention decision: the agent decides whether to intervene or not. If so, it selects the appropriate type of text to intervene in the learner's collaborative conversation, forming an iteration. Here, the information about the agent's interventions refers to the agent's historical intervention actions.

[0073] Based on any of the above embodiments, Figure 3 This is a technical structure diagram of the intervention timing and intervention content provided by the present invention, such as Figure 3As shown, each learner's speech is considered a minimum intervention detection unit. The system identifies the learner's collaborative discussion status, analyzes whether the learner has any problems, and determines whether automatic intervention should be implemented. Subsequently, an intervention mechanism is developed through reinforcement learning. After determining the intervention content, the corresponding conversation is selected from the intervention strategy library and presented to the collaborative discussion environment. In the collaborative learning process, the formation of the intelligent agent's intervention mechanism is closely related to the learning environment. The intervention timing strategy, as the behavioral content, influences the collaborative discussion environment. The state and reward in the discussion environment can serve as inputs to the timing strategy, allowing the intervention mechanism to change dynamically, thus forming a loop in the intervention process. The intervention content here refers to the intervention action described above.

[0074] Figure 4 is a schematic diagram of the reinforcement learning model provided by the present invention, such as Figure 4 As shown in the figure, the text information generated by the intelligent agent's intervention text and the learner's collaborative actions is used as a training set and input into the text extraction and feedback model. The feature vectors of these contents are deconstructed to achieve semantic extraction. These feature vectors are input into the reinforcement mechanism function to select intervention actions and feed back to the model and environment for iterative intervention.

[0075] Based on any of the above embodiments, the application of the reinforcement learning algorithm requires defining the state of the system and the set of actionable actions and the incentive mechanism, that is,<S,A,R> . S and A are the collections of all s and a. The reward mechanism R comes from the learner's final learning effect, which can only be obtained after the learner undergoes a series of learning interventions. This requires a specific definition of how to use the final learning effect as a reward when applying the algorithm, and reasonably distribute it to all the experienced states and implemented actions in the process.<S,A,R> After the determination, the optimization of teaching intervention strategy in the process of learners’ collaborative reading can adopt the Q-Learning algorithm in reinforcement learning. The key of this algorithm is to learn and calculate Q-value for each system state and possible system action, which is recorded as Q( t ,t), and update it using the Bellman equation, the calculation formula is as follows:

[0076]

[0077] Where Q(s t ,a t ) is the total value of this incentive behavior, which represents the expectation of the impact of a series of future behaviors. t ,a t ) is the incentive function, S×A→Δ(R), is the incentive generated by the intelligent agent action, and is a designed deterministic function. P(s t+1 ,a t+1 |st ,a t ) indicates the implementation of action a t The system state s is reached t+1 ,a t+1 The probability of , γ represents the current discount rate, which is a constant, is the new state s t+1 The maximum value of the incentive expectation of all possible actions. Once Q(s t ,a t ) is calculated, the system can directly select the intervention discourse with the highest Q value and present it to the online collaborative reading seminar through the conversation agent, i.e. the intelligent agent, so that learners can perceive their own learning status in order to maximize the learners' learning effect.

[0078] First, intervention content must be presented according to expert definitions, supplemented by randomization rules (the same intervention content can be presented in a gain or loss framework). The learner's final learning results must be observed and recorded. Once a sufficient amount of data has been accumulated, training can begin. The randomization rules are necessary to increase the diversity of conversational text and accumulate sufficient data for reinforcement learning applications.

[0079] The algorithm flow is described as follows:

[0080] 1. Carry out an application experiment of collaborative reading learning and provide teaching intervention according to randomization.

[0081] The goal of the intelligent agent is to learn six strategies, which are used to guide the intervention actions in each state. The following is a specific definition of strategy: strategy is a mapping, which is denoted as π:S→Δ(A) in this paper. Δ(A) represents the probability distribution of system actions. In this case, the probability P(a) of taking a certain action a after executing the strategy in state s is t =a|s t =s) is denoted as π(a|s). Since the strategy is already determined, it can be written as π: S → A. The classification of the learned strategies is shown in Table 1.

[0082] Table 1 Classification of learning problem diagnosis and intervention strategies

[0083]

[0084] In previous studies, intelligent agents generally only considered obtaining environmental incentives for themselves. The innovation of the present invention is reflected in the complexity of the collaborative scenario: there are multiple learners involved, and the learners' speeches are detected by the intelligent agent to form incentives, which serve as the basis for the subsequent intervention. In this case, if the collaborative community only cares about maximizing its own expected rewards without considering the above context (i.e., the solution in the reinforcement learning of a single intelligent participant) and chooses the action of acceleration, it may cause the intervention to get out of control. The expected reward of the intelligent agent is the highest when choosing acceleration, and the expected discount needs to be accumulated in the real situation, so the present invention subtracts the current incentive state from the training function below. Each intelligent agent must take into account the previous semantic analysis and the actions of the real learners when determining their strategies.

[0085] The intelligent agent learns and iterates after t steps. For each intelligent agent, given the sampled data, the update rule of the intelligent agent learning is as follows:

[0086] Q(s t ,a t )←Q(s t ,a t )+α·(R t +γ·max(Q(s t+1 ,·))-Q(s t ,a t ))

[0087] Q(s t ,a t ) is the current action value, Q(s t+1 ,a t+1 ) is the action value to be obtained, s t is the state at step t, a t is the action at step t (same for step t+1), R t is the reward of step t, α is the learning rate, γ is the discount rate, and the present invention adopts a greedy strategy to combine the single-step reward and the maximum Q value and R that can be obtained after state transfer. t +γ·max(Q(s t+1 ,·)) as the target for each step update.

[0088] 2. Record the final learning effect of each learner and calculate R(s,a).

[0089] Reward function R(s t ,a t ) (abbreviated as R function): Agent is in state s t Take action a t Transfer to state a t+1The instantaneous reward is determined in advance based on the context of the collaborative session. The agent's ultimate goal is to maximize the cumulative instantaneous reward. To ensure contextual connection, this is done cumulatively, which is the innovation of this invention:

[0090] R t =r t +γr t+1 +γ 2 r t+1 +…+γ t-1 r t+1 ,0<γ≤1

[0091] At each step t, t = 1, 2, ..., of interacting with the environment, the agent perceives the state of the environment s t ∈S, then choose an action After executing the action, the environment feeds back a numerical reward and the next environmental state to the intelligent agent. Traditional intervention is that the probability and reward value of the current state transferring to the next state only depend on the current state and the selected action, and have nothing to do with the historical state and historical action. However, because of the integration of several situations, it is related to the previous actions of the learner and the intelligent agent, thus giving the effect of the previous text.

[0092] 3. Update Q(s,a) based on the data collected in steps 1 and 2.

[0093] The first and second steps need to be repeated to continuously provide training samples to the intelligent agent so that the Q function can be fully trained.

[0094] 4. Provide intervention strategies based on Q(s,a) and compare whether the learning effect has improved, while continuing to record data.

[0095] The matching learning strategy library focuses on: (1) providing individual metacognitive suggestions (Individual Metacognitive Suggestions) (2) providing group knowledge integration suggestions (Group Knowledge Integration) (3) promoting group communication suggestions (Communication Encouragement) (4) group progress coordination suggestions (Progress Coordination). See Table 1 above.

[0096] Table 1 shows the classification of intervention opportunities. The intervention content varies according to the intervention opportunity. The following is a detailed description of the intervention content. See Table 2.

[0097] Table 2 Example of problem representation and intervention strategy library

[0098]

[0099] 5. Combine the data from steps 1, 2, and 4, continue updating Q(s, a), and repeat steps 4 and 5 until the desired effect is achieved.

[0100] In combination with the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solutions to be protected by the present invention are as follows:

[0101] At present, collaborative learning interventions at home and abroad are mainly focused on the cognitive level of learners, but this technical solution can provide learners with motivational intervention information based on the previous text. The present invention seeks a solution that can provide learning support for learners who do not fully master the metacognitive strategies of collaborative learning, and help group members to effectively share, interact, construct, and collaborate. Although there is a possibility that their learning performance has not been improved, the intelligent agent, as a provider of learning suggestions, can help learners learn to learn collaboratively. A more obvious innovation is the innovation of the learning context, which is a motivational intervention based on the previous and next contexts, and the intelligent agent not only takes into account the incentive value of the intelligent agent, but also takes into account the previous and next contexts to avoid possible repetitive content, thereby avoiding cognitive pressure on learners.

[0102] Based on any of the above embodiments, the present invention provides a collaborative learning intervention system supported by reinforcement learning. Figure 5 This is the architecture diagram of the collaborative learning intervention system supported by reinforcement learning provided by the present invention, such as Figure 5 As shown, the system includes:

[0103] Data acquisition module 510, for detecting the status of the collaborative discussion environment based on the perceived state s t Select an intervention action a from the intervention action space t , when performing intervention action a t Afterwards, the reward r is given by the collaborative discussion environment t and the state s at the next moment t+1 , and then obtain the transfer data (s t, a t ,r t ,s t+1 ); the state s t A collaborative session including all learners in the collaborative discussion environment;

[0104] The data accumulation module 520 is used to repeatedly call the data accumulation module until a preset amount of transfer data is obtained;

[0105] a network training module 530 for training the Q network based on the preset amount of transfer data, and updating the Q value function using an improved Q learning method during the training process;

[0106] An action decision module 540 is configured to determine, from the intervention action space, a current intervention action corresponding to the maximum Q value in the current state using the trained Q network;

[0107] The intervention suggestion module 550 is used to make intervention suggestions to the learner based on the current intervention action.

[0108] The system provided by the embodiment of the present invention provides learners with motivational intervention information based on previous context by applying reinforcement learning to collaborative learning discussion scenarios. The previous context includes both the intervention discourse information previously implemented by the intelligent agent and the previous collaborative conversation information of all learners. This can provide learning support for learners who do not fully master the metacognitive strategies of collaborative learning, promote collaborative activities among learners and the development of individual metacognitive abilities, and avoid repetition and loss of control of intervention, which would cause cognitive stress for learners.

[0109] It is understandable that the detailed functional implementation of each of the above modules can be found in the introduction of the aforementioned method embodiment, and will not be repeated here.

[0110] In addition, an embodiment of the present invention provides another collaborative learning intervention device supported by reinforcement learning, which includes: a memory and a processor;

[0111] The memory is used to store computer programs;

[0112] The processor is configured to implement the method in the above embodiment when executing the computer program.

[0113] In addition, the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method in the above embodiment is implemented.

[0114] Based on the method in the above embodiment, an embodiment of the present invention provides a computer program product. When the computer program product runs on a processor, the processor executes the method in the above embodiment.

[0115] It will be easily understood by those skilled in the art that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A collaborative learning intervention method supported by reinforcement learning, characterized in that: include: S101 is based on the perceived state of the collaborative discussion environment t Select an intervention action a from the intervention action space t , when performing intervention action a t Afterwards, the reward r is given by the collaborative discussion environment t and the state s at the next moment t+1 , and then obtain the transfer data ; The state s t A collaborative session including all learners in the collaborative discussion environment; S102 repeats step S101 until a preset amount of transfer data is obtained; S103 trains the Q network based on the preset amount of transfer data, and during the training process, uses an improved Q learning method to update the Q value function; S104 uses the trained Q network to determine the current intervention action corresponding to the maximum Q value in the current state from the intervention action space; S105 provides intervention suggestions to the learner based on the current intervention action; The improved Q-learning method specifically updates the Q-value function based on the following formula: Where, It is The Q value of the step, is the status The maximum Q value under It is The state of the step, It is Intervention during stepping, It is The state of the step, It is Step reward, is the learning rate, is the discount rate; The intervention action space includes: personal metacognition suggestions, group progress coordination suggestions, group communication suggestions at the beginning of the task, group communication suggestions throughout the learning process, group knowledge integration suggestions in the example stage, and group knowledge integration suggestions in the argumentation stage.

2. The method according to claim 1, characterized in that S105 specifically includes: Based on the current intervention action, randomly select a corresponding gain frame or loss frame; The selected gain or loss frames are presented in a collaborative discussion environment to provide intervention suggestions to learners.

3. The method according to any one of claims 1 to 2, characterized in that Before S104, it also included: Based on the historical intervention actions performed and the historical collaboration sessions of all learners, it is determined whether to intervene in the learner.

4. A collaborative learning intervention system supported by reinforcement learning, characterized in that: include: Data acquisition module, used to perceive the state of the collaborative discussion environment t Select an intervention action a from the intervention action space t , when performing intervention action a t Afterwards, the reward r is given by the collaborative discussion environment t and the state s at the next moment t+1 , and then obtain the transfer data ; The state s t A collaborative session including all learners in the collaborative discussion environment; A data accumulation module, configured to repeatedly call the data accumulation module until a preset amount of transfer data is obtained; A network training module, configured to train the Q network based on the preset amount of transfer data, and during the training process, update the Q value function using an improved Q learning method; an action decision module, configured to determine, from the intervention action space, a current intervention action corresponding to the maximum Q value in the current state using the trained Q network; An intervention suggestion module, configured to make intervention suggestions to the learner based on the current intervention action; The improved Q-learning method in the network training module specifically updates the Q-value function based on the following formula: Where, It is The Q value of the step, is the status The maximum Q value under It is The state of the step, It is Intervention during stepping, It is The state of the step, It is Step reward, is the learning rate, is the discount rate; The intervention action space in the data acquisition module includes: personal metacognitive suggestions, group progress coordination suggestions, group communication suggestions at the beginning of the task, group communication suggestions throughout the learning process, group knowledge integration suggestions in the example stage, and group knowledge integration suggestions in the argumentation stage.

5. The system according to claim 4, characterized in that The intervention suggestion module is specifically used to: Based on the current intervention action, randomly select a corresponding gain frame or loss frame; The selected gain or loss frames are presented in a collaborative discussion environment to provide intervention suggestions to learners.

6. The system according to any one of claims 4 to 5, characterized in that It also includes an intervention judgment module for: Based on the historical intervention actions performed and the historical collaboration sessions of all learners, it is determined whether to intervene in the learner.

Images