Interaction method and device, electronic equipment and storage medium

Abstract

The invention provides an interaction method and device, electronic equipment and a storage medium. According to the embodiment of the invention, a target game is identified from a currently collected game question card; acquiring the game progress of the target game in real time in the game process of the target game, and acquiring sensing data of game participants of the target game in real time based on a sensing module; then, according to the game progress and the sensing data, when it is determined that intervention needs to be conducted on the game at present, a current corresponding intervention strategy is obtained; and finally, determining a current game operation behavior to be executed by utilizing an intervention tool indicated by the intervention strategy, and outputting the game operation behavior, so that the game participant executes the game operation behavior in a user interaction mode. On the basis, the learning machine can timely know lonely emotions and helpless emotions generated in the game process of the child, and reasonably intervenes and guides the user through an anthropomorphic intervention means, so that the user experience is greatly improved.

Description

Technical Field

[0001] The present disclosure relates to the field of early education interaction technologies, and in particular, to an interaction method, apparatus, electronic device, and storage medium. Background Art

[0002] With the continuous development of intelligent technologies and early education concepts, early education game products such as sudoku chess are becoming important tools for parents to cultivate children's logical thinking and problem-solving abilities. These games not only help children learn in a pleasant game, but also stimulate their curiosity and creativity.

[0003] However, existing early education game products often focus on the games themselves and are relatively rich in dimensions such as game types and age segments for games, but lack the sense of interaction brought by traditional parent-child interaction methods. This situation may lead to children feeling lonely and helpless during the learning process, thus affecting children's learning interest and effects. Summary of the Invention

[0004] To overcome the problems in the related art, the present disclosure provides an interaction method, apparatus, electronic device, and storage medium.

[0005] In a first aspect of the present disclosure, an interaction method is provided, which is applied to a learning machine. The method includes:

[0006] Identifying a target game from currently collected game question cards;

[0007] During the game process of the target game, collecting the game progress of the target game in real time, and collecting perception data of the game participants of the target game based on a sensing module in real time; the perception data includes at least one of voice, expression, and action;

[0008] When it is determined that the game needs to be intervened based on the game progress and the perception data, obtaining a corresponding intervention strategy at present;

[0009] Determining a currently to-be-executed game operation behavior by using an intervention tool indicated by the intervention strategy, and outputting the game operation behavior, so that the game participant executes the game operation behavior through a user interaction method.

[0010] Optionally, the determining a currently to-be-executed game operation behavior by using an intervention tool indicated by the intervention strategy, and outputting the game operation behavior, so that the game participant executes the game operation behavior through a user interaction method includes:

[0011] Determining a currently to-be-executed game operation behavior by using a main intervention tool indicated by the intervention strategy, and outputting the game operation behavior, so that the game participant executes the game operation behavior through a user interaction method;

[0012] After the game participant completes the game operation behavior, if it is determined that the game cannot continue due to the game operation behavior, the standby intervention tool indicated by the intervention strategy is used to determine the game operation behavior to be executed based on the game progress or the current game progress, and the game operation behavior is output, so that the game participant executes the game operation behavior through a user interaction method.

[0013] Optionally, the outputting the game operation behavior includes:

[0014] Select the current output method according to the weights of the output methods supported by the learning machine;

[0015] Determine whether the game participant cannot understand the game operation behavior due to the current output method. If so, select an output method from the other unused output methods according to the weights of the other unused output methods, and use the selected output method as the current output method, and return to the step of determining whether the game participant cannot understand the game operation behavior due to the current output method.

[0016] Optionally, after it is determined that the game participant cannot understand the game operation behavior due to the current output method, the method further includes:

[0017] Adjust the weight of the current output method, and the adjusted weight indicates that the priority of the current output method is lower than the priority of the current output method indicated by the weight before adjustment.

[0018] Optionally, the determining that the game needs to be intervened currently according to the game progress and the perception data includes:

[0019] Input the game progress and the perception data into a multimodal large model, so that the multimodal large model infers the user emotional state indicated by the perception data according to the specified user behavior included in the perception data, and determines that the game needs to be intervened currently according to the game progress and the user emotional state.

[0020] Optionally, the game question card includes a game question card corresponding to a sudoku chess game;

[0021] The real-time acquisition of the game progress of the target game includes:

[0022] Collect an image of the physical chessboard based on a camera module, and determine the user chess game according to the image collected by the camera module, where the user chess game is formed by physical chess pieces placed on the physical chessboard;

[0023] Determine the game progress of the target game according to the user's chess game and the game tasks corresponding to the game process.

[0024] Optionally, the perception data includes data in the form of dynamic media; and / or, the output of the game operation behavior includes:

[0025] Present the game operation behavior to the user based on the output method of dynamic media, where the dynamic media includes at least one of video and audio.

[0026] A second aspect of the present disclosure provides an interaction device, the device includes:

[0027] A game acquisition module, configured to identify a target game from the currently acquired game question cards;

[0028] A user perception module, configured to, during the game process of the target game, collect the game progress of the target game in real time, and, based on a sensing module, collect the perception data of the game participants of the target game in real time; the perception data includes at least one of voice, expression, and action;

[0029] An intervention decision module, configured to obtain a current corresponding intervention strategy when it is determined according to the game progress and the perception data that intervention in the game is currently required;

[0030] An intervention output module, configured to use the intervention tool indicated by the intervention strategy to determine the currently to-be-executed game operation behavior, and output the game operation behavior, so that the game participant executes the game operation behavior through a user interaction method.

[0031] Optionally, when the intervention output module is configured to use the intervention tool indicated by the intervention strategy to determine the currently to-be-executed game operation behavior, and output the game operation behavior, so that the game participant executes the game operation behavior through a user interaction method, it is specifically configured to:

[0032] Use the primary intervention tool indicated by the intervention strategy to determine the currently to-be-executed game operation behavior, and output the game operation behavior, so that the game participant executes the game operation behavior through a user interaction method;

[0033] After the game participant executes the game operation behavior, if it is determined that the game cannot continue to be executed due to the game operation behavior, use the backup intervention tool indicated by the intervention strategy to determine the to-be-executed game operation behavior based on the game progress or the current game progress, and output the game operation behavior, so that the game participant executes the game operation behavior through a user interaction method.

[0034] Optionally, when the intervention output module is used to output the game operation behavior, it is specifically used for:

[0035] Select the current output mode according to the weights of the output modes supported by the learning machine.

[0036] Determine whether the game participant cannot understand the game operation behavior due to the current output mode. If so, select an output mode from the other unused output modes according to the weights of the other unused output modes, and use the selected output mode as the current output mode, and return to the step of determining whether the game participant cannot understand the game operation behavior due to the current output mode.

[0037] Optionally, after determining that the game participant cannot understand the game operation behavior due to the current output mode, the device further includes a weight adjustment module:

[0038] Adjust the weight of the current output mode, and the adjusted weight indicates that the priority of the current output mode is lower than the priority of the current output mode indicated by the weight before adjustment.

[0039] Optionally, when the intervention decision module is used to determine that the game needs to be intervened according to the game progress and the perception data, it is specifically used for:

[0040] Input the game progress and the perception data into a multi-modal large model, so that the multi-modal large model infers the user's emotional state indicated by the perception data according to the specified user behavior included in the perception data, and determines that the game needs to be intervened according to the game progress and the user's emotional state.

[0041] Optionally, the game question card includes a game question card corresponding to the sudoku chess game; when the game collection module is used to collect the game progress of the target game in real time, it is specifically used for:

[0042] Collect an image of the physical chessboard based on the camera module, and determine the user chess game according to the image collected by the camera module. The user chess game is formed by physical chess pieces placed on the physical chessboard.

[0043] Determine the game progress of the target game according to the user chess game and the game task corresponding to the game process.

[0044] Optionally, the perception data includes data in the form of dynamic media; and / or, when the intervention output module is used to output the game operation behavior, it is specifically used for:

[0045] Present the game operation behavior to the user based on the output mode of dynamic media, where the dynamic media includes at least one of video and audio.

[0046] A third aspect of the present disclosure provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the method described in the first aspect is implemented.

[0047] A fourth aspect of the present disclosure provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the method described in the first aspect is implemented.

[0048] The technical solutions provided by the embodiments of the present disclosure may include the following beneficial effects:

[0049] In the embodiments of the present disclosure, the learning machine can collect the perception data of the game participants of the target game in real time based on the sensing module, and jointly determine whether to intervene in the user and the specific intervention strategy to be adopted by combining the perception data and the game progress that the current user is facing. Finally, based on the intervention tool indicated by the intervention strategy, generate the to-be-executed game operation behavior corresponding to the game progress, and prompt the game operation behavior to the user to form an effective guidance and prompt for the child. Since the learning machine determines the intervention strategy by obtaining the user state in real time through the sensing module and combining the game progress that the user is actually facing in real time, the learning machine actually has a completely anthropomorphic input form and acts as an anthropomorphic game assistant. Based on this, the learning machine can not only timely learn the loneliness and helplessness emotions generated by children during the game, but also timely intervene and guide the user more reasonably through more anthropomorphic intervention means, greatly improving the user experience.

[0050] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. Description of the Drawings

[0051] The drawings here are incorporated into the specification and constitute a part of the present disclosure, showing the embodiments that conform to the present disclosure, and are used together with the specification to explain the principles of the present disclosure.

[0052] Figure 1 It is a flowchart of an interaction method shown in some exemplary embodiments.

[0053] Figure 2 It is an application scenario diagram of an interaction method shown in some exemplary embodiments.

[0054] Figure 3 It is a flowchart of another interaction method shown in some exemplary embodiments.

[0055] Figure 4 It is a block diagram of an interaction device shown in some exemplary embodiments.

[0056] Figure 5 It is a hardware structure diagram of an electronic device shown in some exemplary embodiments. Detailed implementation manners

[0057] Here, the exemplary embodiments will be described in detail, and the examples are shown in the drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation manners described in the following exemplary embodiments do not represent all implementation manners consistent with the present disclosure. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0058] As described in the background art, existing early education game products often focus on the games themselves, but often lead to children likely feeling lonely and helpless during the learning process, thereby affecting children's learning interest and effects.

[0059] In view of this, the present disclosure provides an interaction method, device, electronic device, and storage medium. Next, the embodiments of the present disclosure will be described in detail.

[0060] The first aspect of the present disclosure provides an interaction method. Please refer to Figure 1 , which may include steps S101 to S104.

[0061] Step S101, identify a target game from the currently collected game question cards.

[0062] Please refer to Figure 2 , which is an exemplary embodiment of the present disclosure. The user can place the game question card at a specified position on the learning machine so that the learning machine can identify the game corresponding to the game question card. The game question card can be a physical item made of materials such as paper or plastic printed with information such as images, texts, chessboards, identification codes, etc., or can also be simulated by items such as a tablet computer or a mobile phone. The game question card contains game information, such as a certain chess game or a certain sudoku puzzle surface.

[0063] Step S102, during the game process of the target game, collect the game progress of the target game in real time, and collect the perception data of the game participants of the target game in real time based on the sensing module; the perception data includes at least one of voice, expression, and action.

[0064] Among them, the perception data refers to the data collected for game participants (i.e., users) (which can include various modalities of data). For example, it can be data based on dynamic media, and these data can represent speech, expressions, and actions through anthropomorphic senses. For instance, sensors such as cameras facing the user deployed on the learning machine or in the environment, microphones deployed in the environment, eye trackers facing the user, and body sensors worn on the user's body can be deployed. And the perception data can be videos or images captured by the camera, environmental data captured by the microphone (which can be denoised and separated from human voices), the position where the user is currently looking, the user's heart rate curve, etc. That is, the learning machine is connected to at least one sensor (i.e., the sensing module), and each sensor is used to collect the user's perception data through one or more dynamic media. Dynamic media, also known as time-based media or dynamic form media, refers to media that has a strong correlation with absolute time or relative time, such as audio, video (including animations), and the collection of multiple events occurring within a specific time period.

[0065] These sensors (i.e., sensors for collecting perception data based on dynamic media) can enable the learning machine to simulate human senses and truly obtain user information in the user's environment, thereby giving the user an unexpected sense of companionship. For example, when the user frowns and the heart rate accelerates, the learning machine can promptly give guidance such as "Please don't be impatient. You can try to solve the problem from a certain angle."

[0066] The game played by the user can be carried out offline. For example, by placing physical chess pieces on a physical chessboard to achieve game interaction. It can be understood that the physical chessboard can be served by the above-mentioned game card, or other physical chessboards can also be deployed in the environment, and the game card is only used to select the game. The present disclosure does not make any limitations in this regard. For a sudoku chess, a sudoku puzzle (problem surface) can be drawn on the physical chessboard, and the chess pieces can be any Arabic numeral from 1 to 9. When the user places a certain chess piece in a corresponding position, it is equivalent to the user filling in the Arabic numeral corresponding to the chess piece at that position.

[0067] At this time, the game card includes the game card corresponding to the sudoku chess game; the real-time collection of the game progress of the target game can include: collecting an image of the physical chessboard based on the camera module, and determining the user's chess game according to the image captured by the camera module, where the user's chess game is formed by the physical chess pieces placed on the physical chessboard; determining the game progress of the target game according to the user's chess game and the game tasks corresponding to the game process.

[0068] Thus, the user can regard the game card (or other physical chessboards) as a real chessboard and interact with the chessboard based on real chess pieces, thereby truly participating in the game through physical actions. During this process, both the interaction effect and the user's interaction experience will be enhanced. At the same time, it also avoids the potential health or psychological impact of the electronic screen on children (such as the impact on concentration).

[0069] In order to reduce the cost of the sensing module and obtain higher sensing accuracy, at least one surface of the above-mentioned chess piece can be provided with an identification code for identifying the above-mentioned chess piece (of course, the identification of the game card can also be realized based on the technology described here). The above-mentioned body is used to determine the current user chess game according to the identification code included in the image captured by the above-mentioned camera module. Among them, the identification code can be realized based on technologies such as O ID identification code, two-dimensional code, and text. Among them, the O ID (Optical Identify) identification code, also known as the O ID invisible code, has the characteristics of being almost invisible to the naked eye and having better adaptability to close-range scanning. In addition, the body can also be provided with a fill light for the field of view angle range of the camera module, so that the image quality collected by the camera module meets the identification standard of the identification code (that is, the body is provided with a fill light, and the irradiation range of the fill light overlaps all or part of the field of view angle range of the camera module).

[0070] Step S103, when it is determined that the game needs to be intervened according to the game progress and the perception data, obtain the corresponding intervention strategy at present.

[0071] The intervention strategy can be used to indicate the currently recommended intervention tool and output method. Among them, the intervention tool can be understood as "the algorithm or software and hardware module used for intervention". When the current game progress is input into the intervention tool, the currently to-be-executed game operation behavior can be obtained (that is, it is expected to prompt the user what operation to perform, or it is expected that the user plays the game with what idea), and the output method is the way to inform the user of the game operation behavior. In the above step, the intervention tool indicated by the intervention strategy can be multiple intervention tools with a priority relationship. For example, there can be a primary intervention tool and a backup intervention tool. Similarly, the learning machine can also have multiple output methods, such as outputting through voice (at the same time, different tones and timbres can also be set), outputting through images (such as displaying the current chess game on the screen and intervening the user through arrows, icons, text, etc.), outputting through video (such as displaying the current chess game on the screen and then playing the corresponding intervention animation, such as the animation of "picking up the chess piece and placing it in a certain position"), etc. The output method indicated by the intervention strategy can also be multiple output methods with a priority relationship and weights.

[0072] Regarding the determination of "whether game intervention is needed" and the determination of intervention strategies in the steps, exemplarily, at least one combination of software algorithms can be used to implement the above steps. For example, the user emotion information indicated by the perception data can be determined first (for example, it can be determined based on algorithms such as face recognition algorithm, expression classification algorithm, speech recognition algorithm, etc.), and it can be determined whether the current game progress meets a certain game goal (such as whether the sudoku is completed currently, or the duration since the user last placed the correct piece in the corresponding position at the current moment). Then, in the case where the user has not reached a certain game goal currently, the current intervention level score is determined by combining the current user emotion information (and information in dimensions such as the duration since the user last placed the correct piece in the corresponding position at the current moment), and based on the mapping relationship between the pre-set intervention level score and the intervention strategy, it is determined whether game intervention is needed currently and the current intervention strategy.

[0073] Exemplarily again, the above steps can be implemented based on a multimodal large model. For example, the determination of "currently needing to intervene in the game" according to the game progress and the perception data may include: inputting the game progress and the perception data into the multimodal large model, so that the multimodal large model infers the user emotion state indicated by the perception data based on the specified user behaviors included in the perception data, and determines that game intervention is needed currently according to the game progress and the user emotion state.

[0074] The specified user behaviors are the pre-set user behaviors, such as frowning, scratching the head, pursing the lips, biting the fingers, etc., and these user behaviors can be adapted to the specified user emotion states (such as anxiety, irritability, confusion, etc.). The above process can be implemented based on prompts (i.e., Prompts), or can be achieved by adjusting the structure of the model. For example, the large model can be used to obtain and understand the user request corresponding to the task (at this time, the user request can be set by the operator when the device leaves the factory, or can be set actively by the user), determine the execution goal (that is, "judging whether intervention needs to be executed currently, and what execution method to adopt in the case of needing intervention"), decompose the task into multiple goals according to the execution goal (for example, first determine the user behaviors included in the perception data, then determine the user emotion information indicated by the perception data, and so on until the execution goal is achieved) and several sub-tasks corresponding to each goal (the principle is the same above; of course, sub-tasks are not necessary), dispatch the tools corresponding to each sub-task from the tool library, dispatch the functional models corresponding to each sub-task from the functional model library (such as action recognition model, emotion inference model), complete the combined calculation or decision of data and models, and form a strategy plan corresponding to the input task.

[0075] Based on this, the execution efficiency and accuracy of the method can be further improved, and the user can be intervened in a suitable manner at the time when the user most needs intervention or the intervention effect is better.

[0076] Step S104, use the intervention tool indicated by the intervention strategy to determine the game operation behavior to be executed currently, and output the game operation behavior, so that the game participant executes the game operation behavior through the user interaction method.

[0077] As mentioned above, the learning machine can preset multiple output methods (in specific use, the user interaction method can be randomly selected, or indicated by the intervention strategy, or can also be specified in advance by the user). These output methods can include the method of outputting based on the dynamic media form. The perception data includes data in the dynamic media form; the output of the game operation behavior can include: presenting the game operation behavior to the user based on the output method of the dynamic media, where the dynamic media includes at least one of video and audio.

[0078] It can be considered that dynamic media has a special affinity for the main audience (children) of the learning machine. Based on this, the learning machine can obtain perception data in visual form, and then based on the multimodal large model, comprehensively determine whether to intervene in the user and the specific intervention tool to be used by combining the perception data and the game progress that the current user is facing. Finally, based on this intervention tool, generate the dynamic media content corresponding to this game progress. Since the learning machine determines the intervention strategy by obtaining the user status in real time through dynamic media and combining the game progress that the user is actually facing in real time, the learning machine actually acts as an anthropomorphic game assistant (with a completely anthropomorphic input and output form). This assistant can combine the recognition of the chessboard state, the child's expressions and actions to achieve omni-directional state perception, and play the interference content to the user based on dynamic media, thus simulating the scene of a real teacher or parent accompanying on the spot. Based on this, the learning machine can not only timely learn the lonely and helpless emotions generated by children during the game, but also timely and more reasonably intervene in and guide the user through more anthropomorphic intervention means, greatly improving the user experience.

[0079] Some steps of the above method can also have more optional implementation manners.

[0080] For example, in the dimension of intervention tools, determining the current game operation behavior to be executed by using the intervention tools indicated by the intervention strategy and outputting the game operation behavior, so that the game participant executes the game operation behavior in a user interaction manner, may include: determining the current game operation behavior to be executed by using the primary intervention tool indicated by the intervention strategy and outputting the game operation behavior, so that the game participant executes the game operation behavior in a user interaction manner; after the game participant executes the game operation behavior, if it is determined that the game cannot continue to be executed due to the game operation behavior, then determining the game operation behavior to be executed based on the game progress or the current game progress by using the backup intervention tool indicated by the intervention strategy, and outputting the game operation behavior, so that the game participant executes the game operation behavior in a user interaction manner.

[0081] In other words, the intervention tools can include a primary intervention tool (i.e., the preferred intervention tool) and a backup intervention tool. Different intervention tools can have different intervention tendencies. For example, in games such as Sudoku and chess, in fact, each step and each game goal may not necessarily have a unique correct solution, but can have multiple ideas; even in the same solution idea, different operation sequences can also be derived. Therefore, different intervention tools can correspond to different game difficulties or different problem-solving ideas.

[0082] The primary intervention tool can correspond to a more difficult game difficulty. That is to say, it can first try to give the user weak guidance or first try to guide the user to adopt a more standardized idea to solve the problem; when this guidance is ineffective (such as the user still cannot understand or the user still makes a wrong operation), the intervention tool can be replaced in time, and another idea (such as a simpler idea rather than a more standardized idea) can be tried to determine the recommended game operation behavior to be executed, and this operation is output to the user. Based on this, the flexibility of the learning machine in the educational process for the user can be greatly improved, the user experience and educational effect can be enhanced, and the negative impact caused by stereotypically repeating the same content to the user can be avoided.

[0083] Since the learning machine has output the game operation behavior to the user, it is possible to determine whether this game operation behavior causes the game to be unable to continue based on the user's subsequent operations. For example, after the learning machine outputs the game operation behavior to the user, it can collect the game progress again after a preset time interval. The difference between the game progress collected this time and the game progress before outputting this game operation behavior can be regarded as being caused by this game operation behavior. If this difference causes the game to be unable to continue (for example, the user performs an operation that does not conform to the game rules, or the user places the wrong chess piece in a certain position), then a backup intervention tool can be used to re-output the game operation behavior (the definition of "unable to continue" can be specifically related to the game rules and user settings; for example, it can be considered that if the operation actually performed by the user does not conform to the operation indicated by the above game operation behavior, then the game has already "been unable to continue"). Alternatively, the user's game operation behavior can also be collected through sensors, and after the user performs an operation, it can be determined whether this operation causes the game to be unable to continue (the principle is the same as above). It should be understood that in the description of the present disclosure, "not operating within a certain time period" itself can also be regarded as an operation; if the user does not perform any operation within the preset time period, it can also be regarded as "the current game operation behavior causes the game to be unable to continue"; because this indicates that the user cannot continue to complete the game normally according to this game operation behavior (for example, the user may not understand the idea behind this game operation behavior at all).

[0084] In the dimension of the current output method, the output of the game operation behavior may include: selecting the current output method according to the weights of the output methods supported by the learning machine; determining whether the game participant cannot understand the game operation behavior due to the current output method. If so, according to the weights of the other unused output methods, select an output method from the other unused output methods, and use the selected output method as the current output method, and return to the step of determining whether the game participant cannot understand the game operation behavior due to the current output method.

[0085] Similarly to the above, since the learning machine has output the game operation behavior to the user based on a certain output method, it is possible to determine whether the game participant cannot understand the game operation behavior according to the user's subsequent operations; if so, re-output the game operation behavior in another output method, and then determine again whether the game participant can understand the game operation behavior, thereby greatly improving the flexibility and educational effect of education.

[0086] For a learning machine, "whether the game participant cannot understand the game operation behavior due to the current output method (that is, when the user's operation is incorrect and the intervention is ineffective, it is considered that the current error is caused by the interaction method)" and "the game cannot continue to execute due to the game operation behavior (that is, it is considered that the current error is caused by the game operation behavior)" can be homogeneous to a certain extent, that is, if one of them occurs, it is considered that the other also occurs (therefore, the optional implementation details and beneficial effects regarding this can be referred to the above explanations. For example, when the user places the wrong chess piece, it is considered that "the user cannot understand the game operation behavior due to the current output method", which will not be elaborated here); or, they can also be heterogeneous (or only considered homogeneous when no user instruction is clearly received). For example, based on other algorithms, large models, or based on the user's instruction, to determine which specific situation has occurred currently. For example, when the game cannot continue, the user's voice collected by the microphone can be used to determine whether the user directly expresses "cannot understand the video" (that is, caused by the interaction method) or "cannot understand the idea" (that is, caused by the game operation behavior).

[0087] If the embodiments in the above two dimensions are comprehensively applied, then after the user performs an operation that does not conform to the game rules, the above two dimensions can be adjusted comprehensively or sequentially according to the preset logic. For example, the output method can be changed first, and then it is determined again whether the game cannot continue (for example, first output a certain game operation behavior to the user in the form of voice, and when the game does not proceed normally, then directly display the game operation behavior on the screen in the form of video). If the game still cannot continue (for example, the user still performs incorrect operations, or the user still does not perform any operations), then switch to the backup intervention tool to re-output the game operation behavior, and then use a certain interaction method to output the new game operation behavior to the user, and then repeat the above steps.

[0088] Of course, it is also possible to replace both the intervention tool and the output method after the same judgment; or replace the intervention tool first, and then replace the output method if it still does not work. Since the deployment logic of the implementation method has been clarified above, various optional technical combination methods will not be enumerated here.

[0089] In addition, each intervention tool and each output method can also have corresponding weights, so that the learning machine can generate a memory of the user. For example, after it is determined above that the game participant cannot understand the game operation behavior due to the current output method, the method can further include: adjusting the weight of the current output method, and the adjusted weight indicates that the priority of the current output method is lower than the priority of the current output method indicated by the weight before adjustment.

[0090] In other words, in the steps related to "switching the output mode" and "switching the intervention tool" above, after switching the output mode or the intervention tool, the weights of each output mode or intervention tool can be adjusted (for example, reducing the weight corresponding to the output mode or intervention tool before switching), so as to lower the priority of the output mode or intervention tool before switching, so that in the subsequent game process, other output modes or intervention tools are more likely to be used instead of this output mode or intervention tool (that is, if this output mode or intervention tool fails to achieve the expected effect, then in the subsequent process, this output mode or intervention tool can be used as little as possible). For example, when the voice prompt is ineffective, the video mode is more likely to be used to prompt the user in the subsequent process, so as to take care of the different emphases of different children on the game ideas and intervention methods as much as possible, and achieve a better educational effect and user experience.

[0091] It can be understood that in this embodiment, the steps of determining the intervention tool and the output mode in the above text can be adaptively adjusted. For example, in addition to combining the perception data and the current game progress, the weights (priorities) of the output mode and the intervention tool should also be combined to comprehensively determine the currently used intervention strategy, which will not be elaborated here.

[0092] The following combines Figure 3 The flowchart shown briefly reviews the embodiments of the present disclosure. First, the user can insert the question card into the specified position so that the learning machine can identify the question card and determine the game goal (and game rules). During the game process, the learning machine can collect images of the physical chessboard area and determine the physical chess pieces placed by the user on the chessboard, so as to determine the user's current game progress; at the same time, the learning machine can also collect the user's perception data through anthropomorphic sensing devices.

[0093] When the game is not completed, the learning machine can input the game progress and perception data into the multi-modal large model. The multi-modal large model can decompose the task of "obtaining an intervention strategy based on the perception data and game progress" and refine the subtasks to obtain at least one task, and then execute the task to determine whether the user should be intervened at present. If so, obtain an intervention tool combination (i.e., a preset intervention tool), and determine the primary intervention tool and the backup intervention tool (the backup intervention tool is not limited to one intervention tool, but can also be multiple intervention tools with an order; in addition, the primary and backup output methods can also be determined, similar to the intervention tool), and output the intervention strategy accordingly. Finally, the learning machine can determine the game operation behavior based on the primary intervention tool (for example, input the current game progress into a certain algorithm to obtain the recommended next game operation to be executed), and then determine whether the operations performed by the user within a certain period of time thereafter, or a certain number of operations performed by the user thereafter, conform to the game operation behavior. If not (for example, the operation does not conform to the game rules, the operation is not the correct answer, or there is no operation within a certain period of time), the output method and the intervention tool can be replaced according to the preset logic, and then the game operation behavior is obtained again, the game operation behavior is output, and the above judgment is executed.

[0094] In summary, the present disclosure can sense the chessboard state on the child's desktop and the child's expressions and actions through the device camera and image recognition technology, and sense the child's voice through the device microphone and voice recognition technology. Then, through intelligent decision-making technology, planning, memory, and tool invocation are performed, and finally, the intervention learning process is automatically implemented.

[0095] Corresponding to the embodiments of the foregoing method, the present disclosure also provides embodiments of a device and a terminal to which the device is applied.

[0096] A second aspect of the present disclosure provides an interaction device, please refer to Figure 4 , the device includes:

[0097] A game collection module 401, configured to identify a target game from the currently collected game question cards;

[0098] A user perception module 402, configured to, during the game process of the target game, collect the game progress of the target game in real time, and collect the perception data of the game participants of the target game in real time based on a sensing module; the perception data includes at least one of voice, expression, and action;

[0099] An intervention decision module 403, configured to obtain a current corresponding intervention strategy when it is determined according to the game progress and the perception data that the game needs to be intervened currently;

[0100] The intervention output module 404 is configured to determine the game operation behavior to be currently executed by using the intervention tool indicated by the intervention strategy, and output the game operation behavior, so that the game participant executes the game operation behavior through a user interaction method.

[0101] Optionally, when the intervention output module is configured to determine the game operation behavior to be currently executed by using the intervention tool indicated by the intervention strategy, and output the game operation behavior, so that the game participant executes the game operation behavior through a user interaction method, it is specifically configured to:

[0102] Determine the game operation behavior to be currently executed by using the primary intervention tool indicated by the intervention strategy, and output the game operation behavior, so that the game participant executes the game operation behavior through a user interaction method;

[0103] After the game participant executes the game operation behavior, if it is determined that the game cannot continue to be executed due to the game operation behavior, then use the backup intervention tool indicated by the intervention strategy to determine the game operation behavior to be executed based on the game progress or the current game progress, and output the game operation behavior, so that the game participant executes the game operation behavior through a user interaction method.

[0104] Optionally, when the intervention output module is configured to output the game operation behavior, it is specifically configured to:

[0105] Select the current output method according to the weights of the output methods supported by the learning machine;

[0106] Determine whether the game participant cannot understand the game operation behavior due to the current output method. If so, select an output method from the other unused output methods according to the weights of the other unused output methods, and use the selected output method as the current output method, and return to the step of determining whether the game participant cannot understand the game operation behavior due to the current output method.

[0107] Optionally, after determining that the game participant cannot understand the game operation behavior due to the current output method, the device further includes a weight adjustment module:

[0108] Adjust the weight of the current output method, and the adjusted weight indicates that the priority of the current output method is lower than the priority of the current output method indicated by the weight before adjustment.

[0109] Optionally, when the intervention decision module is configured to determine that the game needs to be intervened currently according to the game progress and the perception data, it is specifically configured to:

[0110] Input the game progress and the perception data into a multi-modal large model, so that the multi-modal large model infers the user's emotional state indicated by the perception data based on the specified user behavior included in the perception data, and determines that the game needs to be intervened currently according to the game progress and the user's emotional state.

[0111] Optionally, the game card includes the game card corresponding to the sudoku chess game; when the game acquisition module is used to collect the game progress of the target game in real time, it is specifically used for:

[0112] Collect an image of the physical chessboard based on the camera module, and determine the user's chess game according to the image collected by the camera module, where the user's chess game is formed by physical chess pieces placed on the physical chessboard;

[0113] Determine the game progress of the target game according to the user's chess game and the game tasks corresponding to the game process.

[0114] Optionally, when the intervention output module is used to output the game operation behavior, it is specifically used for:

[0115] Present the game operation behavior to the user based on the output method of dynamic media, where the dynamic media includes at least one of video and audio.

[0116] For the implementation processes of the functions and roles of each module in the above device, please refer to the implementation processes of the corresponding steps in the above method for details, and will not be elaborated here.

[0117] Adaptively, the present disclosure also provides a computer program product, including computer programs / instructions, which when executed by a processor implement the method provided in the foregoing embodiments.

[0118] For the device embodiments and the computer program product embodiments, since they basically correspond to the method embodiments, the relevant parts can refer to the partial description of the method embodiments. In addition, the device embodiments described above are only illustrative, and the modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place, or may be distributed to multiple network modules.

[0119] Adaptively, some embodiments of the present disclosure provide an electronic device, please refer to Figure 5 , which shows the structure of the electronic device. The electronic device includes a memory and a processor. The memory is used to store computer instructions that can run on the processor, and the processor is used to implement the method shown in any of the foregoing embodiments when executing the computer instructions.

[0120] Adaptively, the present disclosure also provides a non-transitory computer-readable storage medium including instructions, such as a memory including instructions, and the above instructions can be executed by an electronic device or a processor of the electronic device to complete the method shown in any of the foregoing embodiments. For example, the non-transitory computer-readable storage medium may be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0121] It should be understood that in some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve the desired result. Also, the various embodiments provided by the present disclosure may be applied independently, or may be combined and applied comprehensively. The present disclosure aims to cover any variations, uses, or adaptive changes of the present disclosure, and these variations, uses, or adaptive changes follow the general principles of the present disclosure and include the common general knowledge or conventional technical means in the technical field not claimed in the present disclosure. In addition, the content provided above is only a preferred embodiment of the present disclosure and is not used to limit the present disclosure. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present disclosure shall be included within the scope of protection of the present disclosure.

Claims

1. An interactive method, characterized in that: Applied to a learning machine, the method comprises: Identify the target game from the currently collected game cards; During the game process of the target game, the game progress of the target game is collected in real time, and the perception data of the game participants of the target game is collected in real time based on the sensor module; the perception data includes at least one of voice, expression and action; When it is determined that the game currently needs to be intervened according to the game progress and the perception data, obtaining a current corresponding intervention strategy; The intervention tool indicated by the intervention strategy is used to determine the game operation behavior to be currently executed, and the game operation behavior is output, so that the game participant executes the game operation behavior through user interaction.

2. The interactive method according to claim 1, characterized in that: The intervention tool using the intervention strategy instruction determines the current game operation behavior to be executed, and outputs the game operation behavior so that the game participant executes the game operation behavior through user interaction, including: Determine the current game operation behavior to be executed using the main intervention tool indicated by the intervention strategy, and output the game operation behavior so that the game participant executes the game operation behavior through user interaction; After the game participant has completed the game operation behavior, if it is determined that the game cannot continue to be executed due to the game operation behavior, the backup intervention tool indicated by the intervention strategy is used to determine the game operation behavior to be executed based on the game progress or the current game progress, and the game operation behavior is output, so that the game participant can perform the game operation behavior through user interaction.

3. The interactive method according to claim 1 or 2, characterized in that: The outputting the game operation behavior includes: Selecting a current output mode according to the weights of the output modes supported by the learning machine; Determine whether the current output mode causes the game participants to be unable to understand the game operation behavior. If so, select an output mode from the other output modes that have not been used according to the weights of the other output modes that have not been used, and use the selected output mode as the current output mode, and return to the step of determining whether the current output mode causes the game participants to be unable to understand the game operation behavior.

4. The interactive method according to claim 3, characterized in that: After determining that the game participants are unable to understand the game operation behavior due to the current output method, the method further includes: adjusting the weight of the current output method, the adjusted weight indicating that the priority of the current output method is lower than the priority of the current output method indicated by the weight before adjustment.

5. The interactive method according to claim 1, characterized in that: The determining, based on the game progress and the perception data, that an intervention in the game is currently required includes: The game progress and the perception data are input into a multimodal large model, so that the multimodal large model can infer the user emotional state indicated by the perception data based on the specified user behavior contained in the perception data, and determine the current need for intervention in the game based on the game progress and the user emotional state.

6. The interactive method according to claim 1, characterized in that: The game question cards include game question cards corresponding to the Sudoku game; The real-time acquisition of the game progress of the target game includes: Based on the image of the physical chessboard captured by the camera module, a user chess game is determined according to the image captured by the camera module, wherein the user chess game is formed by physical chess pieces placed on the physical chessboard; The game progress of the target game is determined according to the game tasks corresponding to the user chess game and the game process.

7. The interactive method according to claim 1, characterized in that: The sensory data includes data in the form of dynamic media; And / or, the outputting the game operation behavior includes: The game operation behavior is presented to the user based on an output method of dynamic media, wherein the dynamic media includes at least one of video and audio.

8. An interactive device, characterized in that: The device comprises: A game collection module is used to identify the target game from the currently collected game cards; A user perception module is used to collect the game progress of the target game in real time during the game process of the target game, and to collect the perception data of the game participants of the target game in real time based on the sensor module; the perception data includes at least one of voice, expression and action; An intervention decision module, for obtaining a current corresponding intervention strategy when it is determined that the game needs to be intervened according to the game progress and the perception data; The intervention output module is used to determine the game operation behavior to be currently executed using the intervention tool indicated by the intervention strategy, and output the game operation behavior so that the game participant can execute the game operation behavior through user interaction.

9. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images