Interaction control method based on eye movement track analysis and teaching machine thereof

Abstract

The invention discloses an interaction control method based on eye movement track analysis and a teaching machine thereof, and the method comprises the steps: obtaining the attention duration of the eye movement track of a target user for each video in a teaching video in the playing process of the teaching video based on a visual camera; obtaining a first target video of which the attention duration is greater than or equal to a preset duration, and displaying the first target video on a display interface of the teaching machine to obtain a second target video and a text prompt word; performing feature extraction on the second target video to obtain visual features of the second target video; performing feature extraction on the text cue words to obtain text features of the text cue words; and inputting the visual features and the text features into a pre-trained large language model to obtain a description text of the to-be-analyzed target output by the large language model. According to the invention, the analysis and description of the specific target in the video are completed, the interested content of the target user is focused, and the learning enthusiasm and learning efficiency of the target user are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to an interactive control method based on eye movement trajectory analysis and a teaching machine thereof. Background Art

[0002] With the development of artificial intelligence technology, the use of large language models for teaching interaction has gradually been promoted. In some related technologies, the visual encoder can be directly used to extract the global image feature sequence from the original video, and the global image feature sequence can be mapped to the semantic space of the large language model to obtain the video description text output by the large language model.

[0003] However, the existing teaching interaction methods can only extract the global image features of the video, but it is difficult to extract the local image features of the video. Therefore, in the scenario where only specific targets in the video need to be described, the existing teaching interaction methods are difficult to complete the analysis and description of specific targets in the video, and cannot focus on the content that the target users are interested in, resulting in low students' learning enthusiasm, which affects students' learning efficiency. Summary of the invention

[0004] The present invention provides an interactive control method based on eye movement trajectory analysis and a teaching machine thereof, which are used to solve the existing technical problems that it is difficult to complete the analysis and description of specific targets in a video and it is impossible to focus on the content that the target user is interested in, thereby improving the learning enthusiasm and learning efficiency of the target user.

[0005] In a first aspect, the present invention provides an interactive control method based on eye movement trajectory analysis, which is applied to a teaching machine, wherein the teaching machine is equipped with a visual camera, and the interactive control method based on eye movement trajectory analysis comprises:

[0006] Acquiring, based on the visual camera, the target user's eye movement track during the teaching video playback process for the attention duration of each video in the teaching video;

[0007] A first target video having an attention duration greater than or equal to a preset duration is obtained, and the first target video is displayed on a display interface of the teaching machine to obtain a second target video and a text prompt word; the text prompt word is input by the target user on the display interface, and the second target video includes a plurality of images, each of which includes a target to be analyzed selected by the target user;

[0008] Performing feature extraction on the second target video to obtain visual features of the second target video;

[0009] Performing feature extraction on the text prompt word to obtain text features of the text prompt word;

[0010] The visual features and the text features are input into a pre-trained large language model to obtain a description text of the target to be analyzed output by the large language model.

[0011] In a second aspect, the present invention provides a teaching machine, wherein the teaching machine is equipped with a visual camera, and the teaching machine comprises:

[0012] A monitoring module, for obtaining, based on the visual camera, the target user's eye movement track during the teaching video playback process, and the attention duration of each video in the teaching video;

[0013] The interactive display module is used to obtain a first target video whose attention time is greater than or equal to a preset time, and display the first target video on the display interface of the teaching machine to obtain a second target video and a text prompt word; the text prompt word is input by the target user on the display interface, and the second target video includes multiple images, each image includes a target to be analyzed selected by the target user;

[0014] An acquisition module, used for performing feature extraction on the second target video to obtain visual features of the second target video;

[0015] A feature extraction module, used to extract features of the text prompt word to obtain text features of the text prompt word;

[0016] The prediction module is used to input the visual features and the text features into a pre-trained large language model to obtain a description text of the target to be analyzed output by the large language model.

[0017] In a third aspect, the present invention further provides a teaching machine, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the interactive control method based on eye movement trajectory analysis as described in the first aspect is implemented.

[0018] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the interactive control method based on eye movement trajectory analysis as described in the first aspect.

[0019] In a fifth aspect, the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements the interactive control method based on eye movement trajectory analysis as described in the first aspect.

[0020] The interactive control method based on eye movement trajectory analysis provided by the present invention allows the target user to select the target to be analyzed by himself, performs feature extraction on the second target video containing the target to be analyzed selected by the target user to obtain visual features, then performs feature extraction on the text prompt words of the target user to obtain text features, and uses a large language model to analyze the visual features and text features to obtain a description text of the target to be analyzed, thereby completing the analysis and description of specific targets in the video, focusing on the content that the target user is interested in, improving the learning enthusiasm of the target user, and further improving the learning efficiency of the target user. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the present invention or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0022] Figure 1 It is a flow chart of the interactive control method based on eye movement trajectory analysis provided by the present invention;

[0023] Figure 2 It is a structural schematic diagram of the teaching machine provided by the present invention;

[0024] Figure 3 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0026] In the description of the present invention, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0027] In the description of the present invention, the term "for example" is used to mean "used as an example, illustration or explanation". Any embodiment described as "for example" in the present invention is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is given to enable any technician in the field to implement and use the present invention. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art can recognize that the present invention can be implemented without using these specific details. In other examples, well-known structures and processes will not be elaborated in detail to avoid obscuring the description of the present invention with unnecessary details. Therefore, the present invention is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed in the present invention.

[0028] See also Figure 1 , Figure 1 : is a flow chart of an interactive control method based on eye movement trajectory analysis provided by the present invention. The interactive control method based on eye movement trajectory analysis is applied to a teaching machine, in which a visual camera is installed. The interactive control method based on eye movement trajectory analysis includes:

[0029] Step 10: obtaining the target user's eye movement trajectory and the attention duration of each video in the teaching video during the teaching video playback process based on the visual camera.

[0030] The scenario of the embodiment of the present invention is an application scenario of teaching by playing video on a teaching machine, that is, the target user (any student) follows the video in front of the teaching machine to learn. Therefore, in the process of playing the teaching video on the teaching machine, the target user's eye movement track is obtained through a visual camera for the attention time of each video in the teaching video, wherein the attention time is also the length of time the target user stays on the video.

[0031] Step 20, obtaining a first target video whose attention duration is greater than or equal to a preset duration, and displaying the first target video on a display interface of the teaching machine to obtain a second target video and a text prompt word.

[0032] Furthermore, the teaching machine determines whether the target user's eye movement track focuses on each video in the teaching video for a duration greater than or equal to a preset duration, wherein the preset duration is set according to actual conditions, such as 2 seconds or 3 seconds.

[0033] If it is determined that the attention duration is greater than or equal to the preset duration, the teaching machine determines the video corresponding to the attention duration as the first target video, and at the same time, the teaching machine displays the first target video on the display interface of the teaching machine. At this time, the target user can select the corresponding target to be analyzed in the first target video, and enter a text prompt word in the display interface. After the target user selects the corresponding target to be analyzed, the teaching machine obtains the second target video according to the target to be analyzed, wherein the second target video includes multiple images, each of which contains the target to be analyzed selected by the target user.

[0034] Specifically, the teaching machine is embedded with a target user interaction processing module, a visual feature encoding module, a target mask decoding module and a graphic feature fusion module. Among them, the target user interaction processing module can obtain the second target video and the text prompt word of the target user.

[0035] Step 30: extract features from the second target video to obtain visual features of the second target video.

[0036] Step 40: extract features from the text prompt words to obtain text features of the text prompt words.

[0037] Specifically, the target user's text prompt word prompt is input into the text encoding network to obtain the text feature RT (i.e., text sequence feature) generated by the text encoding network. The dimension of RT is [textlength, d], where textlength is the length of the text word (token) and d is the dimension of each word.

[0038] Step 50: Input the visual features and text features into the pre-trained large language model to obtain the description text of the target to be analyzed output by the large language model.

[0039] The embodiment of the present invention allows the target user to select the target to be analyzed by himself, performs feature extraction on the second target video containing the target to be analyzed selected by the target user to obtain visual features, and then performs feature extraction on the text prompt words of the target user to obtain text features. The visual features and text features are analyzed using a large language model to obtain a description text of the target to be analyzed, thereby completing the analysis and description of specific targets in the video, focusing on the content that the target user is interested in, improving the learning enthusiasm of the target user, and further improving the learning efficiency of the target user.

[0040] In one embodiment, extracting features from the second target video to obtain visual features of the second target video includes:

[0041] Determine the target mask, depth features, video history features and encoding features of the first frame image; the first frame image is the first frame image of the second target video;

[0042] The next frame image of the first frame image is used as the current frame image;

[0043] Determine the depth feature of the current frame image based on the target mask of the current frame image and the previous frame image of the current frame image;

[0044] Determine the encoding feature of the current frame image based on the depth feature of the current frame image and the video history feature of the previous frame image of the current frame image;

[0045] Inputting the encoded features of the current frame image into the mask decoding network to obtain the target mask of the current frame image output by the mask decoding network;

[0046] Determine the video history feature of the current frame image based on the coding feature of the current frame image and the video history feature of the previous frame image of the current frame image;

[0047] The step of taking the next frame image of the current frame image as the current frame image, returning a target mask based on the current frame image and a frame image before the current frame image, and determining a depth feature of the current frame image, until the target mask, depth feature, video history feature, and encoding feature of each image in the second target video are obtained;

[0048] The video history features of the last frame image in the second target video are used as the visual features of the second target video.

[0049] In one embodiment, the target user interaction processing module is used to convert the target user's interaction operation of selecting a target into a signal executable by the system. The core of this module is the SAM model (SegmentAnything Model).

[0050] Specifically, the teaching machine can play a first target video, which may contain multiple targets, such as people, objects, animals, plants, specific areas, etc.; while the target user is watching the video, the target user can pause the video at any time, and the picture will stay on the image of the frame at that time. The target user can interact with the system, select a target of interest in the image as the target to be analyzed, and determine the processing time, thereby generating interactive information; wherein the processing time refers to the length of the video that the target user wants the system to analyze.

[0051] Furthermore, the teaching machine responds to the interaction information of the target user and can determine the target to be analyzed in the first target video and the processing time selected by the target user based on the interaction information, and determine the second target video based on the target to be analyzed in the first target video and the processing time selected by the target user.

[0052] For example, the first target video is half an hour long. The target user can select the target to be analyzed on the image at the 10th minute of the first target video and determine the processing time to be 15 minutes. That is, the target user hopes that the system will analyze a video of 15 minutes in length. The teaching machine can extract the second target video based on the video clips from the 10th to the 25th minute of the first target video.

[0053] Optionally, the target user may select the target to be analyzed by left-clicking the mouse or selecting the frame. It should be noted that, in this embodiment, the target user selects the target to be analyzed on which image, and the frame image is used as the first frame image I0 in the second target video, and the first frame image I0 is the first frame image of the second target video.

[0054] Furthermore, the target user's interaction information is used as a prompt, and the first frame image I0 and the interaction information are input into the SAM model of the target user interaction processing module to obtain the target mask mask0 of the first frame image I0 output by the SAM model.

[0055] The visual feature encoding module is used to generate visual features. Visual features should contain the following two types of information: one is the fusion of the mask information of the target to be analyzed, which can achieve the description of the specified target; the other is to contain all the image frame information, so that the model has the ability to describe the entire video. With the idea of ​​semi-supervised tasks, the visual feature encoding module can store historical image features, combine historical results with the current image, and predict the current result. The specific method is: store the historical features of the video frame in the memory, merge them with the previous target mask results and the current image features, update the historical features of the video and the current image features, and then decode the current image features to obtain the target mask result of the current image. In this process, the historical features of the video merge each frame of the image and its target mask information, which can be used to understand and describe specific targets in the video.

[0056] Furthermore, the visual feature encoding module performs splicing processing on the first frame image I0 and the target mask mask0 of the first frame image: the first frame image I0 and the target mask mask0 of the first frame image are spliced ​​in dimension to obtain a spliced ​​image; the spliced ​​image is input into a visual feature extraction network (such as a visual feature encoder ViT) for processing, and the visual feature extraction network can perform feature extraction processing on the spliced ​​image to obtain a depth feature R0 of the first frame image, and use the depth feature R0 of the first frame image as the video history feature RH0 of the first frame image; extract the encoding feature RN0 of the first frame image, and store the above-extracted feature in the server memory.

[0057] Furthermore, the next frame image of the first frame image is used as the current frame image.

[0058] In one embodiment, for each current frame image I n , can be based on the current frame image I n With the current frame image I n The previous frame image I n-1 The target mask mask n-1 , determine the current frame image I n The deep features R n .

[0059] Specifically, the current frame image I n With the current frame image I n The previous frame image I n-1 The target mask mask n-1 The spliced ​​images are spliced ​​in the dimension, and the visual feature extraction network is used to extract features from the spliced ​​images to obtain the current frame image I n The deep features R n .

[0060] Further, based on the current frame image I n The deep features R n and the current frame image I n The previous frame image I n-1 Video history features RH n-1 , determine the current frame image I n The encoding feature RN n .

[0061] Specifically, the current frame image I n The deep features R n As the index vector Q, the previous frame image I n-1 Video history features RH n-1 As the key vector K and value vector V, the three are input into the encoding feature update network to obtain the current frame image I output by the encoding feature update network n The encoding feature RN n , current frame image I n The encoding feature RN n Fusion of the previous frame image I n-1 Video history features.

[0062] Furthermore, the current frame image I n The encoding feature RN n Input to the mask decoding network to obtain the target mask of the current frame image output by the mask decoding network n .

[0063] Specifically, the current frame image I n The encoding feature RN n, as the mask decoding feature, is input into the mask decoding network of the target mask decoding module (the mask decoding network can adopt structures such as deconvolution and upsampling), and the target mask mask of the current frame image output by the mask decoding network is obtained. n .

[0064] The target mask of the current frame image n Can be used to generate the next frame image I n+1 The target mask mask n+1 , and then the masks {mask1,...,mask n}, and acts on the original video frame image to realize the visualization of the target segmentation result and provide the visualization result to the target user.

[0065] Further, based on the current frame image I n The encoding feature RN n and the current frame image I n The previous frame image I n-1 Video history features RH n-1 , the current frame image I can be determined n Video history features RH n .

[0066] Specifically, the previous frame image I n-1 Video history features RH n-1 As the index vector Q, the current frame image I n The encoding feature RN n As the key vector K and value vector V, the three are input into the historical feature update network to obtain the current frame image I output by the historical feature update network n Video history features RH n , current frame image I n Video history features RH n Fusion of the current frame image I n The encoding feature RN n .

[0067] Furthermore, the next frame image of the current frame image is taken as the current frame image, and the target mask based on the current frame image and the previous frame image of the current frame image is returned to determine the step of depth features of the current frame image until the target mask, depth features, video history features and encoding features of each image in the second target video are obtained.

[0068] It can be understood that for the second target video with a length of n (n is determined by the duration selected by the target user and the video frame rate), its encoding features can be expressed as {RN0, RN1, ..., RN n}, which combines the information of historical target mask, historical frame and current frame, and can be used to predict the target mask of the current frame, while the video history features {RH0,RH1,...,RH n} contains both the target mask information and the video history information of the previous frame. The final video history feature RH n Contains the historical information of the entire video.

[0069] Furthermore, the video history features of the last frame image in the second target video are used as the visual features of the second target video. In this embodiment, assuming that the length of the second target video is n, the last frame image I in the second target video is n The video history feature is RH n , which is used as the visual feature of the second target video.

[0070] The embodiment of the present invention provides a target user interaction function, where the target user selects the target to be analyzed, and uses the SAM model as an interaction bridge to convert the target user's interaction operation into visual segmentation of specific targets, thereby achieving the selection of the target of interest.

[0071] In one embodiment, the target mask, depth features, video history features and coding features of the first frame image are determined, including: determining the interaction information of the target user; inputting the first frame image and the interaction information into the SAM model to obtain the target mask of the first frame image output by the SAM model; splicing the first frame image and the target mask of the first frame image to obtain a spliced ​​image; performing feature extraction on the spliced ​​image based on a visual feature extraction network to obtain the depth features of the first frame image and the video history features of the first frame image; and extracting the coding features of the first frame image.

[0072] Specifically, the teaching machine can play a first target video (i.e., the original video), which may contain multiple targets, which may be people, objects, animals, plants, specific areas, etc.; while the target user is watching the video, the target user can pause the video at any time, and the picture will stay on the image of the frame at that moment. The target user can interact with the system, select a target of interest in the image as the target to be analyzed, and determine the processing time, thereby generating interactive information; wherein the processing time refers to the length of the video that the target user wants the system to analyze.

[0073] Furthermore, the teaching machine responds to the interaction information of the target user and can determine the target to be analyzed in the first target video and the processing time selected by the target user based on the interaction information, and determine the second target video based on the target to be analyzed in the first target video and the processing time selected by the target user.

[0074] It should be noted that, in this embodiment, the target user selects the target to be analyzed on which image, and the frame image is used as the first frame image I0 in the second target video. The first frame image I0 is the first frame image of the second target video.

[0075] Furthermore, the target user's interaction information is used as a prompt, and the first frame image I0 and the interaction information are input into the SAM model of the target user interaction processing module to obtain the target mask mask0 of the first frame image I0 output by the SAM model.

[0076] In one embodiment, after obtaining the target mask, depth features, video history features and encoding features of each image in the second target video, it also includes: based on the target mask of each image, performing target segmentation on each image respectively to obtain the target segmentation result of each image; and returning the target segmentation result of each image to the target user.

[0077] After obtaining the target mask of each image, target segmentation is performed on each image based on the target mask of each image to obtain the target segmentation result of each image; the target segmentation result of each image is returned to the target user to realize the visualization of the target segmentation result, so as to facilitate the target user to intuitively experience it.

[0078] In one embodiment, obtaining a second target video and a text prompt word of a target user includes: responding to interaction information of the target user, determining a target to be analyzed in the first target video and a processing time selected by the target user based on the interaction information; determining a second target video based on the target to be analyzed in the first target video and a processing time selected by the target user; and obtaining a text prompt word of the target user.

[0079] It should be noted that if the target user has not selected the target to be analyzed, the system may not perform the segmentation processing operation of the target user interaction. At this time, the dimension of the mask in the encoding feature update network is set to 0, and the system outputs a description text that will not be directed to a specific target in the video, but will describe the global image in the video; if the target user only selects the target to be analyzed but has not determined the processing time, the system only processes the current frame image and outputs a specific target description result for the current frame image; if the target user neither selects the target to be analyzed nor determines the processing time, the system outputs a global description text for the current frame image.

[0080] The embodiment of the present invention can adjust the system function in time according to the interactive settings such as the target and duration selected by the target user, and can realize the description of pictures / videos and local / global information.

[0081] In one embodiment, visual features and text features are input into a pre-trained large language model to obtain a description text of the target to be analyzed output by the large language model, including: dimensionally adjusting the visual features so that the visual features and the text features have the same feature dimensions; inputting the dimensionally adjusted visual features and text features into the pre-trained large language model to obtain a description text of the target to be analyzed output by the large language model.

[0082] In one embodiment, the core of the image-text feature fusion module is a pre-trained large language model (LLM), which fuses visual features and text features and inputs them into the large language model. The large language model can perform causal reasoning based on visual features and text features and output a descriptive text of the target to be analyzed.

[0083] In one embodiment, the target user's text prompt word prompt is input into the text encoding network to obtain the text feature R generated by the text encoding network. T , R T The dimension is [textlength, d], where textlength is the length of the text word (token) and d is the dimension of each word.

[0084] In this embodiment, assuming that the length of the second target video is n, the last frame image I in the second target video n The video history feature is RH n , which is used as the visual feature of the second target video, that is, the visual feature is RH n It is understandable that in order to make the visual feature RH n With text features R T With the same feature dimension, the MLP structure (generally using several layers of linear layers) can be used to adjust the visual feature RH n Dimension, visual features RH after dimension adjustment n The dimension is [M×M, d], where M=ImageSize (image size) / PatchSize (ViT partition image window size).

[0085] It should be noted that if the video history feature RH is not used n Instead of using the existing method as a visual feature, the depth features {R0, R1, ..., R n} as visual features, the dimension of the visual features will increase to [n×M×M, d], which will increase the system's computational overhead. In addition, for long videos, the dimension of the visual features may exceed the maximum processing token length of the large language model, thus affecting the generation of description text.

[0086] Furthermore, the visual features and text features that have been dimensionally adjusted are concatenated in the column dimension, combined with rotational position encoding, and input into the pre-trained large language model to obtain the descriptive text of the target to be analyzed output by the large language model, thereby realizing the description of the selected target in the video.

[0087] In one embodiment, for an application scenario where video teaching is not played by a teaching machine, that is, a teacher teaching on site, the interactive control method based on eye movement trajectory analysis in an embodiment of the present invention includes:

[0088] Obtain visual information of the teaching environment, including depth map and two-dimensional map;

[0089] Capturing the area where the target user's eyes focus on the teaching medium based on visual information, and generating a teaching medium heat map based on the area where the target user's eyes focus on the teaching medium;

[0090] Based on the teaching media heat map, the target users’ attention to the style, content, and layout of the teaching media is analyzed to obtain the analysis results.

[0091] Specifically, the teaching environment may be a facility environment that can provide teaching activities, and the teaching environment may be an indoor environment or an outdoor environment, which is not specifically limited in the embodiment of the present invention. For example, the teaching environment may be a classroom or an outdoor classroom.

[0092] Taking the classroom as an example, the teaching machine can be installed at an appropriate location in the classroom to ensure that the facial areas of all target users can be covered. The teaching machine is equipped with a visual camera, which is installed above the teaching machine to obtain depth maps and two-dimensional maps at the same time.

[0093] Visual cameras can obtain three-dimensional information of objects in the scene, including depth maps, point clouds, etc. This is crucial for capturing the target user's eye position, because the three-dimensional position information of the eye can more accurately reflect its gaze direction than a two-dimensional image. You can choose a suitable visual camera to ensure that it has high imaging accuracy and fast processing speed to meet the needs of capturing the target user's eye position in real time.

[0094] Eye tracking is a type of biometric technology that detects eye movement to infer the direction of a person's gaze. In eye tracking or face recognition applications, depth maps can be used as auxiliary information to help eye positioning algorithms more accurately lock the position of the eye. By combining depth maps with two-dimensional image information, the algorithm can more effectively filter out background noise and interference, improving the accuracy and robustness of eye positioning. In the target user's facial image obtained by the visual camera, the eye tracking algorithm can locate the position of the eye and further calculate its gaze point.

[0095] Here, the teaching medium refers to the carrier for delivering teaching information in the teaching process. Specifically in the classroom teaching process, the teaching medium may be, for example, blackboard writing, electronic courseware, specimens, experimental equipment, etc. The blackboard writing here may be electronic blackboard writing or traditional chalk blackboard writing, which is not specifically limited in the embodiment of the present invention.

[0096] The target user's eye focus area on the teaching medium refers to the area on the teaching medium where the target user's gaze point is projected. For example, for a certain page of electronic courseware, the target user 1's focus area is the geometric image on the electronic courseware, the target user 2's focus area is the formula on the electronic courseware, and so on. The target user's eye focus area on the teaching medium can be captured. In addition, the length of time each target user focuses on a certain area on the teaching medium can also be captured. For example, target user 3 focused on area 1 on the electronic courseware for 10 seconds and focused on area 2 for 20 seconds.

[0097] Based on the target user's eye focus area on the teaching medium, a teaching medium heat map is generated. The teaching medium heat map reflects the target user's attention level in each area of ​​the teaching medium.

[0098] The generation of the teaching media heat map can be obtained through data analysis methods. For example, according to the number of target users who follow each area and the length of time each target user pays attention, a comprehensive analysis is performed to obtain analysis data, and then the analysis data is mapped to the teaching media to generate a teaching media heat map. The depth of color in the teaching media heat map reflects the degree of attention of the target users to the area. For example, it can be configured that the darker the color, the higher the degree of attention; conversely, the lighter the color, the lower the degree of attention. For example, in the teaching media heat map, the areas that the winning target users focus on are marked in red, and other areas of attention are marked with gradient colors to reflect the degree of attention. This method can quantify and intuitively reflect the target users' attention to the teaching media.

[0099] The teaching medium heat map can be obtained through a pre-trained heat map generation model. The depth map and two-dimensional map in the visual information are input into the pre-trained heat map generation model to obtain the teaching medium heat map output by the model.

[0100] In one embodiment, the teaching medium heat map can intuitively reflect the target user's eye focus area on the teaching medium. Therefore, teaching behavior analysis can be performed based on the teaching medium heat map. The goal of interactive control based on eye movement trajectory analysis here is to evaluate the target user's attention, understanding or interest in different teaching media.

[0101] For example, different teaching media heat maps can be generated by different blackboard content and different blackboard writing styles to analyze the activity and attention of target users on different teaching media. By analyzing the high-heat areas and low-heat areas in the heat map, the parts of the teaching media that target users are most interested in and least interested in can be identified; by observing the differences in the feedback of target users on different teaching media in different time periods, it can be analyzed whether there are specific learning behavior patterns.

[0102] On this basis, the teaching media can be optimized through analysis results. According to the heat map analysis results, the content and form of the teaching media can be adjusted to improve the learning interest and participation of the target users. For high-heat areas, the content can be further enriched and interactive links can be added. For low-heat areas, the reasons need to be found (such as too high content difficulty, monotonous form, etc.) and corresponding improvements need to be made.

[0103] The embodiment of the present invention generates a teaching medium heat map based on the target user's eye focus area on the teaching medium, and the teaching medium heat map can more intuitively show the target user's focus. On this basis, interaction is performed based on the teaching medium heat map to understand the target user's attention distribution in the classroom, realize automatic analysis of feedback information, and then adjust the teaching methods and strategies according to the analysis results, which can improve the teaching effect.

[0104] In one embodiment, the style of the teaching medium refers to the unique visual, auditory or interactive characteristics of the teaching blackboard or courseware when presenting information, which together constitute the overall atmosphere and experience felt by the target user when contacting the teaching medium. The style of the teaching medium may include, for example, traditional style (such as blackboard writing), multimedia style (such as multimedia courseware combining multiple media elements such as text, images, audio, video, etc.), gamification style, etc.

[0105] The content of teaching media refers to the specific knowledge, information, skills or experience delivered and displayed to the target users through teaching media. The content of teaching media may include subject knowledge, such as geometric figures, mathematical formulas, conclusion sentences, circled content, etc.

[0106] The layout of teaching media refers to the spatial arrangement and organization of various elements (such as text, images, charts, buttons, etc.) when teaching blackboards or courseware present information. A good layout can enhance the viewing experience of target users and help them better understand and remember information.

[0107] By displaying teaching media of different styles, contents and layouts, we can obtain the corresponding teaching media heat map, and on this basis, analyze the target users' attention to the style, content and layout of the teaching media to obtain the analysis results.

[0108] For example, it can be clearly seen from the heat map that the target users may show a higher degree of attention to the teaching media style with rich colors and pictures, pay more attention to the picture information with expressions, and the explanation area of ​​certain key concepts or formulas has a higher heat intensity, indicating that these are the content that the target users focus on and learn. Teaching media with a reasonable layout and clear information hierarchy are more likely to attract the attention of target users. For example, teaching media that adopt the layout of "general-specific-general" or "problem-solving" often have a more even heat distribution. The reasonable use of elements such as titles, subtitles, and pictures can also help to increase the attractiveness of the layout. These elements are often shown as areas with higher heat intensity in the heat map.

[0109] The embodiment of the present invention analyzes the target users' attention to the style, content, and layout of the teaching media based on the teaching media heat map, and obtains analysis results, which can enable teachers to better meet the learning needs of target users when producing teaching media, thereby improving teaching effectiveness and the learning experience of target users.

[0110] In one embodiment, based on the teaching medium heat map, the attention of the target users to the style, content, and layout of the teaching medium is analyzed to obtain the analysis results, and then the following is further included:

[0111] Based on the analysis results of each teaching medium, a target teaching medium is screened from each teaching medium, and / or the teaching medium is iteratively optimized.

[0112] Specifically, the target teaching medium can be a teaching medium with a high degree of attention in terms of style, content, and layout. Each teaching medium can be a teaching medium produced for different teachers, that is, a teaching medium with different styles, contents, and layouts, or a teaching medium obtained after optimization in the previous iteration. For example, a threshold of the degree of attention can be set in advance, and teaching media with a degree of attention greater than the threshold can be used as the target teaching medium, thereby screening out the target teaching medium. It can promote the production level of teaching media for teachers throughout the school, thereby improving the effectiveness of knowledge transfer and improving the quality of teaching.

[0113] Iterative optimization of teaching media can be achieved by regularly analyzing the teaching media after the last optimization to understand the target users' attention level and usage experience with the teaching media, implementing the redesigned teaching media in actual teaching, and obtaining analysis results through teaching media heat maps, thereby achieving continuous iterative optimization of teaching media.

[0114] In addition, the analysis results of each teaching medium can provide the direction for iterative optimization. If any of the style, content, or layout of the teaching medium receives less attention, iterative optimization can be carried out starting from the dimension with less attention.

[0115] In one embodiment, the interactive control method based on eye movement trajectory analysis further includes:

[0116] Determine the teacher's movement trajectory during the teaching process based on visual information, and generate a movement trajectory heat map based on the movement trajectory;

[0117] Analyze teachers’ teaching behaviors based on the movement trajectory heat map.

[0118] Specifically, the teacher's location information is extracted through visual information, and image processing technology, such as feature point detection and tracking algorithm, is used to continuously track the teacher's location information to form a complete movement trajectory. The tracked data can also be cleaned here to remove noise and outliers to ensure the accuracy and continuity of the movement trajectory.

[0119] Then select a suitable data visualization tool, such as ArcGIS, Tableau, QGIS, or matplotlib and seaborn libraries in Python, and import the cleaned teacher movement trajectory data into the visualization tool.

[0120] According to the classroom layout, the parameters of the heat map are set, such as color mapping range, heat point size, transparency, etc. According to the set parameters, a heat map of the teacher's movement trajectory is generated, that is, a movement trajectory heat map, in which the movement trajectory heat map reflects the teacher's movement frequency and stay time in different areas through the depth of color.

[0121] On this basis, the teacher's teaching behavior is analyzed based on the mobile trajectory heat map. Through the mobile trajectory heat map, we can intuitively see the teacher's movement pattern in the classroom, such as frequently visited areas and seldom visited areas.

[0122] Combined with the teacher's movement trajectory, analyze the teacher's teaching style. For example, if the teacher moves frequently in the front and middle of the classroom, it may indicate that he focuses on face-to-face communication with the target users and blackboard demonstrations; if the teacher stays in the back or corner of the classroom for a long time, it may be that he is conducting individual tutoring or observing the target users.

[0123] By observing the hot spots and cold spots in the movement trajectory heat map, you can evaluate the interaction in the class. Hot spots may indicate that the target users are active and highly engaged, while cold spots may indicate that the target users are not engaged or there are blind spots in teaching.

[0124] Based on the analysis results, targeted improvement suggestions can be put forward. For example, for teaching blind spots, teachers can adjust teaching strategies and increase attention to this area; for low participation of target users, teachers can design more interactive links to increase the learning interest and participation of target users.

[0125] In one embodiment, the interactive control method based on eye movement trajectory analysis further includes:

[0126] Perform behavioral recognition on visual information to obtain the classroom behavior of target users;

[0127] Conduct teaching behavior analysis based on the teaching media heat map and the classroom behavior of target users.

[0128] Specifically, the classroom behavior of the target user can be obtained through behavior recognition technology, and behavior recognition can be obtained through human key point detection and target detection. For example, algorithms such as OpenPose are used to extract human key points, such as head, hands, body, etc., as posture information. These key point information can reflect the different postures of the target user, such as sitting upright, raising hands, leaning sideways, lowering the head, etc. Using the target detection model, more detailed detection is performed on specific areas (such as hands) to identify specific hand behaviors (such as taking notes, playing with mobile phones, etc.). Combining posture information with target detection results, the target user behavior is divided into different categories, such as using a mobile phone, lying on the table, turning around, eating, passing things, reading and writing, standing up, looking at the blackboard, etc.

[0129] On this basis, the teaching behavior analysis can be conducted by combining the teaching media heat map and the classroom behavior of the target users. The teaching media heat map can reflect the target users' attention to each area in the teaching media. Through the teaching media heat map and the classroom behavior of the target users, the relationship between the target user behavior and the use of teaching media can be analyzed, and the changes in the classroom behavior of the target users at different stages of media use can be analyzed, such as the target user's gaze behavior when the teacher uses PPT, the target user's note-taking behavior when the teacher writes on the blackboard, etc. The relationship between behavior and teaching effect can also be analyzed, and the impact of different classroom behaviors on teaching effect can be analyzed by combining the target user's academic performance, participation and other data.

[0130] The method provided in the embodiment of the present invention performs teaching behavior analysis based on the teaching medium heat map and the classroom behavior of the target user, and can present classroom data in multiple dimensions, thereby helping teachers to better improve teaching effectiveness.

[0131] In one embodiment, the interactive control method based on eye movement trajectory analysis includes:

[0132] S1, obtains visual information of the teaching environment, which includes depth map and two-dimensional map.

[0133] S2, based on visual information, captures the target user's eye focus area on the teaching medium, and generates a teaching medium heat map based on the target user's eye focus area on the teaching medium.

[0134] S3, based on the teaching media heat map, conduct teaching behavior analysis. Specifically, by obtaining the heat map of all blackboard writings in the entire class, statistical analysis can be performed on the attention paid to different blackboard writing styles and blackboard writing content. For example, by analyzing the heat map of blackboard writing content, it can be clearly and quantitatively seen that the target users pay more attention to information such as geometric figures, conclusion sentences, core formulas, and circled content. At the same time, the heat analysis range of each page of blackboard writing can be analyzed and counted to understand the target users' attention to the layout of the blackboard and the style of content presentation.

[0135] By obtaining the heat map of teachers' courseware, we can also conduct statistical analysis on the attention paid to different courseware styles and courseware contents. We can clearly and quantitatively see that target users pay more attention to picture information and picture information with expressions. At the same time, by analyzing and counting the thermal analysis range of each page of courseware, we can understand the target users' attention to the layout of the courseware and the content presentation style.

[0136] By analyzing the heat maps of different teachers' blackboard writing and courseware, we can screen out excellent blackboard writing and courseware templates to promote the improvement of the blackboard writing and courseware level of teachers throughout the school, thereby improving the effectiveness of knowledge transfer and improving teaching quality.

[0137] S4, determines the movement trajectory of the teacher during the teaching process based on visual information, and generates a movement trajectory heat map based on the movement trajectory; based on the movement trajectory heat map, analyzes the teacher's teaching behavior.

[0138] S5, conducts behavior recognition on visual information to obtain the classroom behavior of the target user; conducts teaching behavior analysis based on the teaching medium heat map and the classroom behavior of the target user.

[0139] The embodiment of the present invention captures the target user's eye focus area on the screen during classroom teaching, and realizes a heat map of the teaching process such as courseware and blackboard writing after processing by AI algorithm, so as to help teachers analyze the target user's focus, promote the improvement of teaching design, improve the quality of courseware and blackboard writing, and thus improve the effectiveness of knowledge transfer; at the same time, it analyzes the teacher's movement trajectory and the target user's classroom status, presents classroom data in multiple dimensions, and helps teachers better improve teaching results.

[0140] The teaching machine provided by the present invention is described below. The teaching machine described below corresponds to the interactive control method based on eye movement trajectory analysis described above.

[0141] See also Figure 2 , Figure 2 The present invention provides a teaching machine according to an embodiment of the present invention. The teaching machine is equipped with a visual camera, and the teaching machine includes:

[0142] Monitoring module 210, for obtaining, based on the visual camera, the target user's eye movement track during the teaching video playback process, and the attention duration of each video in the teaching video;

[0143] The interactive display module 220 is used to obtain a first target video whose attention time is greater than or equal to a preset time, and display the first target video on the display interface of the teaching machine to obtain a second target video and a text prompt word; the text prompt word is input by the target user on the display interface, and the second target video includes multiple images, each image includes a target to be analyzed selected by the target user;

[0144] An acquisition module 230, configured to extract features from the second target video to obtain visual features of the second target video;

[0145] A feature extraction module 240 is used to extract features from the text prompt word to obtain text features of the text prompt word;

[0146] The prediction module 250 is used to input the visual features and the text features into a pre-trained large language model to obtain a description text of the target to be analyzed output by the large language model.

[0147] The teaching machine provided by the present invention allows the target user to select the target to be analyzed by himself, performs feature extraction on the second target video containing the target to be analyzed selected by the target user to obtain visual features, then performs feature extraction on the text prompt words of the target user to obtain text features, and uses a large language model to analyze the visual features and text features to obtain a description text of the target to be analyzed, thereby completing the analysis and description of specific targets in the video, focusing on the content that the target user is interested in, improving the learning enthusiasm of the target user, and further improving the learning efficiency of the target user.

[0148] See also Figure 3 , Figure 3 is a schematic diagram of the structure of an electronic device provided by the present invention. An embodiment of the present invention provides an electronic device 300, including a memory 310, a processor 320, and a computer program 311 stored in the memory 310 and executable on the processor 320. When the processor 320 executes the computer program 311, the following steps are implemented:

[0149] Acquiring, based on the visual camera, the target user's eye movement track during the teaching video playback process for the attention duration of each video in the teaching video;

[0150] A first target video having an attention duration greater than or equal to a preset duration is obtained, and the first target video is displayed on a display interface of the teaching machine to obtain a second target video and a text prompt word; the text prompt word is input by the target user on the display interface, and the second target video includes a plurality of images, each of which includes a target to be analyzed selected by the target user;

[0151] Performing feature extraction on the second target video to obtain visual features of the second target video;

[0152] Performing feature extraction on the text prompt word to obtain text features of the text prompt word;

[0153] The visual features and the text features are input into a pre-trained large language model to obtain a description text of the target to be analyzed output by the large language model.

[0154] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.

[0155] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An interactive control method based on eye movement trajectory analysis, characterized in that: Applied to a teaching machine, the teaching machine is equipped with a visual camera, and the interactive control method based on eye movement trajectory analysis includes: Acquiring, based on the visual camera, the target user's eye movement track during the teaching video playback process for the attention duration of each video in the teaching video; A first target video having an attention duration greater than or equal to a preset duration is obtained, and the first target video is displayed on a display interface of the teaching machine to obtain a second target video and a text prompt word; the text prompt word is input by the target user on the display interface, and the second target video includes a plurality of images, each of which includes a target to be analyzed selected by the target user; Performing feature extraction on the second target video to obtain visual features of the second target video; Performing feature extraction on the text prompt word to obtain text features of the text prompt word; The visual features and the text features are input into a pre-trained large language model to obtain a description text of the target to be analyzed output by the large language model.

2. The interactive control method based on eye movement trajectory analysis according to claim 1, characterized in that: The extracting features of the second target video to obtain visual features of the second target video includes: Determine a target mask, depth features, video history features, and encoding features of a first frame image; the first frame image is a first frame image of the second target video; The next frame image of the first frame image is used as the current frame image; Determining a depth feature of the current frame image based on a target mask of the current frame image and a frame image previous to the current frame image; Determine the encoding feature of the current frame image based on the depth feature of the current frame image and the video history feature of the previous frame image of the current frame image; Inputting the encoded features of the current frame image into a mask decoding network to obtain a target mask of the current frame image output by the mask decoding network; Determine the video history feature of the current frame image based on the coding feature of the current frame image and the video history feature of the previous frame image of the current frame image; Taking the next frame image of the current frame image as the current frame image, returning the target mask based on the current frame image and the previous frame image of the current frame image, and determining the depth feature of the current frame image, until the target mask, depth feature, video history feature and encoding feature of each image in the second target video are obtained; The video history features of the last frame image in the second target video are used as the visual features of the second target video.

3. The interactive control method based on eye movement trajectory analysis according to claim 2, characterized in that: The step of determining the target mask, depth features, video history features, and encoding features of the first frame image includes: Determining interaction information of the target user; Inputting the first frame image and the interaction information into a SAM model to obtain a target mask of the first frame image output by the SAM model; Performing splicing processing on the first frame image and the target mask of the first frame image to obtain a spliced ​​image; Based on a visual feature extraction network, feature extraction processing is performed on the spliced ​​image to obtain a depth feature of the first frame image and a video history feature of the first frame image; Extract the encoding features of the first frame image.

4. The interactive control method based on eye movement trajectory analysis according to claim 2, characterized in that: After obtaining the target mask, depth feature, video history feature and encoding feature of each image in the second target video, the method further includes: Based on the target mask of each image, target segmentation is performed on each image to obtain the target segmentation result of each image; The target segmentation result of each image is returned to the target user.

5. The interactive control method based on eye movement trajectory analysis according to claim 1, characterized in that: The specific steps of obtaining the second target video include: In response to the interaction information of the target user, determining the target to be analyzed in the first target video and the processing duration selected by the target user based on the interaction information; The second target video is determined based on the target to be analyzed in the first target video and the processing time selected by the target user.

6. The interactive control method based on eye movement trajectory analysis according to claim 1, characterized in that: The step of inputting the visual features and the text features into a pre-trained large language model to obtain a description text of the target to be analyzed output by the large language model includes: Performing dimension adjustment processing on the visual feature so that the visual feature and the text feature have the same feature dimension; The visual features that have been dimensionally adjusted and the text features are input into a pre-trained large language model to obtain a description text of the target to be analyzed output by the large language model.

7. The interactive control method based on eye movement trajectory analysis according to any one of claims 1 to 6, characterized in that: The interactive control method based on eye movement trajectory analysis also includes: Acquire visual information of the teaching environment, wherein the visual information includes a depth map and a two-dimensional map; Capturing the area of ​​interest of the target user's eyes on the teaching medium based on the visual information, and generating a teaching medium heat map based on the area of ​​interest of the target user's eyes on the teaching medium; Based on the teaching medium heat map, the target users' attention to the style, content, and layout of the teaching medium is analyzed to obtain analysis results.

8. The interactive control method based on eye movement trajectory analysis according to claim 7, characterized in that: The interactive control method based on eye movement trajectory analysis also includes: Determine the movement trajectory of the teacher during the teaching process based on the visual information, and generate a movement trajectory heat map based on the movement trajectory; Based on the movement trajectory heat map, the teacher's teaching behavior is analyzed.

9. The interactive control method based on eye movement trajectory analysis according to claim 7, characterized in that: The interactive control method based on eye movement trajectory analysis also includes: Performing behavior recognition on the visual information to obtain classroom behavior of the target user; Based on the teaching medium heat map and the classroom behavior of the target user, teaching behavior analysis is performed.

10. A teaching machine, characterized in that: The teaching machine is equipped with a visual camera, and the teaching machine comprises: A monitoring module, for obtaining, based on the visual camera, the target user's eye movement track during the teaching video playback process, and the attention duration of each video in the teaching video; The interactive display module is used to obtain a first target video whose attention time is greater than or equal to a preset time, and display the first target video on the display interface of the teaching machine to obtain a second target video and a text prompt word; the text prompt word is input by the target user on the display interface, and the second target video includes multiple images, each image includes a target to be analyzed selected by the target user; An acquisition module, used for performing feature extraction on the second target video to obtain visual features of the second target video; A feature extraction module, used to extract features of the text prompt word to obtain text features of the text prompt word; The prediction module is used to input the visual features and the text features into a pre-trained large language model to obtain a description text of the target to be analyzed output by the large language model.

Images