Interactive control method and teaching machine based on eye movement trajectory analysis

Abstract

The present invention discloses an interactive control method based on eye movement trajectory analysis and its teaching machine, comprising: obtaining the target user's eye movement trajectory for each video in the teaching video during the teaching video playback process based on a visual camera; obtaining a first target video with an attention duration greater than or equal to a preset duration, and displaying the first target video on the display interface of the teaching machine to obtain a second target video and a text prompt word; performing feature extraction on the second target video to obtain the visual features of the second target video; performing feature extraction on the text prompt word to obtain the text features of the text prompt word; inputting the visual features and text features into a pre-trained large language model to obtain a description text of the target to be analyzed output by the large language model. The present invention completes the analysis and description of specific targets in the video, focuses on the content that the target user is interested in, and improves the target user's learning enthusiasm and learning efficiency.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to an interactive control method based on eye movement trajectory analysis and a teaching machine thereof. Background Art

[0002] With the development of artificial intelligence (AI) technology, the use of large language models for interactive teaching has gradually become popular. In some related technologies, a visual encoder can be used to directly extract a global image feature sequence from the original video. This global image feature sequence is then mapped into the semantic space of a large language model, resulting in the output of a video description.

[0003] However, existing teaching interaction methods can only extract the global image features of the video, but it is difficult to extract the local image features of the video. Therefore, in the scenario where only specific targets in the video need to be described, the existing teaching interaction methods are difficult to complete the analysis and description of specific targets in the video, and cannot focus on the content that the target users are interested in, resulting in low students' learning enthusiasm, which affects students' learning efficiency. Summary of the Invention

[0004] The present invention provides an interactive control method based on eye movement trajectory analysis and a teaching machine thereof, which are used to solve the existing technical problems of difficulty in completing the analysis and description of specific targets in videos and inability to focus on the content that the target users are interested in, thereby improving the learning enthusiasm and efficiency of the target users.

[0005] In a first aspect, the present invention provides an interactive control method based on eye movement trajectory analysis, which is applied to a teaching machine equipped with a visual camera. The interactive control method based on eye movement trajectory analysis includes:

[0006] Acquiring, based on the visual camera, the target user's eye movement trajectory during the playback of the teaching video for the duration of attention paid to each video in the teaching video;

[0007] Obtaining a first target video whose viewing duration is greater than or equal to a preset duration, and displaying the first target video on a display interface of the teaching machine to obtain a second target video and a text prompt word; the text prompt word is input by the target user on the display interface, and the second target video includes multiple images, each image containing a target to be analyzed selected by the target user;

[0008] performing feature extraction on the second target video to obtain visual features of the second target video;

[0009] Performing feature extraction on the text prompt word to obtain text features of the text prompt word;

[0010] The visual features and the text features are input into a pre-trained large language model to obtain a description text of the target to be analyzed output by the large language model.

[0011] In a second aspect, the present invention provides a teaching machine, wherein the teaching machine is equipped with a visual camera, and the teaching machine comprises:

[0012] A monitoring module, configured to obtain, based on the visual camera, the target user's eye movement trajectory and the duration of attention paid to each video in the teaching video during the playing of the teaching video;

[0013] An interactive display module is configured to obtain a first target video having an attention duration greater than or equal to a preset duration, and display the first target video on a display interface of the teaching machine to obtain a second target video and a text prompt word; the text prompt word is input by the target user on the display interface, and the second target video includes multiple images, each image containing a target to be analyzed selected by the target user;

[0014] an acquisition module, configured to perform feature extraction on the second target video to obtain visual features of the second target video;

[0015] A feature extraction module is used to extract features of the text prompt word to obtain text features of the text prompt word;

[0016] A prediction module is used to input the visual features and the text features into a pre-trained large language model to obtain a descriptive text of the target to be analyzed output by the large language model.

[0017] In a third aspect, the present invention also provides a teaching machine, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the interactive control method based on eye movement trajectory analysis as described in the first aspect is implemented.

[0018] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the interactive control method based on eye movement trajectory analysis as described in the first aspect.

[0019] In a fifth aspect, the present invention further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the interactive control method based on eye movement trajectory analysis as described in the first aspect.

[0020] The interactive control method based on eye movement trajectory analysis provided by the present invention allows a target user to select a target to be analyzed on his / her own, performs feature extraction on a second target video containing the target to be analyzed selected by the target user to obtain visual features, then performs feature extraction on a text prompt word of the target user to obtain text features, and uses a large language model to analyze the visual features and text features to obtain a description text of the target to be analyzed, thereby completing the analysis and description of a specific target in the video, focusing on the content that the target user is interested in, improving the target user's learning enthusiasm, and thereby improving the target user's learning efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the present invention or related technologies, the following is a brief introduction to the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0022] Figure 1 1 is a flow chart of an interactive control method based on eye movement trajectory analysis provided by the present invention;

[0023] Figure 2 It is a structural diagram of the teaching machine provided by the present invention;

[0024] Figure 3 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0025] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making any creative efforts shall fall within the scope of protection of the present invention.

[0026] In the description of the present invention, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, a feature specified as "first" or "second" may explicitly or implicitly include one or more of the specified features. In the description of the present invention, "plurality" means two or more, unless otherwise specifically defined.

[0027] In the description of the present invention, the term "for example" is used to mean "used as an example, illustration or illustration". Any embodiment of the present invention described as "for example" is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is given to enable any person skilled in the art to implement and use the present invention. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art can recognize that the present invention can be implemented without using these specific details. In other examples, well-known structures and processes are not elaborated in detail to avoid obscuring the description of the present invention with unnecessary details. Therefore, the present invention is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed herein.

[0028] See Figure 1 , Figure 1 : This is a flow chart of an interactive control method based on eye movement trajectory analysis provided by the present invention. The interactive control method based on eye movement trajectory analysis is applied to a teaching machine equipped with a visual camera. The interactive control method based on eye movement trajectory analysis includes:

[0029] Step 10: obtaining the target user's eye movement trajectory and the attention duration of each video in the teaching video during the teaching video playback process based on the visual camera.

[0030] The present invention is based on a teaching machine playing video tutorials, where a target user (any student) follows the video while the machine is in front of the machine. Therefore, while the machine is playing the tutorial video, a visual camera is used to capture the target user's eye movement trajectory, which measures the duration of attention spent on each video. This duration is the duration of time the target user remains focused on the video.

[0031] Step 20: Obtain a first target video with an attention duration greater than or equal to a preset duration, and display the first target video on a display interface of the teaching machine to obtain a second target video and a text prompt word.

[0032] Furthermore, the teaching machine determines whether the target user's eye movement track for each video in the teaching video is greater than or equal to a preset time, wherein the preset time is set according to actual conditions, such as 2 seconds or 3 seconds.

[0033] If the duration of attention is determined to be greater than or equal to a preset duration, the teaching machine identifies the video corresponding to that duration as the first target video and displays the first target video on the teaching machine's display interface. The target user can then select a target to be analyzed from the first target video and enter a text prompt in the display interface. After the target user selects the target to be analyzed, the teaching machine obtains a second target video based on the target to be analyzed. The second target video includes multiple images, each of which contains the target to be analyzed selected by the target user.

[0034] Specifically, the teaching machine is embedded with a target user interaction processing module, a visual feature encoding module, a target mask decoding module, and a graphic feature fusion module. Among them, the target user interaction processing module can obtain the second target video and the target user's text prompt word.

[0035] Step 30: extract features from the second target video to obtain visual features of the second target video.

[0036] Step 40: extract features of the text prompt words to obtain text features of the text prompt words.

[0037] Specifically, the target user's text prompt word prompt is input into the text encoding network to obtain the text feature RT (i.e., text sequence feature) generated by the text encoding network. The dimension of RT is [textlength, d], where textlength is the length of the text word (token) and d is the dimension of each word.

[0038] Step 50: Input the visual features and text features into the pre-trained large language model to obtain the descriptive text of the target to be analyzed output by the large language model.

[0039] The embodiment of the present invention allows the target user to select the target to be analyzed on his / her own, performs feature extraction on the second target video containing the target to be analyzed selected by the target user to obtain visual features, then performs feature extraction on the text prompt words of the target user to obtain text features, and uses a large language model to analyze the visual features and text features to obtain a description text of the target to be analyzed, thereby completing the analysis and description of specific targets in the video, focusing on the content that the target user is interested in, improving the learning enthusiasm of the target user, and thereby improving the learning efficiency of the target user.

[0040] In one embodiment, performing feature extraction on the second target video to obtain visual features of the second target video includes:

[0041] Determine the target mask, depth features, video history features, and encoding features of a first frame image; the first frame image is the first frame image of the second target video;

[0042] The next frame image of the first frame image is used as the current frame image;

[0043] Determine the depth feature of the current frame image based on the target mask of the current frame image and the previous frame image of the current frame image;

[0044] Determine the encoding features of the current frame image based on the depth features of the current frame image and the video history features of the previous frame image of the current frame image;

[0045] Input the encoded features of the current frame image into the mask decoding network to obtain the target mask of the current frame image output by the mask decoding network;

[0046] Determining the video history features of the current frame image based on the coding features of the current frame image and the video history features of the previous frame image of the current frame image;

[0047] The step of taking the next frame image of the current frame image as the current frame image, returning a target mask based on the current frame image and the frame image before the current frame image, and determining the depth feature of the current frame image, until the target mask, depth feature, video history feature and encoding feature of each image in the second target video are obtained;

[0048] The video history features of the last frame image in the second target video are used as the visual features of the second target video.

[0049] In one embodiment, the target user interaction processing module is used to convert the target user's interactive operation of selecting a target into a signal executable by the system. The core of this module is the SAM model (SegmentAnything Model).

[0050] Specifically, the teaching machine can play a first target video, which may contain multiple targets, such as people, objects, animals, plants, specific areas, etc.; while the target user is watching the video, the target user can pause the video at any time, and the screen will stay on the image of the frame at that moment. The target user can interact with the system, select a target of interest in the image as the target to be analyzed, and determine the processing time, thereby generating interactive information; the processing time refers to the length of the video that the target user wants the system to analyze.

[0051] Furthermore, the teaching machine responds to the target user's interaction information and can determine the target to be analyzed in the first target video and the processing time selected by the target user based on the interaction information, and determine the second target video based on the target to be analyzed in the first target video and the processing time selected by the target user.

[0052] For example, the first target video is half an hour long. The target user can select the target to be analyzed on the image at the 10th minute of the first target video and determine the processing time to be 15 minutes. That is, the target user wants the system to analyze a video of 15 minutes in length. The teaching machine can then extract the second target video based on the video clips from the 10th to the 25th minute of the first target video.

[0053] Optionally, the target user may select the target to be analyzed by left-clicking the mouse or selecting the target by a frame. It should be noted that, in this embodiment, the image on which the target user selects the target to be analyzed is used as the first frame image I0 in the second target video. The first frame image I0 is the first frame image of the second target video.

[0054] Furthermore, the target user's interaction information is used as a prompt, and the first frame image I0 and the interaction information are input into the SAM model of the target user interaction processing module to obtain the target mask mask0 of the first frame image I0 output by the SAM model.

[0055] The visual feature encoding module is used to generate visual features. Visual features should contain the following two types of information: first, the mask information of the target to be analyzed is integrated to enable the description of the specified target; second, the information of all image frames is included, so that the model has the ability to describe the entire video. Leveraging the idea of ​​semi-supervised tasks, the visual feature encoding module can store historical image features and combine historical results with the current image to predict the current result. The specific approach is: store the historical features of the video frame in memory, fuse them with the previous target mask result and the current image features, update the historical features of the video and the current image features, and then decode the current image features to obtain the target mask result of the current image. In this process, the historical features of the video are integrated with each frame of the image and its target mask information, which can be used to understand and describe specific targets in the video.

[0056] Furthermore, the visual feature encoding module performs splicing processing on the first frame image I0 and the target mask mask0 of the first frame image: the first frame image I0 and the target mask mask0 of the first frame image are spliced ​​in dimension to obtain a spliced ​​image; the spliced ​​image is input into a visual feature extraction network (such as a visual feature encoder ViT) for processing, and the visual feature extraction network can perform feature extraction processing on the spliced ​​image to obtain a depth feature R0 of the first frame image, and use the depth feature R0 of the first frame image as the video history feature RH0 of the first frame image; extract the encoding feature RN0 of the first frame image, and store the above-extracted feature in the server memory.

[0057] Furthermore, the next frame image of the first frame image is used as the current frame image.

[0058] In one embodiment, for each current frame image I n , can be based on the current frame image I n With the current frame image I n The previous frame image I n-1 The target mask mask n-1 , determine the current frame image I n The deep feature R n .

[0059] Specifically, the current frame image I n With the current frame image I n The previous frame image I n-1 The target mask mask n-1 The spliced ​​images are spliced ​​in the dimension, and the visual feature extraction network is used to extract features from the spliced ​​images to obtain the current frame image I n The deep feature R n .

[0060] Furthermore, based on the current frame image I n The deep feature R n and the current frame image I n The previous frame image I n-1 Video history features RH n-1 , determine the current frame image I n The encoding feature RN n .

[0061] Specifically, the current frame image I n The deep feature R n As the index vector Q, the previous frame image I n-1 Video history features RH n-1 As the key vector K and value vector V, the three are input into the encoding feature update network to obtain the current frame image I output by the encoding feature update network n The encoding feature RN n , current frame image I n The encoding feature RN n Fusion of the previous frame image I n-1 Video history features.

[0062] Furthermore, the current frame image I n The encoding feature RN n Input to the mask decoding network to obtain the target mask of the current frame image output by the mask decoding network n .

[0063] Specifically, the current frame image I n The encoding feature RN n, as the mask decoding feature, is input into the mask decoding network of the target mask decoding module (the mask decoding network can adopt structures such as deconvolution and upsampling), and the target mask mask of the current frame image output by the mask decoding network is obtained. n .

[0064] The target mask of the current frame image n Can be used to generate the next frame image I n+1 The target mask mask n+1 , and then the masks {mask1,...,mask n}, and acts on the original video frame image to realize the visualization of the target segmentation result and provide the visualization result to the target user.

[0065] Furthermore, based on the current frame image I n The encoding feature RN n and the current frame image I n The previous frame image I n-1 Video history features RH n-1 , the current frame image I can be determined n Video history features RH n .

[0066] Specifically, the previous frame image I n-1 Video history features RH n-1 As the index vector Q, the current frame image I n The encoding feature RN n As the key vector K and value vector V, the three are input into the historical feature update network to obtain the current frame image I output by the historical feature update network n Video history features RH n , current frame image I n Video history features RH n Fusion of the current frame image I n The encoding feature RN n .

[0067] Furthermore, the next frame image of the current frame image is used as the current frame image, and the target mask based on the current frame image and the previous frame image of the current frame image is returned to determine the depth feature of the current frame image, until the target mask, depth feature, video history feature and encoding feature of each image in the second target video are obtained.

[0068] It can be understood that for the second target video of length n (n is determined by the duration and video frame rate selected by the target user), its encoding features can be expressed as {RN0, RN1, ..., RN n}, which combines the information of historical target mask, historical frame and current frame, and can be used to predict the target mask of the current frame, while the video history features {RH0, RH1, ..., RH n} contains both the target mask information and the video history information of the previous frame. The final video history feature RH n Contains historical information of the entire video.

[0069] Furthermore, the video history features of the last frame image in the second target video are used as the visual features of the second target video. In this embodiment, assuming that the length of the second target video is n, the last frame image I in the second target video is n The video history feature is RH n , which is used as the visual feature of the second target video.

[0070] The embodiment of the present invention provides a target user interaction function, in which the target user selects the target to be analyzed, and uses the SAM model as an interaction bridge to convert the target user's interactive operation into the segmentation of a visually specific target, thereby achieving the selection of the target of interest.

[0071] In one embodiment, the target mask, depth features, video history features and coding features of the first frame image are determined, including: determining the interaction information of the target user; inputting the first frame image and the interaction information into a SAM model to obtain the target mask of the first frame image output by the SAM model; splicing the first frame image and the target mask of the first frame image to obtain a spliced ​​image; performing feature extraction on the spliced ​​image based on a visual feature extraction network to obtain the depth features of the first frame image and the video history features of the first frame image; and extracting the coding features of the first frame image.

[0072] Specifically, the teaching machine can play a first target video (i.e., the original video), which may contain multiple targets, which can be people, objects, animals, plants, specific areas, etc.; while the target user is watching the video, the target user can pause the video at any time, and the screen will stay on the image of the frame at that moment. The target user can interact with the system, select a target of interest in the image as the target to be analyzed, and determine the processing time, thereby generating interactive information; wherein, the processing time refers to the length of the video that the target user wants the system to analyze.

[0073] Furthermore, the teaching machine responds to the target user's interaction information and can determine the target to be analyzed in the first target video and the processing time selected by the target user based on the interaction information, and determine the second target video based on the target to be analyzed in the first target video and the processing time selected by the target user.

[0074] It should be noted that, in this embodiment, the target user selects the target to be analyzed on which image, and the frame image is used as the first frame image I0 in the second target video. The first frame image I0 is the first frame image of the second target video.

[0075] Furthermore, the target user's interaction information is used as a prompt, and the first frame image I0 and the interaction information are input into the SAM model of the target user interaction processing module to obtain the target mask mask0 of the first frame image I0 output by the SAM model.

[0076] In one embodiment, after obtaining the target mask, depth features, video history features and encoding features of each image in the second target video, it also includes: based on the target mask of each image, performing target segmentation on each image respectively to obtain the target segmentation result of each image; and returning the target segmentation result of each image to the target user.

[0077] After obtaining the target mask of each image, target segmentation is performed on each image based on the target mask of each image to obtain the target segmentation result of each image; the target segmentation result of each image is returned to the target user to realize the visualization of the target segmentation result, which is convenient for the target user to intuitively experience.

[0078] In one embodiment, obtaining a second target video and a text prompt word of a target user includes: responding to interaction information of the target user, determining a target to be analyzed in the first target video and a processing time selected by the target user based on the interaction information; determining a second target video based on the target to be analyzed in the first target video and a processing time selected by the target user; and obtaining a text prompt word of the target user.

[0079] It should be noted that if the target user does not select the target to be analyzed, the system may not perform the segmentation processing operation of the target user interaction. At this time, the dimension of the mask in the encoding feature update network is set to 0, and the description text output by the system will not be for a specific target in the video, but for the global image in the video; if the target user only selects the target to be analyzed but does not determine the processing time, the system only processes the current frame image and outputs the specific target description result of the current frame image; if the target user neither selects the target to be analyzed nor determines the processing time, the system outputs the global description text of the current frame image.

[0080] The embodiment of the present invention can adjust the system function in a timely manner according to the interactive settings such as the target and duration selected by the target user, and can realize the description of pictures / videos and local / global information.

[0081] In one embodiment, visual features and text features are input into a pre-trained large language model to obtain a descriptive text of the target to be analyzed output by the large language model, including: dimensionally adjusting the visual features so that the visual features and the text features have the same feature dimensions; inputting the dimensionally adjusted visual features and text features into the pre-trained large language model to obtain a descriptive text of the target to be analyzed output by the large language model.

[0082] In one embodiment, the core of the image-text feature fusion module is a pre-trained large language model (LLM), which integrates visual features and text features and inputs them into the large language model. The large language model can perform causal reasoning based on visual features and text features and output a descriptive text of the target to be analyzed.

[0083] In one embodiment, the target user's text prompt word prompt is input into the text encoding network to obtain the text feature R generated by the text encoding network. T , R T The dimension is [textlength, d], where textlength is the length of the text word (token) and d is the dimension of each word.

[0084] In this embodiment, assuming that the length of the second target video is n, the last frame image I in the second target video is n The video history feature is RH n , which is used as the visual feature of the second target video, that is, the visual feature is RH n It is understood that in order to make the visual feature RH n With text feature R T With the same feature dimension, the MLP structure (generally using several layers of linear layers) can be used to adjust the visual feature RH n Dimension, visual feature RH after dimension adjustment n The dimension is [M×M, d], where M=ImageSize (image size) / PatchSize (ViT partition image window size).

[0085] It should be noted that if the video history feature RH is not used n Instead of using the existing method as a visual feature, the depth feature {R0, R1, ..., R n} as visual features, the dimension of the visual features will increase to [n×M×M, d], which will increase the system's computational requirements. In addition, for long videos, the dimension of the visual features may exceed the maximum processing token length of the large language model, thus affecting the generation of description text.

[0086] Furthermore, the visual features and text features that have undergone dimension adjustment are spliced ​​in the column dimension, combined with rotation position encoding, and input into the pre-trained large language model to obtain the descriptive text of the target to be analyzed output by the large language model, thereby realizing the description of the selected target in the video.

[0087] In one embodiment, for an application scenario where video teaching is not played through a teaching machine, that is, a teacher teaching on-site scenario, the interactive control method based on eye movement trajectory analysis in an embodiment of the present invention includes:

[0088] Obtain visual information of the teaching environment, including depth maps and two-dimensional maps;

[0089] Capturing the target user's eye focus area on the teaching medium based on visual information, and generating a teaching medium heat map based on the target user's eye focus area on the teaching medium;

[0090] Based on the teaching media heat map, the target users' attention to the style, content, and layout of the teaching media is analyzed to obtain the analysis results.

[0091] Specifically, the teaching environment can be a facility environment that can provide teaching activities. The teaching environment can be an indoor environment or an outdoor environment, which is not specifically limited in the embodiment of the present invention. For example, the teaching environment can be a classroom or an outdoor classroom.

[0092] Taking the classroom as an example, the teaching machine can be installed in an appropriate location in the classroom to ensure that the facial areas of all target users are covered. The teaching machine is equipped with a visual camera, which is installed above the teaching machine to simultaneously obtain depth maps and 2D images.

[0093] A visual camera can capture three-dimensional information about objects in a scene, including depth maps and point clouds. This is crucial for capturing the target user's eye position, as 3D eye position information more accurately reflects their gaze than a 2D image. Choosing an appropriate visual camera ensures high imaging accuracy and fast processing speed to meet the requirements for capturing the target user's eye position in real time.

[0094] Eye tracking is a biometric technology that infers gaze direction by detecting eye movement. In eye tracking or facial recognition applications, depth maps can serve as auxiliary information, helping eye localization algorithms more accurately locate eye positions. By combining depth maps with 2D image information, the algorithm can more effectively filter out background noise and interference, improving the accuracy and robustness of eye localization. In facial images of a target user captured by a visual camera, eye tracking algorithms can locate the eye position and further calculate the gaze point.

[0095] Here, teaching media refers to the medium used to transmit teaching information during the teaching process. Specifically, in the classroom teaching process, teaching media can include, for example, blackboard writing, electronic courseware, specimens, experimental equipment, etc. The blackboard writing here can be electronic blackboard writing or traditional chalk blackboard writing, which is not specifically limited in the embodiments of the present invention.

[0096] The target user's eye focus area on the teaching medium refers to the area on the teaching medium where the target user's gaze is projected. For example, on a certain page of electronic courseware, target user 1's eye focus area is the geometric image on the electronic courseware, target user 2's eye focus area is the formula on the electronic courseware, and so on. This allows us to capture the eye focus areas of each target user on the teaching medium. Furthermore, we can also capture the duration of each target user's attention on a specific area of ​​the teaching medium. For example, target user 3 focused on area 1 on the electronic courseware for 10 seconds and on area 2 for 20 seconds.

[0097] Based on the target user's eye focus areas on the teaching media, a teaching media heat map is generated. The teaching media heat map reflects the target user's attention level in each area of ​​the teaching media.

[0098] The generation of a teaching media heat map can be achieved through data analysis methods. For example, based on the number of target users following each area and the length of time each target user pays attention, a comprehensive analysis is conducted to obtain analytical data, which is then mapped to the teaching media to generate a teaching media heat map. The color depth in the teaching media heat map reflects the target user's level of attention to that area. For example, the darker the color, the higher the level of attention; conversely, the lighter the color, the lower the level of attention. For example, in the teaching media heat map, the areas that the target user pays close attention to are marked in red, and other areas of attention are marked with gradient colors to reflect the level of attention. This method can quantify and intuitively reflect the target user's attention to the teaching media.

[0099] The teaching medium heat map can be obtained through a pre-trained heat map generation model. The depth map and two-dimensional map in the visual information are input into the pre-trained heat map generation model to obtain the teaching medium heat map output by the model.

[0100] In one embodiment, the teaching medium heat map can intuitively reflect the target user's eye focus area on the teaching medium. Therefore, teaching behavior analysis can be performed based on the teaching medium heat map. The goal of the interactive control based on eye movement trajectory analysis here is to evaluate the target user's attention, understanding or interest points on different teaching media.

[0101] For example, heat maps of different teaching media can be generated based on different blackboard writing content and styles to analyze target users' activity and attention on different teaching media. By analyzing the high- and low-heat areas in the heat map, the parts of the teaching media that target users are most and least interested in can be identified. By observing the differences in target users' feedback on different teaching media over different time periods, specific learning behavior patterns can be analyzed.

[0102] Based on this, the analysis results can be used to optimize teaching media. Based on the heat map analysis results, the content and format of the teaching media can be adjusted to increase target users' learning interest and engagement. For high-profile areas, the content can be further enriched and interactive elements can be added. For low-profile areas, the reasons need to be identified (e.g., excessively difficult content, monotonous format, etc.) and appropriate improvements can be made.

[0103] This embodiment of the present invention generates a heat map of the teaching medium based on the target user's eye focus on the teaching medium. This heat map can more intuitively display the target user's attention. Furthermore, interaction based on the heat map can be used to understand the target user's attention distribution in class, automatically analyze feedback information, and then adjust teaching methods and strategies based on the analysis results, thereby improving teaching effectiveness.

[0104] In one embodiment, the style of an instructional medium refers to the unique visual, auditory, or interactive characteristics of a teaching blackboard or courseware when presenting information. These characteristics collectively constitute the overall atmosphere and experience perceived by the target user when engaging with the instructional medium. Examples of instructional medium styles include traditional styles (e.g., blackboard writing), multimedia styles (e.g., multimedia courseware that combines multiple media elements such as text, images, audio, and video), and gamification styles.

[0105] The content of instructional media refers to the specific knowledge, information, skills, or experiences conveyed and presented to target users through instructional media. Examples of instructional media content include subject matter knowledge, such as geometric figures, mathematical formulas, conclusions, and circled content.

[0106] The layout of teaching media refers to the spatial arrangement and organization of various elements (such as text, images, charts, and buttons) when presenting information on a blackboard or courseware. A good layout can enhance the viewing experience of the target user and help them better understand and retain the information.

[0107] By displaying teaching media of different styles, contents and layouts, a corresponding teaching media heat map can be obtained. On this basis, the target users' attention to the style, content and layout of the teaching media can be analyzed to obtain the analysis results.

[0108] For example, a heat map clearly shows that target users are likely to show greater interest in teaching media styles that are colorful and feature both illustrations and text, and are particularly interested in images that convey a sense of expression. Areas explaining key concepts or formulas have higher heat intensities, indicating that these are the areas where target users focus their attention and learning. Teaching media with a well-organized layout and clear information hierarchy are more likely to attract target users' attention. For example, teaching media using a "general-specific-general" or "problem-solution" layout often have a more even heat distribution. Appropriate use of elements such as titles, subtitles, and images also contributes to a more engaging layout. These elements often appear as areas of higher heat intensity in a heat map.

[0109] The embodiment of the present invention analyzes the target users' attention to the style, content, and layout of the teaching media based on the teaching media heat map, and obtains analysis results, which can enable teachers to better meet the learning needs of target users when producing teaching media, thereby improving teaching effectiveness and the learning experience of target users.

[0110] In one embodiment, based on the teaching medium heat map, the target users' attention to the style, content, and layout of the teaching medium is analyzed to obtain the analysis results, and then the following steps are further included:

[0111] Based on the analysis results of each teaching medium, a target teaching medium is screened from each teaching medium, and / or the teaching medium is iteratively optimized.

[0112] Specifically, the target teaching medium can be one that has received high attention in terms of style, content, or layout. Each teaching medium can be tailored to a specific teacher, with different styles, content, and layouts, or it can be a teaching medium optimized from a previous iteration. For example, a threshold for attention can be pre-set, and teaching media with attention exceeding that threshold can be selected as target teaching media. This can improve the quality of teaching media produced by teachers across the entire school, thereby enhancing the effectiveness of knowledge transfer and improving teaching quality.

[0113] For iterative optimization of teaching media, we can regularly analyze the teaching media after the last optimization to understand the target users' attention level and usage experience for the teaching media, implement the redesigned teaching media in actual teaching, and obtain analysis results through the teaching media heat map, so as to achieve continuous iterative optimization of teaching media.

[0114] In addition, the analysis results of each teaching medium can provide the direction of iterative optimization. If any of the style, content, and layout of the teaching medium has a low level of attention, iterative optimization can be carried out starting from the dimension with lower attention.

[0115] In one embodiment, the interactive control method based on eye movement analysis further includes:

[0116] Determine the teacher's movement trajectory during the teaching process based on visual information, and generate a movement trajectory heat map based on the movement trajectory;

[0117] Analyze teachers' teaching behaviors based on the movement trajectory heat map.

[0118] Specifically, the teacher's location information is extracted from visual information. Image processing techniques, such as feature point detection and tracking algorithms, are then used to continuously track the teacher's location, forming a complete movement trajectory. The tracked data can also be cleaned to remove noise and outliers, ensuring the accuracy and continuity of the movement trajectory.

[0119] Then select a suitable data visualization tool, such as ArcGIS, Tableau, QGIS or matplotlib and seaborn libraries in Python, and import the cleaned teacher movement trajectory data into the visualization tool.

[0120] Set the heat map parameters according to the classroom layout, such as color mapping range, heat point size, and transparency. Based on the set parameters, generate a heat map of the teacher's movement trajectory, also known as the movement trajectory heat map. The movement trajectory heat map uses the depth of color to reflect the teacher's movement frequency and residence time in different areas.

[0121] On this basis, we analyze teachers' teaching behaviors based on the movement trajectory heat map. Through the movement trajectory heat map, we can intuitively see the teacher's movement patterns in the classroom, such as frequently visited areas and rarely visited areas.

[0122] The teacher's movement trajectory can be used to analyze the teacher's teaching style. For example, if the teacher moves frequently in the front and middle of the classroom, it may indicate that they focus on face-to-face communication with target users and blackboard demonstrations. If the teacher spends more time in the back or corner of the classroom, it may be providing individual tutoring or observing target users.

[0123] By observing the hot and cold spots in the movement trajectory heat map, you can assess classroom interaction. Hot spots may indicate active and highly engaged target users, while cold spots may indicate low engagement or blind spots in the teaching process.

[0124] Based on the analysis results, targeted improvement suggestions can be made. For example, to address blind spots in teaching, teachers can adjust teaching strategies and increase attention to those areas. To address low engagement among target users, teachers can design more interactive sessions to increase their learning interest and engagement.

[0125] In one embodiment, the interactive control method based on eye movement analysis further includes:

[0126] Perform behavioral recognition on visual information to obtain the classroom behavior of target users;

[0127] Conduct teaching behavior analysis based on the teaching media heat map and the classroom behavior of target users.

[0128] Specifically, the classroom behavior of the target user can be obtained through behavior recognition technology, and behavior recognition can be obtained through human key point detection and target detection. For example, algorithms such as OpenPose are used to extract human key points, such as head, hands, body, etc., as posture information. These key point information can reflect the different postures of the target user, such as sitting upright, raising hands, sideways, lowering the head, etc. Using the target detection model, more detailed detection is performed on specific areas (such as hands) to identify specific hand behaviors (such as taking notes, playing with mobile phones, etc.). Combining posture information and target detection results, the target user behavior is divided into different categories, such as using a mobile phone, lying on the table, turning around, eating, passing things, reading and writing, standing up, looking at the blackboard, etc.

[0129] On this basis, we can combine the teaching media heat map with the target users' classroom behavior to conduct teaching behavior analysis. The teaching media heat map can reflect the target users' attention to various areas of the teaching media. By combining the teaching media heat map and the target users' classroom behavior, we can analyze the relationship between target user behavior and teaching media usage, and analyze changes in target users' classroom behavior at different stages of media usage, such as their gaze behavior when the teacher is using PowerPoint, or their note-taking behavior when the teacher is writing on the blackboard. We can also analyze the relationship between behavior and teaching effectiveness. By combining data such as the target users' academic performance and engagement, we can analyze the impact of different classroom behaviors on teaching effectiveness.

[0130] The method provided in the embodiment of the present invention performs teaching behavior analysis based on the teaching medium heat map and the classroom behavior of the target users, and can present classroom data in multiple dimensions, thereby helping teachers to better improve teaching effectiveness.

[0131] In one embodiment, the interactive control method based on eye movement analysis includes:

[0132] S1, obtains visual information of the teaching environment, which includes depth map and two-dimensional map.

[0133] S2, based on visual information, captures the target user's eye focus area on the teaching medium, and generates a teaching medium heat map based on the target user's eye focus area on the teaching medium.

[0134] S3 conducts teaching behavior analysis based on teaching media heat maps. Specifically, by obtaining heat maps of all blackboard writing throughout the class, statistical analysis can be performed on the attention paid to different blackboard writing styles and content. For example, heat map analysis of blackboard writing content can clearly and quantitatively show that target users pay more attention to information such as geometric figures, conclusion statements, core formulas, and circled content. Simultaneously, the heat analysis range of each page of blackboard writing can be analyzed to understand the target users' attention to the layout and content presentation style of the blackboard writing.

[0135] By obtaining the heat map of teacher's courseware, we can also conduct statistical analysis on the attention paid to different courseware styles and courseware content. We can clearly and quantitatively see that target users pay more attention to picture information and picture information with expressions. At the same time, by analyzing and counting the heat analysis range of each page of courseware, we can understand the target users' attention to the layout of the courseware and the content presentation style.

[0136] By analyzing the heat maps of different teachers' blackboard writing and courseware, we can screen out excellent blackboard writing and courseware templates, promote the improvement of the blackboard writing and courseware level of teachers throughout the school, thereby improving the effectiveness of knowledge transfer and improving teaching quality.

[0137] S4, based on visual information, determines the teacher's movement trajectory during the teaching process, and generates a movement trajectory heat map based on the movement trajectory; based on the movement trajectory heat map, analyzes the teacher's teaching behavior.

[0138] S5, conducts behavioral recognition on visual information to obtain the classroom behavior of the target user; conducts teaching behavior analysis based on the teaching medium heat map and the classroom behavior of the target user.

[0139] The embodiment of the present invention captures the target user's eye focus area on the screen during classroom teaching, and processes it through an AI algorithm to achieve a heat map of the teaching process such as courseware and blackboard writing, thereby helping teachers analyze the target user's focus, promote the improvement of teaching design, improve the quality of courseware and blackboard writing, and thus improve the effectiveness of knowledge transfer; at the same time, it analyzes the teacher's movement trajectory and the target user's classroom status, presents classroom data in multiple dimensions, and helps teachers better improve teaching effectiveness.

[0140] The teaching machine provided by the present invention is described below. The teaching machine described below corresponds to the interactive control method based on eye movement trajectory analysis described above.

[0141] See Figure 2 , Figure 2 This is a schematic diagram of an embodiment of a teaching machine provided by an embodiment of the present invention. The teaching machine is equipped with a visual camera and includes:

[0142] The monitoring module 210 is configured to obtain, based on the visual camera, the target user's eye movement trajectory and the duration of attention paid to each video in the teaching video during the teaching video playback process;

[0143] The interactive display module 220 is configured to obtain a first target video having an attention duration greater than or equal to a preset duration, and display the first target video on the display interface of the teaching machine to obtain a second target video and a text prompt word; the text prompt word is input by the target user on the display interface, and the second target video includes multiple images, each image containing a target to be analyzed selected by the target user;

[0144] An acquisition module 230 is configured to perform feature extraction on the second target video to obtain visual features of the second target video;

[0145] A feature extraction module 240 is used to extract features of the text prompt word to obtain text features of the text prompt word;

[0146] The prediction module 250 is configured to input the visual features and the text features into a pre-trained large language model to obtain a descriptive text of the target to be analyzed output by the large language model.

[0147] The teaching machine provided by the present invention allows a target user to select a target to be analyzed on his / her own, performs feature extraction on a second target video containing the target to be analyzed selected by the target user to obtain visual features, then performs feature extraction on a text prompt word of the target user to obtain text features, and uses a large language model to analyze the visual features and text features to obtain a description text of the target to be analyzed, thereby completing the analysis and description of a specific target in the video, focusing on the content that the target user is interested in, improving the target user's learning enthusiasm, and thereby improving the target user's learning efficiency.

[0148] See Figure 3 , Figure 3 is a structural diagram of an electronic device provided by the present invention. An embodiment of the present invention provides an electronic device 300, including a memory 310, a processor 320, and a computer program 311 stored in the memory 310 and executable on the processor 320. When the processor 320 executes the computer program 311, the following steps are implemented:

[0149] Acquiring, based on the visual camera, the target user's eye movement trajectory during the playback of the teaching video for the duration of attention paid to each video in the teaching video;

[0150] Obtaining a first target video whose viewing duration is greater than or equal to a preset duration, and displaying the first target video on a display interface of the teaching machine to obtain a second target video and a text prompt word; the text prompt word is input by the target user on the display interface, and the second target video includes multiple images, each image containing a target to be analyzed selected by the target user;

[0151] performing feature extraction on the second target video to obtain visual features of the second target video;

[0152] Performing feature extraction on the text prompt word to obtain text features of the text prompt word;

[0153] The visual features and the text features are input into a pre-trained large language model to obtain a description text of the target to be analyzed output by the large language model.

[0154] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus the necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or certain parts of the embodiment.

[0155] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. An interactive control method based on eye movement trajectory analysis, characterized in that: Applied to a teaching machine equipped with a visual camera, the interactive control method based on eye movement trajectory analysis includes: Acquiring, based on the visual camera, the target user's eye movement trajectory during the playback of the teaching video for the duration of attention paid to each video in the teaching video; Obtaining a first target video whose viewing duration is greater than or equal to a preset duration, and displaying the first target video on a display interface of the teaching machine to obtain a second target video and a text prompt word; the text prompt word is input by the target user on the display interface, and the second target video includes multiple images, each image containing a target to be analyzed selected by the target user; performing feature extraction on the second target video to obtain visual features of the second target video; Performing feature extraction on the text prompt word to obtain text features of the text prompt word; Inputting the visual features and the text features into a pre-trained large language model to obtain a descriptive text of the target to be analyzed output by the large language model; The step of extracting features from the second target video to obtain visual features of the second target video includes: Determining a target mask, depth features, video history features, and encoding features of a first frame image; the first frame image being the first frame image of the second target video; The next frame image of the first frame image is used as the current frame image; Determining a depth feature of the current frame image based on a target mask of the current frame image and a frame image previous to the current frame image; Determining encoding features of the current frame image based on the depth features of the current frame image and the video history features of the previous frame image of the current frame image; Inputting the encoded features of the current frame image into a mask decoding network to obtain a target mask of the current frame image output by the mask decoding network; Determining the video history feature of the current frame image based on the coding feature of the current frame image and the video history feature of the previous frame image of the current frame image; Taking the next frame image of the current frame image as the current frame image, returning to the target mask based on the current frame image and the frame image before the current frame image, and determining the depth feature of the current frame image, until the target mask, depth feature, video history feature, and encoding feature of each image in the second target video are obtained; using the video history features of the last frame of the second target video as the visual features of the second target video; The determining of the target mask, depth features, video history features, and encoding features of the first frame image includes: Determining interaction information of the target user; Inputting the first frame image and the interaction information into a SAM model to obtain a target mask of the first frame image output by the SAM model; performing a splicing process on the first frame image and the target mask of the first frame image to obtain a spliced ​​image; Based on a visual feature extraction network, feature extraction processing is performed on the spliced ​​image to obtain the depth feature of the first frame image and the video history feature of the first frame image; Extracting coding features of the first frame image; After obtaining the target mask, depth feature, video history feature and encoding feature of each image in the second target video, the method further includes: Based on the target mask of each image, target segmentation is performed on each image to obtain the target segmentation result of each image; The target segmentation result of each image is returned to the target user.

2. The interactive control method based on eye movement analysis according to claim 1, characterized in that: The specific steps of obtaining the second target video include: In response to the interaction information of the target user, determining the target to be analyzed in the first target video and the processing duration selected by the target user based on the interaction information; The second target video is determined based on the target to be analyzed in the first target video and the processing time selected by the target user.

3. The interactive control method based on eye movement analysis according to claim 1, characterized in that: Inputting the visual features and the text features into a pre-trained large language model to obtain a description text of the target to be analyzed output by the large language model includes: Performing dimension adjustment processing on the visual feature so that the visual feature and the text feature have the same feature dimension; The visual features and the text features that have undergone dimension adjustment are input into a pre-trained large language model to obtain a descriptive text of the target to be analyzed output by the large language model.

4. The interactive control method based on eye movement analysis according to any one of claims 1 to 3, characterized in that: The interactive control method based on eye movement trajectory analysis also includes: Acquiring visual information of the teaching environment, wherein the visual information includes a depth map and a two-dimensional map; capturing the target user's eye focus area on the teaching medium based on the visual information, and generating a teaching medium heat map based on the target user's eye focus area on the teaching medium; Based on the teaching medium heat map, the target users' attention to the style, content, and layout of the teaching medium is analyzed to obtain analysis results.

5. The interactive control method based on eye movement analysis according to claim 4, characterized in that: The interactive control method based on eye movement trajectory analysis also includes: determining a movement trajectory of the teacher during the teaching process based on the visual information, and generating a movement trajectory heat map based on the movement trajectory; Based on the movement trajectory heat map, the teacher's teaching behavior is analyzed.

6. The interactive control method based on eye movement analysis according to claim 4, characterized in that: The interactive control method based on eye movement trajectory analysis also includes: Performing behavior recognition on the visual information to obtain the classroom behavior of the target user; Based on the teaching medium heat map and the classroom behavior of the target user, a teaching behavior analysis is performed.

7. A teaching machine, characterized in that: The teaching machine is equipped with a visual camera, and the teaching machine includes: A monitoring module, configured to obtain, based on the visual camera, the target user's eye movement trajectory and the duration of attention paid to each video in the teaching video during the playing of the teaching video; An interactive display module is configured to obtain a first target video having an attention duration greater than or equal to a preset duration, and display the first target video on a display interface of the teaching machine to obtain a second target video and a text prompt word; the text prompt word is input by the target user on the display interface, and the second target video includes multiple images, each image containing a target to be analyzed selected by the target user; an acquisition module, configured to perform feature extraction on the second target video to obtain visual features of the second target video; A feature extraction module is used to extract features of the text prompt word to obtain text features of the text prompt word; A prediction module, configured to input the visual features and the text features into a pre-trained large language model to obtain a descriptive text of the target to be analyzed output by the large language model; The step of extracting features from the second target video to obtain visual features of the second target video includes: Determining a target mask, depth features, video history features, and encoding features of a first frame image; the first frame image being the first frame image of the second target video; The next frame image of the first frame image is used as the current frame image; Determining a depth feature of the current frame image based on a target mask of the current frame image and a frame image previous to the current frame image; Determining encoding features of the current frame image based on the depth features of the current frame image and the video history features of the previous frame image of the current frame image; Inputting the encoded features of the current frame image into a mask decoding network to obtain a target mask of the current frame image output by the mask decoding network; Determining the video history feature of the current frame image based on the coding feature of the current frame image and the video history feature of the previous frame image of the current frame image; Taking the next frame image of the current frame image as the current frame image, returning to the target mask based on the current frame image and the frame image before the current frame image, and determining the depth feature of the current frame image, until the target mask, depth feature, video history feature, and encoding feature of each image in the second target video are obtained; using the video history features of the last frame of the second target video as the visual features of the second target video; The determining of the target mask, depth features, video history features, and encoding features of the first frame image includes: Determining interaction information of the target user; Inputting the first frame image and the interaction information into a SAM model to obtain a target mask of the first frame image output by the SAM model; performing a splicing process on the first frame image and the target mask of the first frame image to obtain a spliced ​​image; Based on a visual feature extraction network, feature extraction processing is performed on the spliced ​​image to obtain the depth feature of the first frame image and the video history feature of the first frame image; Extracting coding features of the first frame image; After obtaining the target mask, depth feature, video history feature and encoding feature of each image in the second target video, the method further includes: Based on the target mask of each image, target segmentation is performed on each image to obtain the target segmentation result of each image; The target segmentation result of each image is returned to the target user.

Images