Gaze image marking method based on meta-space self-feedback learning

Abstract

The embodiment of the invention discloses a staring image marking method based on meta-space self-feedback learning, and relates to the technical field of image processing, and the method comprises the steps: obtaining a face image sequence of a target learner, extracting and obtaining a staring feature corresponding to each image frame, inputting the staring features into a trained mimicry image generation model, and obtaining a target image sequence; generating a mimicry image by combining a mimicry image generation model with a preset image with a staring direction label; and performing feature extraction on the mimicry image to obtain a mimicry staring feature, performing feature splicing on the mimicry staring feature and the staring feature, outputting the spliced staring feature, performing calculation to obtain the staring direction of the target learner, and obtaining a staring mark image. According to the embodiment of the invention, the gazing directions of a large amount of image data can be respectively marked in batches, the attention state of the learner in the learning process can be obtained, timely intervention can be provided by a teacher or a lecturer through the attention transfer state at different times, and the learning efficiency and high efficiency of the learner are ensured.

Description

Technical Field

[0001] The present application relates to the technical field of image processing, and in particular to a gaze image marking method based on metaspace self-feedback learning. Background Art

[0002] In order to study the classroom concentration, engagement, activity and other related classroom performance states of learners in the teaching environment of smart classrooms, video data of learners in teaching scenarios is usually collected for analysis and research, and then the students' attention state in the classroom is analyzed from the learners' gaze direction, that is, following the learners' gaze direction. However, the learners' gaze in the classroom is usually changeable, and each gaze situation lasts for a short time, making it difficult to capture the students' gaze situation over a period of time. When learners are not focused, they are often accompanied by a shift in attention, which is not easy for teachers to detect, and it is difficult to judge the students' actual learning state.

[0003] At present, the gaze marking of learners in the classroom, that is, the direction of the learners' gaze, is usually done manually. This method will cause certain errors and subjectivity in the learners' gaze marking, and is easily affected by the personal experience of the markers. Different markers may have different marking results for the same image to be marked, resulting in ambiguous markings, and manual marking will consume a lot of manpower and material resources. In addition, due to changes in light, there may be noise points in the collected images, which will affect the accuracy of gaze calculation.

[0004] Therefore, the current limitations of the clarity of the collected images and the subjectivity and ambiguity of manual image labeling will to a certain extent affect the analysis of learners' gaze states, and it is impossible to objectively and accurately judge students' attention in class. Summary of the invention

[0005] The embodiment of the present application provides a gaze image marking method based on metaspace self-feedback learning to solve the defects in the above-mentioned related technologies. The technical solution is as follows: In a first aspect, an embodiment of the present application provides a gaze image marking method based on metaspace self-feedback learning, comprising: Obtain a face image sequence of a target learner; Inputting each image frame in the face image sequence into a metaspace encoder to extract gaze features corresponding to each image frame; Inputting the gaze feature into a trained mimetic image generation model, and generating a mimetic image by combining the mimetic image generation model with a preset image with a gaze direction label; Extracting features from the mimicry image to obtain a mimicry gaze feature, performing feature splicing on the mimicry gaze feature and the gaze feature, and outputting the spliced ​​gaze feature; The gaze direction of the target learner is calculated based on the spliced ​​gaze features, and a gaze mark image is obtained based on the image frame corresponding to the gaze direction mark.

[0006] In an optional solution of the first aspect, obtaining a face image sequence of a target learner includes: An image sequence of the target learner is obtained, face detection is performed on each image frame in the image sequence, and each image frame in the image sequence is cropped based on the result of the face detection, and a corresponding face image sequence is output.

[0007] In an optional solution of the first aspect, after acquiring the face image sequence of the target learner, the method further includes: Performing denoising on each image frame in the face image sequence using a trained diffusion model to obtain a denoised purified image sequence; The step of inputting each image frame in the face image sequence into a metaspace encoder to extract gaze features corresponding to each image frame is performed based on the purified image sequence.

[0008] In an optional solution of the first aspect, after obtaining the purified image sequence after denoising, the method further includes: The purified image sequence and the corresponding face image sequence are respectively input into the metaspace encoder, and the step of extracting the gaze feature corresponding to each of the image frames is performed until a first gaze direction of the target learner in the image frame of the purified image sequence and a second gaze direction of the target learner in the image frame of the face image sequence are calculated; A gaze direction difference between the first gaze direction and the second gaze direction is calculated. If the gaze direction difference is greater than a preset threshold, the purified image sequence is re-input into the metaspace encoder, and a gaze direction calculated based on image frames of the purified image sequence is output.

[0009] In an optional solution of the first aspect, inputting the gaze feature into a trained mimetic image generation model, and generating a mimetic image by combining the mimetic image generation model with a preset image with a gaze direction label, includes: Inputting the gaze feature into a trained mimetic image generation model, extracting eye features from the gaze feature through the mimetic image generation model, and adjusting the distribution of the eye features and other features except the eye features, so that the image frame corresponding to the gaze feature has the same feature distribution as the preset image with the gaze direction label; generating a corresponding mimicry image by combining the eye features in the gaze features and other features except the eye features in the preset image with the gaze direction label; Wherein, the trained mimicry image generation model is trained based on the preset image with gaze direction label.

[0010] In an optional solution of the first aspect, the method further includes: Determining the change of the gaze direction of the target learner within the preset time period based on the gaze mark images of the target learner at multiple moments within the preset time period; The gaze direction change condition is compared with a gaze direction change threshold, and the attention state of the target learner in the preset time period is determined based on the comparison result.

[0011] In a second aspect, the embodiment of the present application further provides a gaze image marking device based on metaspace self-feedback learning, comprising: An image sequence acquisition module is used to acquire a face image sequence of a target learner; A first gaze feature extraction module, used for inputting each image frame in the face image sequence into a metaspace encoder to extract a gaze feature corresponding to each image frame; A mimetic image generation module, used for inputting the gaze feature into a trained mimetic image generation model, and generating a mimetic image through the mimetic image generation model in combination with a preset image with a gaze direction label; A second gaze feature extraction module is used to extract features from the mimicry image to obtain a mimicry gaze feature, perform feature splicing on the mimicry gaze feature and the gaze feature, and output the spliced ​​gaze feature; A gaze direction marking module is used to calculate the gaze direction of the target learner based on the spliced ​​gaze features, and obtain a gaze marked image based on the image frame corresponding to the gaze direction mark.

[0012] In a third aspect, an embodiment of the present application further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method provided in the first aspect of the embodiment of the present application or any one of the implementation methods of the first aspect is implemented.

[0013] In a fourth aspect, the present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method provided by the first aspect of the embodiment of the present application or any one of the implementations of the first aspect.

[0014] The beneficial effects brought about by the technical solutions provided by some embodiments of the present application include at least: The embodiment of the present application can provide richer data support for the evaluation of learning attention by collecting image data of each student in the classroom and analyzing the gaze direction of the students based on video data, thereby more accurately evaluating the learning attention of each student. Through simple arrangement, the gaze direction of each learner in the classroom can be identified, so that the status of each student in the classroom can be confirmed, avoiding the subjectivity of identifying the gaze direction, and being able to mark large-scale image data in batches, allowing teaching staff to focus on improving teaching effectiveness and improving the teaching and research efficiency of teaching staff.

[0015] The embodiment of the present application generates a mimetic image based on a collected sequence of learner facial images combined with a preset image with a gaze direction label. The mimetic image contains the eye features of the image to be marked, and its other features such as background and lighting are all original information in the preset image with a gaze direction label. In this way, the accuracy of extracting the gaze features of the mimetic image can be improved and the calculation difficulty can be reduced. In addition, the gaze features extracted based on the facial image sequence are used as auxiliary features, and the gaze features and the gaze features of the mimetic image are spliced ​​to identify the gaze direction, which can more accurately mark the gaze direction of the student in the image.

[0016] The embodiment of the present application can derive the attention state of the learner during the learning process. By analyzing the attention transfer state at different times, it is helpful for teachers or instructors to provide timely intervention and ensure the learner's learning efficiency and high efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present application or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is a flow chart of a gaze image marking method based on metaspace self-feedback learning provided in an embodiment of the present application; Figure 2 It is a flow chart of denoising based on a diffusion model of a gaze image marking method based on metaspace self-feedback learning provided in an embodiment of the present application; Figure 3 It is a structural schematic diagram of a gaze image marking device based on metaspace self-feedback learning provided in an embodiment of the present application; Figure 4 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0020] The terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device including a series of steps or modules is not limited to the listed steps or modules, but may optionally include steps or modules that are not listed, or may optionally include other steps or modules that are inherent to these processes, methods, products or devices.

[0021] It should be noted that the terms "first\second" involved in the present application are only used to distinguish similar objects, and do not represent a specific order for the objects. It is understandable that "first\second" can be interchanged with a specific order or sequence where permitted. It should be understood that the objects distinguished by "first\second" can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those described or illustrated herein.

[0022] The present application is described in detail below with reference to specific embodiments.

[0023] Next, combine Figure 1 , introduces a gaze image labeling method based on metaspace self-feedback learning provided by the embodiment of the present application. For details, please refer to Figure 1 , Figure 1 FIG. 1 is a flow chart of a method for marking gaze images based on self-feedback learning of a metaspace provided in an embodiment of the present application. Figure 1 As shown, the method comprises the following steps: S101, obtaining a face image sequence of a target learner; S102, inputting each image frame in the facial image sequence into a metaspace encoder to extract gaze features corresponding to each image frame; S103, inputting the gaze feature into a trained mimetic image generation model, and generating a mimetic image by combining the mimetic image generation model with a preset image with a gaze direction label; S104, extracting features from the mimicry image to obtain a mimicry gaze feature, performing feature splicing on the mimicry gaze feature and the gaze feature, and outputting the spliced ​​gaze feature; S105, calculating the gaze direction of the target learner based on the spliced ​​gaze features, and obtaining a gaze mark image based on the image frame corresponding to the gaze direction mark.

[0024] In some embodiments, obtaining a face image sequence of a target learner in S101 specifically includes: An image sequence of the target learner is obtained, face detection is performed on each image frame in the image sequence, and each image frame in the image sequence is cropped based on the result of the face detection, and a corresponding face image sequence is output.

[0025] It can be understood that the images of all students in the entire classroom can be collected by a camera installed in the classroom, and the images can be segmented through face recognition to obtain the face image of each student.

[0026] In some embodiments, DVS dynamic vision cameras and intelligent tracking cameras can be deployed in the teaching environment to fully collect images of each learner, and the accuracy of the data can be ensured by collecting learners' facial data from all angles in the classroom teaching environment. For example, at the beginning of the class, all cameras can be started to start collecting learners' facial image data. The collected video frame rate can be set to 24fps, and the specific collection time can be based on the teaching class hours. The collected image data is framed in time sequence, and the time interval is set to 900 milliseconds. At this time, the image data will be converted into a facial image sequence.

[0027] In some embodiments, after acquiring the face image sequence of the target learner in S101, the following steps may be performed to pre-process the face image sequence of the target learner, specifically including: Performing denoising on each image frame in the face image sequence using a trained diffusion model to obtain a denoised purified image sequence; Further, S102 and subsequent steps may be performed based on the purified image sequence.

[0028] In some embodiments, since the collected learner images may have noise problems such as blurry images and low resolution due to the distance, as well as image noise caused by light changes in different time periods, complex classroom backgrounds, and equipment noise, a diffusion model can be used to improve the quality of the collected learner images ( ) to purify the image, reduce the noise in the image, and improve the accuracy of gaze direction recognition. You can refer to Figure 2 The flowchart of diffusion model denoising is shown in FIG. 1 . The denoising process specifically includes: The variational autoencoder compresses the above face image sequence and adds a noise image to obtain a noise image sequence; The noisy image sequence is input into the diffusion model ( ) in denoising, where the SC block is used as The main building block of , which contains sub-modules: residual convolution (RConv) block and SwinT block.

[0029] 3×3 convolution can be used to obtain shallow features of noisy images , and then further through the first 1×1 convolution, the above shallow feature Q is evenly divided into two feature map groups and , and then input into the SwinT block and the residual convolution block respectively, and then connect the outputs of the SwinT block and the residual convolution block, and then pass the second 1×1 convolution to generate preliminary features , the specific application formula is: in, represents the split operation, then, and Input into SwinT module and RConv module respectively, and apply the formula: ; in, and The outputs of the SwinT block and the residual convolution block are connected and As input to the second 1×1 convolution, this convolution is Therefore, the final output of the SC block is the preliminary features of the noisy image. It can be expressed as: ; in, Represents a feature concatenation operation.

[0030] Further, a denoising process is performed, specifically including: The downsampling and upsampling process consists of strided convolution and transposed convolution.

[0031] The initial features As input, preliminary features It contains four scale features. Each scale has a residual connection between the downsampling based on 2×2 strided convolution and the upsampling based on 2×2 transposed convolution. In addition, 4 SC blocks are taken between downsampling and upsampling to enhance the local and non-local modeling capabilities. Finally, after 4 SC blocks, a deep feature map containing low-order features and high-order features is obtained. Finally, a 3×3 convolution is used to reconstruct the restored image, thereby obtaining a purified image sequence with noise removed after preprocessing. , which can be expressed as: ; in, Represents 3×3 convolution from deep features Generate purified images in .

[0032] In some embodiments, after obtaining the purified image sequence after denoising, the method further includes: The purified image sequence and the corresponding face image sequence can be respectively input into the metaspace encoder, and S102 and subsequent steps in the above embodiment are executed to obtain the first gaze direction of the target learner in the image frame of the purified image sequence and the second gaze direction of the target learner in the image frame of the face image sequence.

[0033] It can be understood that the image frames of the purified image sequence and the face image sequence synchronously input into the metaspace encoder correspond to each other, that is, the purified image frames simultaneously input into the metaspace encoder are obtained by purifying the image frames in the face image sequence through a diffusion model.

[0034] Furthermore, a gaze direction difference between the first gaze direction and the second gaze direction may be calculated. If the gaze direction difference is greater than a preset threshold, the purified image sequence is re-input into the metaspace encoder, and a gaze direction calculated based on the image frames of the purified image sequence is output.

[0035] It should be noted that when the distance between the front and back of the classroom is large and the number of cameras is small, the facial images of students in the back row of the classroom may have poor clarity and more noise, resulting in a large difference between the purified image and the facial image. At this time, the gaze directions identified by the purified image and the original facial image may be quite different. By re-identifying the gaze direction of the purified image, the model recognition error can be avoided to a certain extent.

[0036] In some embodiments, in S102, each image frame in the facial image sequence is input into a metaspace encoder to extract gaze features corresponding to each image frame, specifically including: Each image frame is processed by a meta-feature encoder, and the high-dimensional image is converted into a low-dimensional feature space through the meta-feature encoder. This feature retains the information related to the gaze direction and filters out the noise features that are not related to the gaze information (such as lighting, background features, etc.). First, the meta-gaze feature can be extracted, and then the meta-gaze feature is further enhanced to obtain the optimized gaze feature.

[0037] In some embodiments, in S103, the gaze feature may be input into a trained mimetic image generation model, and a mimetic image may be generated by combining the mimetic image generation model with a preset image with a gaze direction label, specifically including: The gaze feature can be input into a trained mimetic image generation model, and the eye feature in the gaze feature can be extracted by the mimetic image generation model, and the distribution of the eye feature and other features except the eye feature can be adjusted so that the image frame corresponding to the gaze feature has the same feature distribution as the preset image with the gaze direction label; The corresponding mimicry image is generated by combining the eye features in the gaze features and other features except the eye features in the preset image with the gaze direction label.

[0038] Specifically, the gaze features of the image to be marked, that is, the image frames of the face image sequence, can be mapped to a preset image with a gaze direction label; the eye features related to the gaze direction in the gaze features are retained, and the image to be marked and the preset image with the gaze direction label have the same features other than the gaze features, such as lighting and background, which are not related to the eye features. In this way, the eye features of the image to be marked and other features of the preset image with the gaze direction label can be combined to obtain a mimicking image.

[0039] That is, it is equivalent to mapping the gaze features of the image frames of the face image sequence into the meta-space of the preset image with the gaze direction label, which can not only retain the gaze features of the image to be labeled, but also conform to the feature distribution of the preset image. The gaze image obtained in this way concentrates the feature distribution on the eye features, so that the gaze feature extraction model does not need to pay attention to other features besides the eye features, thereby improving the accuracy of gaze feature extraction.

[0040] It can be understood that the trained mimicry image generation model is obtained during the training process based on the preset images with gaze direction labels.

[0041] Specifically, the mimic image generation model can be trained using publicly available preset images with gaze direction labels. In order to extract more detailed gaze information, a gaze distortion loss function is proposed. , which retains the gaze feature during the encoding process, can first train the feature extractor F that combines ResNet-18 and MLP to estimate the gaze vector of the face image, and then use F to generate the mimic image Medium Calculation , calculated as follows: ; in, is the meta-feature encoder, is a generator, represents the input image, represents the true value. The total loss is calculated as follows: ; in, , , and Represent the weights of each loss respectively. In the meta-space based gaze method, end-to-end training is adopted to optimize the extractor to generate images that retain gaze-related information.

[0042] Further, S104 may be performed to extract features from the mimicry image to obtain a mimicry gaze feature, perform feature splicing on the mimicry gaze feature and the gaze feature, and output the spliced ​​gaze feature, including: Swin Transformer can be used as the backbone network for feature extraction. The generated mimic image is used as input and passed through the Swin Transformer block to further extract gaze-related information, including: ; ; in, Represents the features of the previous layer that enters the windowed multi-head attention layer before the normalization layer. represents the normalization operation, Represents the window multi-head attention operation. Since there is a residual connection between each module, it can be expressed as: ; ; in, Represents the sliding window attention operation, and finally obtains the gaze feature of the mimic image .

[0043] Furthermore, the gaze features of the mimetic image and the gaze feature representation obtained in S102 can be combined through the feature stitching module, so that the model can extract information from multiple different perspectives; the generated mimetic image may be more suitable for directly perceiving the direction of gaze, while the gaze features obtained in the metaspace may help capture subtle eye movements, facial muscles and other information. After combining this information, feature fusion can be performed at a higher level, thereby enhancing the representation ability of the model and the gaze prediction accuracy.

[0044] Further, step S105 is performed to calculate the gaze direction of the target learner based on the spliced ​​gaze features obtained in the above embodiments, and obtain a gaze mark image based on the image frame corresponding to the gaze direction mark.

[0045] It can be understood that the steps of the above embodiment can be performed for multiple target learners in the classroom respectively, and the gaze situation of each target learner can be marked respectively, so as to facilitate the teacher to determine the attention situation of each student in the classroom.

[0046] In some embodiments, based on the gaze mark images of the target learner at multiple moments within a preset time period, the gaze direction change of the target learner within the preset time period is determined; the gaze direction change is compared with a gaze direction change threshold, and the attention state of the target learner within the preset time period is judged based on the comparison result.

[0047] Specifically, the gaze direction mark obtained may include the pitch angle and yaw angle of the target student's gaze direction. The change in the gaze direction of the target student in a given period of time may be easily determined through the numerical changes of the two angles, and the numerical thresholds of the two angles may be set respectively.

[0048] For example, if the change in the pitch angle is greater than the preset pitch angle threshold, it can be judged that the student is looking up and down at this time, that is, the student is looking down at the textbook and looking up at the blackboard in front, and the student's attention is relatively focused. If the change in the yaw angle is greater than the preset yaw angle threshold, it can be judged that the student is looking around at this time, and it can be judged that the student is not focused during this period. The setting of the gaze direction change threshold and the judgment conditions can be set separately according to the specific teaching environment, and the embodiments of the present application are not limited to this.

[0049] The following are device embodiments of the present application, which can be used to execute the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0050] See next Figure 3, which is a structural diagram of a gaze image marking device based on metaspace self-feedback learning provided by an exemplary embodiment of the present application. The device can be implemented as all or part of a terminal through software, hardware, or a combination of both, and can also be integrated on a server as an independent module. The gaze image marking device based on metaspace self-feedback learning in the embodiment of the present application can be applied to a terminal or a cloud. The device 30 includes an image sequence acquisition module 301, a first gaze feature extraction module 302, a mimic image generation module 303, a second gaze feature extraction module 304, and a gaze direction marking module 305, wherein: The image sequence acquisition module 301 is used to acquire the face image sequence of the target learner; The first gaze feature extraction module 302 is used to input each image frame in the face image sequence into a metaspace encoder to extract the gaze feature corresponding to each image frame; The mimetic image generation module 303 is used to input the gaze feature into a trained mimetic image generation model, and generate a mimetic image by combining the mimetic image generation model with a preset image with a gaze direction label; The second gaze feature extraction module 304 is used to extract features from the mimicry image to obtain a mimicry gaze feature, perform feature splicing on the mimicry gaze feature and the gaze feature, and output the spliced ​​gaze feature; The gaze direction marking module 305 is used to calculate the gaze direction of the target learner based on the spliced ​​gaze features, and obtain a gaze marked image based on the image frame corresponding to the gaze direction mark.

[0051] It should be noted that when the device 30 provided in the above embodiment executes the gaze image marking method based on metaspace self-feedback learning, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device provided in the above embodiment and the embodiment of the gaze image marking method based on metaspace self-feedback learning belong to the same concept. The implementation process is detailed in the method embodiment, which will not be repeated here.

[0052] An embodiment of the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the above-mentioned method embodiments when executing the program.

[0053] See also Figure 4 , which is a structural block diagram of an electronic device provided in an embodiment of the present application.

[0054] like Figure 4As shown, the electronic device 400 includes: a processor 401 and a memory 402 .

[0055] In the embodiment of the present application, the processor 401 is the control center of the computer system, which can be a processor of a physical machine or a processor of a virtual machine. The processor 401 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 401 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array).

[0056] The processor 401 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also called a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state.

[0057] The memory 402 may include one or more computer-readable storage media, which may be non-transitory. The memory 402 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments of the present application, the non-transitory computer-readable storage medium in the memory 402 is used to store at least one instruction, which is used to be executed by the processor 401 to implement the method in the embodiment of the present application.

[0058] In some embodiments, the electronic device 400 further includes: a peripheral device interface 403 and at least one peripheral device 404. The processor 401, the memory 402 and the peripheral device interface 403 can be connected via a bus or a signal line. Each peripheral device 404 can be connected to the peripheral device interface 403 via a bus, a signal line or a circuit board. Specifically, the peripheral device 404 includes: a display screen, a camera and an audio circuit. The peripheral device interface 403 can be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 401 and the memory 402.

[0059] In some embodiments of the present application, the processor 401, the memory 402, and the peripheral device interface 403 are integrated on the same chip or circuit board; in some other embodiments of the present application, any one or two of the processor 401, the memory 402, and the peripheral device interface 403 can be implemented on a separate chip or circuit board. This embodiment of the present application does not specifically limit this.

[0060] The electronic device structure block diagram shown in the embodiment of the present application does not constitute a limitation on the electronic device 400. The electronic device 400 may include more or fewer components than shown in the figure, or combine certain components, or adopt a different component arrangement.

[0061] The present application also provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps of the method of any of the above embodiments are implemented. The computer-readable storage medium may include, but is not limited to, any type of disk, including a floppy disk, an optical disk, a DVD, a CD-ROM, a micro drive, and a magneto-optical disk, a ROM, a RAM, an EPROM, an EEPROM, a DRAM, a VRAM, a flash memory device, a magnetic card or an optical card, a nanosystem (including a molecular memory IC), or any type of medium or device suitable for storing instructions and / or data.

[0062] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.

[0063] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A gaze image labeling method based on metaspace self-feedback learning, characterized in that: include: Obtain a face image sequence of a target learner; Inputting each image frame in the face image sequence into a metaspace encoder to extract gaze features corresponding to each image frame; Inputting the gaze feature into a trained mimetic image generation model, and generating a mimetic image by combining the mimetic image generation model with a preset image with a gaze direction label; Extracting features from the mimicry image to obtain a mimicry gaze feature, performing feature splicing on the mimicry gaze feature and the gaze feature, and outputting the spliced ​​gaze feature; The gaze direction of the target learner is calculated based on the spliced ​​gaze features, and a gaze mark image is obtained based on the image frame corresponding to the gaze direction mark.

2. According to claim 1, a gaze image labeling method based on metaspace self-feedback learning is characterized in that: The step of obtaining a face image sequence of a target learner comprises: An image sequence of the target learner is obtained, face detection is performed on each image frame in the image sequence, and each image frame in the image sequence is cropped based on the result of the face detection, and a corresponding face image sequence is output.

3. The gaze image labeling method based on metaspace self-feedback learning according to claim 1, characterized in that: After obtaining the face image sequence of the target learner, the method further includes: Performing denoising on each image frame in the face image sequence using a trained diffusion model to obtain a denoised purified image sequence; The step of inputting each image frame in the face image sequence into a metaspace encoder to extract gaze features corresponding to each image frame is performed based on the purified image sequence.

4. The gaze image labeling method based on metaspace self-feedback learning according to claim 3, characterized in that: After obtaining the purified image sequence after denoising, the method further includes: The purified image sequence and the corresponding face image sequence are respectively input into the metaspace encoder, and the step of extracting the gaze feature corresponding to each of the image frames is performed until a first gaze direction of the target learner in the image frame of the purified image sequence and a second gaze direction of the target learner in the image frame of the face image sequence are calculated; A gaze direction difference between the first gaze direction and the second gaze direction is calculated. If the gaze direction difference is greater than a preset threshold, the purified image sequence is re-input into the metaspace encoder, and a gaze direction calculated based on image frames of the purified image sequence is output.

5. The gaze image labeling method based on metaspace self-feedback learning according to claim 3, characterized in that: The step of inputting the gaze feature into a trained mimetic image generation model, and generating a mimetic image by combining the mimetic image generation model with a preset image with a gaze direction label, comprises: Inputting the gaze feature into a trained mimetic image generation model, extracting eye features from the gaze feature through the mimetic image generation model, and adjusting the distribution of the eye features and other features except the eye features, so that the image frame corresponding to the gaze feature has the same feature distribution as the preset image with the gaze direction label; generating a corresponding mimicry image by combining the eye features in the gaze features and other features except the eye features in the preset image with the gaze direction label; Wherein, the trained mimicry image generation model is trained based on the preset image with gaze direction label.

6. A gaze image labeling method based on metaspace self-feedback learning according to any one of claims 1 to 5, characterized in that: The method further comprises: Determining the change of the gaze direction of the target learner within the preset time period based on the gaze mark images of the target learner at multiple moments within the preset time period; The gaze direction change condition is compared with a gaze direction change threshold, and the attention state of the target learner in the preset time period is determined based on the comparison result.

7. A gaze image marking device based on metaspace self-feedback learning, characterized in that: include: An image sequence acquisition module is used to acquire a face image sequence of a target learner; A first gaze feature extraction module, used for inputting each image frame in the face image sequence into a metaspace encoder to extract a gaze feature corresponding to each image frame; A mimetic image generation module, used for inputting the gaze feature into a trained mimetic image generation model, and generating a mimetic image through the mimetic image generation model in combination with a preset image with a gaze direction label; A second gaze feature extraction module is used to extract features from the mimicry image to obtain a mimicry gaze feature, perform feature splicing on the mimicry gaze feature and the gaze feature, and output the spliced ​​gaze feature; A gaze direction marking module is used to calculate the gaze direction of the target learner based on the spliced ​​gaze features, and obtain a gaze marked image based on the image frame corresponding to the gaze direction mark.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images