Learning process data acquisition method, system and equipment and storage medium

Abstract

The invention discloses a learning process data acquisition method, system and device and a storage medium, and relates to the technical field of data acquisition, and the method comprises the steps: receiving a gaze image outputted by an intelligent head-mounted device; a fixation point of a target user is determined from the fixation image, target question data at the fixation point are acquired, and the target user wears an intelligent head-mounted device; and receiving target handwriting data output by the handwriting board, and associatively storing the target handwriting data and the target question data in a preset target storage space. The question actually watched by the user is recognized through the intelligent head-mounted device, the handwriting of the user is recognized through the handwriting board, the question and the answer data when the user answers the paper question can be collected, the method can be suitable for more learning scenes, the answer process data of the student can be obtained in real time, and the user experience is improved. And the user does not need to carry out additional repeated operation, the automation degree is high, and the operation is simple.

Description

Technical Field

[0001] This application relates to the field of data acquisition technology, and in particular to a method, system, device and storage medium for acquiring data during a learning process. Background Technology

[0002] In the field of education, with the rapid development of information technology, educators have an increasing demand for precise assessment of students' learning processes and personalized teaching. Among these, data on students' problem-solving processes, as important indicators reflecting their knowledge mastery, learning progress, and problem-solving strategies, have high analytical value.

[0003] Currently, the technical means for collecting student learning data, especially test-taking data, mainly focus on the following aspects: First, using online learning platforms or educational software to record various data of students' online answers through programming interfaces (APIs); second, collecting students' paper-based homework answers through scanners or photo upload functions, and then using OCR (Optical Character Recognition) technology for answer recognition and preliminary analysis.

[0004] While the aforementioned technologies have achieved some degree of data collection from students' learning, they still have certain limitations. First, data collection on online platforms is constrained by the format of the questions and user habits, making it difficult to cover all learning scenarios. Second, scanning or taking photos can only check the accuracy of the calculation process and answer, but cannot determine the time taken by students to complete the task. Furthermore, when there are many questions, repeated manual photography or scanning is necessary, which is cumbersome. Therefore, existing technologies are significantly insufficient in collecting detailed data on students' problem-solving processes, and cannot meet the current demand for precise learning analysis in educational practice.

[0005] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0006] The main purpose of this application is to provide a method, system, device and storage medium for collecting learning process data, aiming to solve the technical problem of how to collect student learning process data.

[0007] To achieve the above objectives, this application proposes a method for collecting data during the learning process, the method comprising:

[0008] Receive gaze images output from smart head-mounted devices;

[0009] The gaze point of the target user is determined from the gaze image, and the target question data at the gaze point is obtained, wherein the target user is wearing the smart head-mounted device;

[0010] Receive the target handwriting data output by the handwriting tablet, and associate the target handwriting data with the target question data and store it in a preset target storage space.

[0011] In one embodiment, the step of determining the gaze point of the target user from the gaze image includes:

[0012] Obtain the depth information of the gaze image and the focal length of the camera of the smart head-mounted device;

[0013] The ratio between the depth information and the focal length is determined as the scaling factor of the camera;

[0014] Obtain the first relative offset value between the camera and the center points of the target user's eyes;

[0015] A second relative offset value between the gaze point and the image midpoint of the gaze image is determined based on the first relative offset value and the scaling factor;

[0016] The point in the gaze image that is offset by a second relative offset value relative to a point in the image is taken as the user's gaze point.

[0017] In one embodiment, after the step of determining a second relative offset value between the gaze point and the image midpoint of the gazed image based on the first relative offset value and the scaling factor, the method further includes:

[0018] Receive the gaze angle and gaze depth of the target user output by the eye-tracking module of the smart head-mounted device;

[0019] The actual offset value of the gaze point is calculated based on the gaze angle, gaze depth, and the second relative offset value;

[0020] The point in the gaze image that is offset from a point in the image by the actual offset value is taken as the user's gaze point.

[0021] In one embodiment, the step of acquiring the target question data at the fixation point includes:

[0022] Identify at least one title region in the gaze image;

[0023] When the duration of the gaze point being in the target area of ​​each of the title areas is greater than a preset first duration, text data in the target area is recognized by text recognition technology;

[0024] The text data is used as the target title data at the gaze point.

[0025] In one embodiment, after the step of determining the gaze point of the target user from the gaze image, the method further includes:

[0026] The motion path of the gaze point is acquired within a preset time window;

[0027] If the amplitude of the motion in the motion path is less than a preset amplitude threshold, the prompting module will output a first prompt message.

[0028] In one embodiment, the method further includes:

[0029] Receive the target user's eye-closing time output by the eye-tracking module;

[0030] If the time the eyes are closed is longer than the preset second duration, then the prompting module outputs a second prompt message.

[0031] In one embodiment, after the step of associating and storing the target handwriting data with the target question data in a preset target storage space, the method further includes:

[0032] Upon receiving the target handwriting data output by the handwriting tablet, the timer for the question-solving period begins;

[0033] If the target handwriting data output by the handwriting tablet is not received, the current time is marked as the first time point, and the timer for the pause in writing begins;

[0034] If the pause in writing exceeds a preset third duration, and the distance between the received new handwriting data and the target handwriting data exceeds a preset distance threshold, then the timing of the question-solving time is stopped at the first time point to obtain the question-solving time.

[0035] Alternatively, if the pause in writing exceeds a preset fourth duration, the timing of the problem-solving time is stopped at the first time point to obtain the problem-solving duration, wherein the fourth duration is longer than the third duration.

[0036] Furthermore, to achieve the above objectives, this application also proposes a learning process data acquisition system, which includes:

[0037] The acquisition module is used to receive gaze images output by the smart head-mounted device;

[0038] The question data acquisition module is used to determine the gaze point of the target user from the gaze image and acquire the target question data at the gaze point, wherein the target user is wearing the smart head-mounted device;

[0039] The handwriting data acquisition module is used to receive the target handwriting data output by the handwriting tablet and associate the target handwriting data with the target question data and store it in a preset target storage space.

[0040] In addition, to achieve the above objectives, this application also proposes a learning process data acquisition device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the learning process data acquisition method as described above.

[0041] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the learning process data acquisition method described above.

[0042] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the learning process data acquisition method described above.

[0043] This application provides a method for collecting learning process data. This application acquires gaze images through a camera set on a smart head-mounted device worn by the target user, then determines the gaze point where the target user is actually gazing from the gaze image, and then identifies the target question data at the user's gaze point in the gaze image, that is, the information of the question that the target user is actually gazing at. When the target user writes the question by hand, the target user's handwriting data is collected through a handwriting tablet. The handwriting of the target user and the question that the target user is gazing at are associated and stored in a preset target storage space, thereby collecting learning data during the target user's learning process.

[0044] In summary, this application uses a smart head-mounted device to identify the question the user is actually looking at and a handwriting tablet to recognize the user's handwriting. Compared to online answering, this application can collect the question and answer data when the user answers paper questions, making it applicable to more learning scenarios. Furthermore, compared to scanning or photographing paper questions, this application can obtain the student's answer process data in real time without requiring the user to perform additional repetitive operations, resulting in a high degree of automation and simple operation. Attached Figure Description

[0045] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0046] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This is a flowchart illustrating the data acquisition method for the learning process described in Embodiment 1 of this application.

[0048] Figure 2 This is a first scene diagram related to the implementation embodiment of the learning process data acquisition method of this application;

[0049] Figure 3 This is a second scene diagram related to the implementation embodiment of the learning process data acquisition method of this application;

[0050] Figure 4 This is a third scenario diagram related to the implementation of the learning process data acquisition method of this application;

[0051] Figure 5 This is a fourth scenario diagram related to the implementation of the learning process data acquisition method of this application;

[0052] Figure 6 This is a schematic diagram of the module structure of the learning process data acquisition system in an embodiment of this application;

[0053] Figure 7 This is a schematic diagram of the device structure of the hardware operating environment involved in the learning process data acquisition method in the embodiments of this application.

[0054] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0055] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0056] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0057] The main solution of this application embodiment is: receiving a gaze image output by a smart head-mounted device; determining the gaze point of a target user from the gaze image and obtaining target question data at the gaze point, wherein the target user is wearing the smart head-mounted device; receiving target handwriting data output by a handwriting tablet and associating the target handwriting data with the target question data and storing them in a preset target storage space.

[0058] In the field of education, with the rapid development of information technology, educators have an increasing demand for precise assessment of students' learning processes and personalized teaching. Among these, data on students' problem-solving processes, as important indicators reflecting their knowledge mastery, learning progress, and problem-solving strategies, have high analytical value.

[0059] Currently, the technical means for collecting student learning data, especially test-taking data, mainly focus on the following aspects: First, using online learning platforms or educational software to record various data of students' online answers through programming interfaces (APIs); second, collecting students' paper-based homework answers through scanners or photo upload functions, and then using OCR (Optical Character Recognition) technology for answer recognition and preliminary analysis.

[0060] While the aforementioned technologies have achieved some degree of data collection from students' learning, they still have certain limitations. First, data collection on online platforms is constrained by the format of the questions and user habits, making it difficult to cover all learning scenarios. Second, scanning or taking photos can only check the accuracy of the calculation process and answer, but cannot determine the time taken by students to complete the task. Furthermore, when there are many questions, repeated manual photography or scanning is necessary, which is cumbersome. Therefore, existing technologies are significantly insufficient in collecting detailed data on students' problem-solving processes, and cannot meet the current demand for precise learning analysis in educational practice.

[0061] To address the aforementioned issues, this application provides a method for collecting learning process data. This application acquires gaze images through a camera installed on a smart head-mounted device worn by the target user, then determines the gaze point where the target user is actually gazing from the gaze image, and then identifies the target question data at the user's gaze point in the gaze image, i.e., the information of the question that the target user is actually gazing at. When the target user writes the question by hand, the target user's handwriting data is collected through a handwriting tablet. The handwriting of the target user and the question that the target user is gazing at are associated and stored in a preset target storage space, thereby collecting learning data during the target user's learning process.

[0062] In summary, this application uses a smart head-mounted device to identify the question the user is actually looking at and a handwriting tablet to recognize the user's handwriting. Compared to online answering, this application can collect the question and answer data when the user answers paper questions, making it applicable to more learning scenarios. Furthermore, compared to scanning or photographing paper questions, this application can obtain the student's answer process data in real time without requiring the user to perform additional repetitive operations, resulting in a high degree of automation and simple operation.

[0063] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device or learning process data acquisition device capable of performing the above functions. The following description uses a learning process data acquisition device as an example to illustrate this embodiment and the subsequent embodiments.

[0064] Based on this, embodiments of this application provide a method for collecting data during the learning process, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the data acquisition method for the learning process in this application.

[0065] In this embodiment, the learning process data acquisition method includes steps S10 to S30:

[0066] Step S10: Receive the gaze image output by the smart head-mounted device;

[0067] It should be noted that, in this embodiment, the smart head-mounted device is communicatively connected to the terminal device for collecting learning process data. The smart head-mounted device continuously outputs data captured by the camera to the learning process data collection device for data acquisition. The smart head-mounted device includes, but is not limited to, smart glasses, Mixed Reality (MR) devices (e.g., MR glasses or MR helmets), Augmented Reality (AR) devices (e.g., AR glasses or AR helmets), Virtual Reality (VR) devices (e.g., VR glasses or VR helmets), Extended Reality (XR) devices, or some combination thereof.

[0068] In this embodiment, when the target user is learning, they need to wear a smart head-mounted device to capture the images seen by the target user. The learning process data acquisition device acts as the receiving end, which mainly captures and receives gaze images transmitted from the smart head-mounted device. These gaze images are captured by the smart head-mounted device through its built-in camera or other sensors and reflect the images currently seen by the target user (i.e., the user wearing the smart head-mounted device).

[0069] Step S20: Determine the gaze point of the target user from the gaze image and obtain the target question data at the gaze point, wherein the target user is wearing the smart head-mounted device;

[0070] In this embodiment, the system utilizes image processing techniques and algorithms to analyze the received gaze image to determine the target user's gaze point. After determining the gaze point, text recognition technology can be used to identify the content of the question corresponding to the user's gaze point. Furthermore, the system can retrieve the answer to the question, the corresponding knowledge points, and other data from relevant data sources (such as electronic documents, web pages, or databases).

[0071] In this embodiment, there are multiple ways to determine the gaze point. For example, when the camera of the smart head-mounted device is located at the midpoint between the user's two eyes, the midpoint of the captured gaze image can be directly used as the user's gaze point.

[0072] Specifically, in one feasible implementation, the step of determining the target user's gaze point from the gaze image in step S20 above includes steps S21 to S25:

[0073] Step S21: Obtain the depth information of the gaze image and obtain the focal length of the camera of the smart head-mounted device;

[0074] It should be noted that, in this embodiment, the cameras used to capture external images on various existing smart head-mounted devices are usually not positioned between the user's eyes, but rather on the side. Therefore, the gaze image captured by the camera is offset from the image actually seen by the user. Thus, it is necessary to correct the point of the user's actual gaze by measuring the amount of offset, thereby avoiding recognition errors caused by camera offset.

[0075] In this embodiment, depth information of each pixel in the gaze image, i.e., the distance of these pixels relative to the camera, is obtained through sensors (such as depth sensors or infrared sensors) or a depth camera on the smart head-mounted device. Simultaneously, the focal length information of the camera also needs to be obtained, which is typically accomplished by querying the camera's technical specifications or utilizing internal system calibration data.

[0076] Step S22: Determine the ratio between the depth information and the focal length as the scaling factor of the camera;

[0077] In this embodiment, the ratio between the acquired depth information and focal length information is calculated, and this ratio is the zoom factor of the camera. The zoom factor reflects the relationship between the size of the object in the image captured by the camera and the actual size of the object.

[0078] Step S23: Obtain the first relative offset value between the camera and the center points of the target user's eyes;

[0079] In this embodiment, the relative positional relationship between the camera and the center point of the target user's eyes can be known through the specifications of the smart head-mounted device, namely the first relative offset value. This value describes the positional offset of the camera relative to the center point of the user's eyes.

[0080] Step S24: Determine a second relative offset value between the gaze point and the image midpoint of the gaze image based on the first relative offset value and the scaling factor;

[0081] In this embodiment, the offset of the gaze point relative to the center point in the image can be calculated based on the calculated scaling factor and the first relative offset value. This offset value reflects the positional relationship between the point actually gazed at by the user and the center of the image captured by the camera.

[0082] Step S25: The point in the gaze image that is offset by a second relative offset value relative to a point in the image is taken as the user's gaze point.

[0083] In this embodiment, based on the calculated second relative offset value, a point offset by that value relative to the center point of the gaze image is found in the gaze image and determined as the user's gaze point. This gaze point represents the user's visual focus when viewing the gaze image.

[0084] As an example, please refer to Figure 2 , Figure 2 This is a first scene diagram related to the implementation of the learning process data acquisition method of this application, such as... Figure 2 As shown, when the smart camera is set at point A on the far left of the smart glasses, the center point of the image captured by the camera according to the field of view is A', the center point of the two lenses of the glasses (which can be regarded as the center of the line of sight) is B, and the relative position of point A and point B is X. We need to calculate the position of B' on the corresponding image.

[0085] For a real object, the position of B' is equal to the position X that A moves from to B.

[0086] However, please refer to Figure 3 , Figure 3 This is the second scene diagram involved in the implementation of the learning process data acquisition method of this application. Since the photograph is a scaled-down version of the actual object, there is a certain scaling relationship between the focal plane at the time of shooting and the size of the actual object. Therefore, the position of B' is equal to X multiplied by the scaling factor between the photograph and the actual object.

[0087] Please follow Figure 4 , Figure 4 This is a third scene diagram related to the implementation of the learning process data acquisition method of this application, such as... Figure 4As shown, the depth characteristics of objects in the shooting scene can be obtained through a depth camera or distance sensor. The length of AE in the figure can be known, and the length of AF is the camera focal length. AE / AF can be calculated as the camera's scaling factor. Then, the position of point B' can be obtained by A'+X / (AE / AF). That is, by calculating the X displacement in the vertical and horizontal directions, the point that the user is actually observing can be calculated by superimposing the two.

[0088] In another feasible implementation, after step S24 described above, the method may further include steps S26 to S28:

[0089] Step S26: Receive the gaze angle and gaze depth of the target user output by the eye-tracking module of the smart head-mounted device;

[0090] It should be noted that in actual use, users may not always be looking straight ahead, but may adjust their gaze by moving their eyes. Therefore, when considering the user's actual gaze point, it is also necessary to consider the user's eye data.

[0091] In this embodiment, data is received from the eye-tracking module in the smart head-mounted device. The eye-tracking module can capture and measure the user's eye movements in real time, including the gaze angle (i.e., the angle of rotation of the eyeball relative to the head) and the gaze depth (i.e., the distance between the eye's focal point and the screen or object).

[0092] Step S27: Calculate the actual offset value of the gaze point based on the gaze angle, gaze depth, and the second relative offset value;

[0093] In this embodiment, the actual offset value of the gaze point is calculated by combining the received gaze angle and gaze depth data with the calculated second relative offset value. This actual offset value represents the user's true offset relative to the center point of the image captured by the camera in three-dimensional space.

[0094] Step S28: The point in the gaze image that is offset from the actual offset value relative to the point in the image is taken as the user's gaze point.

[0095] In this embodiment, based on the calculated actual offset value, a point offset by that value relative to the center point of the gaze image is found in the gaze image and determined as the user's final gaze point. This gaze point considers not only the two-dimensional offset in the image captured by the camera, but also the user's actual gaze position and depth in three-dimensional space.

[0096] As an example, specifically, please refer to Figure 5 , Figure 5 This is a fourth scene diagram related to the implementation of the learning process data acquisition method of this application, such as... Figure 5As shown, ∠HGB is the angle of the user's line of sight, HG is the depth obtained, HG = AE, and the length of AB is the distance from the camera position to the center of the two lenses of the glasses. It is known that EG = AH = AB – HB can be obtained through trigonometric functions.

[0097] HB = AE * tan∠HGB;

[0098] EG = AB – AE * tan∠HGB;

[0099] Then calculate EG' on the photo based on the scaling ratio;

[0100] EG' = A' + EG / (AE / AF);

[0101] The calculated EG' is the actual offset.

[0102] The above are only two feasible implementations of the step S20 of determining the gaze point of the target user from the gaze image provided in this embodiment. This embodiment does not specifically limit the specific implementation of the step S10 of determining the gaze point of the target user from the gaze image.

[0103] It should be noted that the above is only one way of setting up the camera provided in this embodiment, and this embodiment does not specifically limit the specific way of setting up the camera.

[0104] Furthermore, in one feasible implementation, the step of obtaining the target question data at the fixation point in step S20 above includes steps A10 to A30:

[0105] Step A10: Identify at least one title region in the gaze image;

[0106] In this embodiment, a single frame typically includes multiple questions. Therefore, it is necessary to analyze the received gaze image to identify areas within the image that may contain questions. These question areas usually have specific visual characteristics, such as specific layout, color, font size, or contrast. These characteristics can be used to distinguish question areas from other image content.

[0107] Step A20: When the duration of the gaze point in the target area of ​​each of the title areas is greater than a preset first duration, the text data in the target area is identified by text recognition technology.

[0108] In this embodiment, the system continuously tracks the user's gaze and records the time the gaze remains in each question area. When the gaze remains in a question area (i.e., the target area) for more than a preset first duration, the system assumes the user is answering a question in that area and triggers the text recognition process. The system uses text recognition technology (such as Optical Character Recognition, OCR) to recognize the text data in the target area.

[0109] Step A30: Use the text data as the target title data at the gaze point.

[0110] In this embodiment, once the text data is recognized, it can be used as the target question data that the user is currently looking at.

[0111] In this embodiment, by identifying the question region in the gaze image and triggering the text recognition process when the user gazes at it for an extended period, the system can more accurately acquire target question data of interest to the user. This reduces the possibility of false recognition and missed recognition.

[0112] Step S30: Receive the target handwriting data output by the handwriting tablet, and associate the target handwriting data with the target question data and store them in a preset target storage space.

[0113] It should be noted that, in this embodiment, the handwriting tablet includes, but is not limited to, a handwriting tablet that allows direct contact writing on the screen using a capacitive pen, and a handwriting tablet that collects handwriting when paper is placed on the capacitive tablet and a pen is used to write on the paper.

[0114] In this embodiment, the device is connected to the handwriting tablet in real time and receives the handwriting data received by the handwriting tablet in real time. After recognizing the target question data, it can be assumed that the user has started writing the answer to the target question. At this time, the target handwriting data output by the handwriting tablet can be regarded as the user's answer to the target question, until it is recognized that the user has seen the next question or has not written for a long time. Therefore, by associating the target handwriting data with the target question data and storing it in the preset target storage space, the collection of user learning process data for the target question data can be completed.

[0115] Through the above steps, this application can collect the questions and answers of users when they answer paper questions, which can be applied to more learning scenarios. Furthermore, it can obtain students' answer process data in real time without requiring users to perform additional repetitive operations. It has a high degree of automation and is easy to operate.

[0116] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description and will not be repeated hereafter. Furthermore, after step S20 described above, the method may further include steps B10 to B20:

[0117] Step B10: Obtain the motion path of the gaze point within a preset time window;

[0118] In this embodiment, the system records and analyzes the motion path of the user's gaze point within a preset sliding time window. This time window can be a fixed period of time, such as 5 seconds, 10 seconds, or longer, depending on the application scenario and user needs. The system constructs the motion path of the gaze point by continuously tracking the user's gaze point and recording its positional changes in two-dimensional or three-dimensional space.

[0119] Step B20: If the motion amplitude in the motion path is less than a preset amplitude threshold, the first prompt message is output through the prompt module.

[0120] It should be noted that in this embodiment, if the user's gaze point moves very little over a period of time, it may indicate that the user is daydreaming or spacing out. In this case, the user can be reminded to continue studying.

[0121] In this embodiment, after obtaining the motion path of the gaze point, the system calculates the motion amplitude in the motion path, that is, the maximum distance or angle of the change in position of the gaze point within the time window. If the calculated motion amplitude is less than a preset amplitude threshold, the system assumes that the user may be in a daze or distracted state, and triggers the prompt module to output the first prompt message.

[0122] The prompting module can be an audio output device, a visual display device, or a vibration device in a smart head-mounted device. When the system detects that the amplitude of the gaze point movement is less than a threshold, the prompting module will output the first prompt information, such as a voice prompt, visual prompt, or vibration prompt, to remind the user to perform other operations.

[0123] By monitoring the movement path of the gaze point, the system can more accurately determine whether the user is in a daze, which helps the system provide more precise interactive feedback.

[0124] Based on the first and / or second embodiments of this application, in the third embodiment of this application, the content that is the same as or similar to the first and / or second embodiments described above can be referred to the above description and will not be repeated hereafter. Furthermore, the method may further include steps C10 to C20:

[0125] Step C10: Receive the target user's eye-closing time output by the eye-tracking module;

[0126] In this embodiment, the system receives data from the eye-tracking module, which includes information about the target user's eye-closing time. By monitoring the user's eye movements and state, the eye-tracking module can accurately determine whether the user has closed their eyes and the duration of eye-closing.

[0127] Eye-tracking modules typically use infrared or visible light cameras and image processing algorithms to track eye movements. When a user closes their eyes, eye movements significantly decrease or stop, and eyelid occlusion also causes specific areas in the image to darken or disappear. The system uses these features to detect eye-closing events and calculate the duration of eye closure.

[0128] Step C20: If the time the eyes are closed is greater than the preset second duration, then the second prompt message is output through the prompt module.

[0129] It should be noted that in this embodiment, when a user is drowsy, the average blinking time will be longer. Therefore, the user's drowsy state can be determined by detecting the duration of blinking.

[0130] In this embodiment, after receiving the eye-closing time data, the system compares it with a preset second duration. If the eye-closing time is greater than the second duration, the system assumes that the user may be fatigued, drowsy, or in need of rest, and triggers the prompt module to output a second prompt message.

[0131] When users are fatigued or drowsy, the system's prompts can remind them to take a short break or adjust their posture, thereby improving user comfort and satisfaction.

[0132] Based on the first to third embodiments of this application, in the fourth embodiment of this application, the content that is the same as or similar to the first to third embodiments described above can be referred to the above description and will not be repeated hereafter. Furthermore, after step S30 above, the method may further include steps D10 to D40:

[0133] Step D10: Upon receiving the target handwriting data output by the handwriting tablet, start timing the time for answering the question;

[0134] In this embodiment, when the system receives the target handwriting data written by the user from the handwriting tablet, that is, when the user begins to perform writing operations on the handwriting tablet (such as doing exercises), the system starts timing and records the time from when the user starts doing exercises.

[0135] Step D20: When no target handwriting data is received from the handwriting tablet, mark the current time as the first time point and start timing the pause duration.

[0136] In this embodiment, when the user stops writing on the handwriting tablet, that is, when the system does not receive the target handwriting data output by the handwriting tablet within a certain period of time, the system considers that the user has stopped writing. At this time, the system marks the current moment as the first time point and starts timing to record the length of time the user has stopped writing.

[0137] Step D30: If the pause time exceeds a preset third duration, and the distance between the received new handwriting data and the target handwriting data exceeds a preset distance threshold, then stop timing the question-solving time at the first time point to obtain the question-solving time.

[0138] In this embodiment, if the user resumes writing after a period of inactivity (exceeding a preset third time interval), but the new handwriting data is spatially far from the previous target handwriting data (exceeding a preset distance threshold), the system assumes the user may have completed the current question and started answering the next question or performed other operations. At this point, the system stops timing the question-answering time at the first available time, obtaining the total time required for the user to complete the current question.

[0139] Step D40, or, if the pause time exceeds a preset fourth duration, then stop timing the problem-solving time at the first time point to obtain the problem-solving duration, wherein the fourth duration is longer than the third duration.

[0140] In this embodiment, in another scenario, if the user does not resume writing within a certain period after stopping writing (exceeding a preset fourth time interval, and the fourth time interval being longer than the third time interval), the system also assumes that the user may have completed the answer to the current question or performed other operations. In this case, the system also stops timing the question-answering time at the first point in time, obtaining the total time required for the user to complete the current question.

[0141] By monitoring users' writing actions and pause times, the system can more accurately determine whether a user has completed the current question and calculate the time spent on the task. This helps users better understand their problem-solving speed and efficiency. Furthermore, by recording information such as the user's problem-solving time and pause times, the system can provide personalized learning suggestions and resource recommendations. For example, for users who are slow at solving problems, the system can recommend more practice questions to improve their speed; for users who are easily distracted, the system can remind them to stay focused.

[0142] In addition, the collected handwriting data can be used to retrieve the target question data through external databases to obtain the correct answers. The correct answers can be compared with the answers written in the handwriting data to see if the user's answers are correct, whether the answering ideas are concise, etc. The questions can also be classified according to the knowledge points of the retrieved target questions to analyze the user's mastery of different knowledge points, thereby comprehensively analyzing the user's learning situation.

[0143] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the data collection method of the learning process in this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0144] This application also provides a learning process data acquisition system; please refer to [reference needed]. Figure 6 The learning process data acquisition system includes:

[0145] Acquisition module 10 is used to receive gaze images output by the smart head-mounted device;

[0146] The question data acquisition module 20 is used to determine the gaze point of the target user from the gaze image and acquire the target question data at the gaze point, wherein the target user is wearing the smart head-mounted device;

[0147] The handwriting data acquisition module 30 is used to receive the target handwriting data output by the handwriting tablet and associate the target handwriting data with the target question data and store it in a preset target storage space.

[0148] Optionally, the question data acquisition module is also used for:

[0149] Obtain the depth information of the gaze image and the focal length of the camera of the smart head-mounted device;

[0150] The ratio between the depth information and the focal length is determined as the scaling factor of the camera;

[0151] Obtain the first relative offset value between the camera and the center points of the target user's eyes;

[0152] A second relative offset value between the gaze point and the image midpoint of the gaze image is determined based on the first relative offset value and the scaling factor;

[0153] The point in the gaze image that is offset by a second relative offset value relative to a point in the image is taken as the user's gaze point.

[0154] Optionally, the question data acquisition module is also used for:

[0155] Receive the gaze angle and gaze depth of the target user output by the eye-tracking module of the smart head-mounted device;

[0156] The actual offset value of the gaze point is calculated based on the gaze angle, gaze depth, and the second relative offset value;

[0157] The point in the gaze image that is offset from a point in the image by the actual offset value is taken as the user's gaze point.

[0158] Optionally, the question data acquisition module is also used for:

[0159] Identify at least one title region in the gaze image;

[0160] When the duration of the gaze point being in the target area of ​​each of the title areas is greater than a preset first duration, text data in the target area is recognized by text recognition technology;

[0161] The text data is used as the target title data at the gaze point.

[0162] Optionally, the learning process data acquisition system is also used for:

[0163] The motion path of the gaze point is acquired within a preset time window;

[0164] If the amplitude of the motion in the motion path is less than a preset amplitude threshold, the prompting module will output a first prompt message.

[0165] Optionally, the learning process data acquisition system is also used for:

[0166] Receive the target user's eye-closing time output by the eye-tracking module;

[0167] If the time the eyes are closed is longer than the preset second duration, then the prompting module outputs a second prompt message.

[0168] Optionally, the learning process data acquisition system is also used for:

[0169] Upon receiving the target handwriting data output by the handwriting tablet, the timer for the question-solving period begins;

[0170] If the target handwriting data output by the handwriting tablet is not received, the current time is marked as the first time point, and the timer for the pause in writing begins;

[0171] If the pause in writing exceeds a preset third duration, and the distance between the received new handwriting data and the target handwriting data exceeds a preset distance threshold, then the timing of the question-solving time is stopped at the first time point to obtain the question-solving time.

[0172] Alternatively, if the pause in writing exceeds a preset fourth duration, the timing of the problem-solving time is stopped at the first time point to obtain the problem-solving duration, wherein the fourth duration is longer than the third duration.

[0173] The learning process data acquisition system provided in this application, employing the learning process data acquisition method in the above embodiments, can solve the technical problem of how to collect student learning process data. Compared with the prior art, the beneficial effects of the learning process data acquisition system provided in this application are the same as those of the learning process data acquisition method provided in the above embodiments, and other technical features of the learning process data acquisition system are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0174] This application provides a learning process data acquisition device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the learning process data acquisition method in the first embodiment described above.

[0175] The following is for reference. Figure 7 The diagram illustrates a structural schematic of a learning process data acquisition device suitable for implementing embodiments of this application. The learning process data acquisition device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 7 The learning process data acquisition device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0176] like Figure 7As shown, the learning process data acquisition device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the learning process data acquisition device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the learning process data acquisition device to communicate wirelessly or wiredly with other devices to exchange data. Although a learning process data acquisition device with various systems is shown in the figure, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems can be implemented alternatively.

[0177] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0178] The learning process data acquisition device provided in this application, employing the learning process data acquisition method in the above embodiments, can solve the technical problem of how to collect student learning process data. Compared with the prior art, the beneficial effects of the learning process data acquisition device provided in this application are the same as those of the learning process data acquisition method provided in the above embodiments, and other technical features in this learning process data acquisition device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0179] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0180] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0181] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the learning process data acquisition method in the above embodiments.

[0182] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0183] The aforementioned computer-readable storage medium may be included in the learning process data acquisition device; or it may exist independently and not assembled into the learning process data acquisition device.

[0184] The aforementioned computer-readable storage medium carries one or more programs. When the aforementioned one or more programs are executed by the learning process data acquisition device, the learning process data acquisition device causes the following: to receive a gaze image output by a smart head-mounted device; to determine the gaze point of a target user from the gaze image and to acquire target question data at the gaze point, wherein the target user is wearing the smart head-mounted device; to receive target handwriting data output by a handwriting tablet and to associate the target handwriting data with the target question data and store them in a preset target storage space.

[0185] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0186] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0187] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0188] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described learning process data acquisition method, thereby solving the technical problem of how to collect student learning process data. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the learning process data acquisition method provided in the above embodiments, and will not be repeated here.

[0189] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the learning process data acquisition method described above.

[0190] The computer program product provided in this application can solve the technical problem of how to collect student learning process data. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as the beneficial effects of the learning process data collection method provided in the above embodiments, and will not be repeated here.

[0191] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A method for collecting data during a learning process, characterized in that, The method includes: Receive gaze images output from smart head-mounted devices; The gaze point of the target user is determined from the gaze image, and the target question data at the gaze point is obtained, wherein the target user is wearing the smart head-mounted device; Receive the target handwriting data output by the handwriting tablet, and associate the target handwriting data with the target question data and store it in a preset target storage space.

2. The learning process data acquisition method as described in claim 1, characterized in that, The step of determining the gaze point of the target user from the gaze image includes: Obtain the depth information of the gaze image and the focal length of the camera of the smart head-mounted device; The ratio between the depth information and the focal length is determined as the scaling factor of the camera; Obtain the first relative offset value between the camera and the center points of the target user's eyes; A second relative offset value between the gaze point and the image midpoint of the gaze image is determined based on the first relative offset value and the scaling factor; The point in the gaze image that is offset by a second relative offset value relative to a point in the image is taken as the user's gaze point.

3. The learning process data acquisition method as described in claim 2, characterized in that, After the step of determining a second relative offset value between the gaze point and the image midpoint of the gazed image based on the first relative offset value and the scaling factor, the method further includes: Receive the gaze angle and gaze depth of the target user output by the eye-tracking module of the smart head-mounted device; The actual offset value of the gaze point is calculated based on the gaze angle, gaze depth, and the second relative offset value; The point in the gaze image that is offset from a point in the image by the actual offset value is taken as the user's gaze point.

4. The learning process data acquisition method as described in claim 1, characterized in that, The step of acquiring the target question data at the fixation point includes: Identify at least one title region in the gaze image; When the duration of the gaze point in the target area of ​​each of the title areas is greater than a preset first duration, the text data in the target area is recognized by text recognition technology; The text data is used as the target title data at the gaze point.

5. The learning process data acquisition method as described in claim 3, characterized in that, After the step of determining the gaze point of the target user from the gaze image, the method further includes: The motion path of the gaze point is acquired within a preset time window; If the motion amplitude in the motion path is less than a preset amplitude threshold, the prompting module will output a first prompt message.

6. The learning process data acquisition method as described in claim 5, characterized in that, The method further includes: Receive the target user's eye-closing time output by the eye-tracking module; If the time the eyes are closed is longer than the preset second duration, then the second prompt message is output through the prompt module.

7. The learning process data acquisition method as described in claim 1, characterized in that, After the step of associating and storing the target handwriting data with the target question data in a preset target storage space, the method further includes: Upon receiving the target handwriting data output by the handwriting tablet, the timer for the question-solving period begins; If the target handwriting data output by the handwriting tablet is not received, the current time is marked as the first time point, and the timer for the pause in writing begins; If the pause in writing exceeds a preset third duration, and the distance between the received new handwriting data and the target handwriting data exceeds a preset distance threshold, then the timing of the question-solving time is stopped at the first time point to obtain the question-solving time. Alternatively, if the pause in writing exceeds a preset fourth duration, the timing of the problem-solving time is stopped at the first time point to obtain the problem-solving duration, wherein the fourth duration is longer than the third duration.

8. A learning process data acquisition system, characterized in that, The system includes: The acquisition module is used to receive gaze images output by the smart head-mounted device; The question data acquisition module is used to determine the gaze point of the target user from the gaze image and acquire the target question data at the gaze point, wherein the target user is wearing the smart head-mounted device; The handwriting data acquisition module is used to receive the target handwriting data output by the handwriting tablet and associate the target handwriting data with the target question data and store it in a preset target storage space.

9. A learning process data acquisition device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the learning process data acquisition method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the learning process data acquisition method as described in any one of claims 1 to 7.

Images