Method, device, equipment and storage medium for collecting homework learning information

Abstract

The present application discloses a text processing method, device, equipment and storage medium. The scene of the reviewer annotating a paper homework document is photographed by a camera to obtain the photographed video, and then each video frame in the video is identified to obtain the reviewer's handwritten annotation content and the identity information of the respondent contained in each video frame. Based on the identity information of the respondent in each video frame identified, the photographed video is segmented to obtain a video frame sequence corresponding to each identity of the respondent. Based on the reviewer's handwritten annotation content contained in each video frame in the video frame sequence corresponding to each identity of the respondent, the homework learning situation of each identity of the respondent can be analyzed. The solution provided by the present application is imperceptible to the reviewer, and does not require the reviewer to change the reviewer's annotation process and habits, nor does it require the reviewer to manually take pictures of the annotated paper homework document, thereby reducing the reviewer's workload.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and more specifically, to a method, device, equipment and storage medium for collecting homework learning information. Background Art

[0002] With the development of video technology and data processing technology, more and more traditional manual tasks can be accomplished by machine operations. For example, in some teaching scenarios, students can answer questions on paper documents, and teachers can make handwritten annotations on students' answers, such as marking the correctness of students' answers by annotating "√" or "×". Among them, the correctness of the answers of the same student to different questions, or the correctness of the answers of different students to the same question, can be called homework learning information. By statistically analyzing the homework learning information, we can obtain students' learning characteristics, grades, etc., which is of great significance for understanding students' learning status and guiding teachers to adjust teaching methods.

[0003] The traditional method generally relies on teachers to conduct offline statistics to obtain homework learning information. This method is time-consuming and labor-intensive, and it is easy to miss something if it relies solely on human memory. To this end, the existing technology currently uses teachers to manually take photos of the corrected paper documents through a camera and upload them to relevant analysis equipment to obtain homework learning information. Obviously, this method requires teachers to manually take photos of each student's homework after correcting the homework, which undoubtedly increases the teacher's workload. Summary of the invention

[0004] In view of the above problems, this application is proposed to provide a method, device, equipment and storage medium for collecting homework learning information, so as to reduce the operation content of teachers in the process of collecting homework learning information and reduce the workload of teachers. The specific scheme is as follows:

[0005] In a first aspect, a method for collecting homework learning information is provided, comprising:

[0006] Acquire a video captured by a camera of a scene in which a reviewer annotates a paper homework document, wherein the paper homework document contains identity information of the respondent;

[0007] Identify each video frame in the video to obtain the reviewer's handwritten annotation content and the identity information of the respondent contained in each video frame;

[0008] Segmenting the video based on the identity information of the respondent to obtain a video frame sequence corresponding to each respondent with an identity;

[0009] Based on the reviewer's handwritten annotations contained in each video frame in the video frame sequence corresponding to each respondent, the homework learning status of each respondent is obtained.

[0010] In a second aspect, a homework learning situation collection device is provided, comprising:

[0011] A video acquisition unit, used to acquire a video shot by a camera of a scene in which a reviewer annotates a paper homework document, wherein the paper homework document contains identity information of the respondent;

[0012] A video frame recognition unit, used to recognize each video frame in the video, and obtain the reviewer's handwritten annotation content and the identity information of the respondent contained in each video frame;

[0013] A video frame segmentation unit, configured to segment the video based on the identity information of the respondent to obtain a video frame sequence corresponding to each respondent with the same identity;

[0014] The homework learning status determination unit is used to obtain the homework learning status of each respondent based on the reviewer's handwritten annotation content contained in each video frame in the video frame sequence corresponding to the respondent of each identity.

[0015] In a third aspect, a homework learning situation collection device is provided, including: a camera, a data processing module and a display module;

[0016] The camera is used to shoot a scene in which the reviewer annotates a paper homework document to obtain a shot video, wherein the paper homework document contains the identity information of the respondent;

[0017] The data processing module is used to execute each step of the above-mentioned homework learning situation collection method;

[0018] The display module is used to display the video captured by the camera in real time.

[0019] In a fourth aspect, a storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the various steps of the above-mentioned homework learning situation collection method are implemented.

[0020] By means of the above technical solution, the present application does not require to change the reviewer's annotation process and habits. The paper homework document after the respondent's answer is annotated in the normal annotation method. The present application uses a camera to shoot the scene of the reviewer annotating the paper homework document, obtains the shot video, and then identifies each video frame in the video to obtain the reviewer's handwritten annotation content contained in each video frame and the respondent's identity information (the paper homework document may contain the respondent's identity information). It can be understood that the reviewer can sequentially annotate the paper homework documents of multiple respondent's different identities in the annotation scenario. In order to conduct statistics on the homework learning status of different respondent, the shot video can be segmented based on the identity information of the respondent in each video frame identified to obtain a video frame sequence corresponding to each respondent with each identity. Based on the reviewer's handwritten annotation content contained in each video frame in the video frame sequence corresponding to each respondent with each identity, the homework learning status of each respondent with each identity can be analyzed.

[0021] The homework learning situation collection solution provided by this application is imperceptible to the reviewer, that is, there is no need to change the reviewer's annotation process and habits, nor is there a need for the reviewer to manually take photos of the annotated paper homework documents. This reduces the reviewer's operations during the homework learning situation collection process and reduces the reviewer's workload. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0023] Figure 1 A schematic diagram of the structure of the homework learning situation collection device provided in the embodiment of the present application;

[0024] Figure 2 A schematic diagram of a process of collecting homework learning information provided in an embodiment of the present application;

[0025] Figure 3 An example of a paper work document with reviewer comments is provided;

[0026] Figure 4 Another schematic diagram of the process of collecting homework learning information provided by the embodiment of the present application;

[0027] Figure 5 This is a schematic diagram of the structure of a homework learning situation collection device disclosed in an embodiment of the present application. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0029] In all industries, there may be a need to collect and analyze homework learning information. Homework learning information refers to the reviewer's annotations on the answer after the respondent answers the question, such as the correctness of the answer result, etc. By collecting the reviewer's comments, the respondent's homework learning information can be obtained, such as the correctness of the same respondent's answers to different questions, or the correctness of different respondent's answers to the same question. By collecting homework learning information, the respondent's learning characteristics and achievements can be obtained, which is of great significance for understanding the respondent's learning status and guiding teachers to adjust teaching methods.

[0030] A typical scenario is that in a teaching scenario, the teacher annotates the answer papers of each student in the class, and obtains the students' homework and learning situation by collecting the content of the teacher's annotations. Of course, in addition to this, there may be a need to collect homework and learning situation in other scenarios, such as in the written test scenario of the company's application process, the written examination officer annotates the applicant's answers, and by collecting the written examination officer's annotations, the applicant's homework and learning situation can be obtained, and so on. This application is not limited to the two scenarios in the above examples. Other scenarios that require reviewers to annotate the paper homework documents answered by the respondent can all be applied to the homework and learning situation collection solution of this application.

[0031] This application provides a homework learning situation collection device, combined with Figure 1 As shown, the homework learning situation collection device may include: a camera 10, a data processing module ( Figure 1 ) and a display module 11, wherein:

[0032] The camera 10 can be fixed on the work surface by a bracket, and the reviewer can annotate the paper work document on the work surface. By adjusting the height and angle of the bracket, the camera can capture the complete paper work document.

[0033] Generally, in order to distinguish paper assignment documents of different respondents, the paper assignment documents may include the respondent's identity information, such as name, student ID number, signature, seal, etc., which can reflect the respondent's identity.

[0034] The camera 10 captures the scene of the reviewer annotating the paper work document to obtain a captured video.

[0035] The display module 11 can be connected to the camera to display the video captured by the camera 10 in real time, so that the reviewer can understand the range and clarity of the camera shooting, so as to adjust the camera angle or the position of the paper work document in time.

[0036] The data processing module can be a server set up in the background, or it can be integrated with the bracket where the camera is located, or integrated with the display module 11. For example, the data processing module and the display module 11 are integrated together, and electronic devices with data processing and display functions can be used, such as tablet computers, notebooks and other electronic devices.

[0037] The data processing module is used to identify each video frame in the video captured by the camera, obtain the reviewer's handwritten annotation content and the identity information of the respondent contained in each video frame, further segment the video based on the respondent's identity information, obtain a video frame sequence corresponding to each respondent's identity, and obtain the homework learning situation of each respondent's identity based on the reviewer's handwritten annotation content contained in each video frame in the video frame sequence corresponding to the respondent's identity.

[0038] Obviously, according to the homework learning situation collection device provided by this application, the reviewer's annotation scene can be automatically photographed through a camera, and the homework learning situation of the respondent can be obtained after processing by a data processing module. For the reviewer, it is imperceptible, that is, there is no need to change the reviewer's annotation process and habits, nor is there a need for the reviewer to manually take a photo of the annotated paper homework document, thereby reducing the reviewer's operation content during the homework learning situation collection process and reducing the reviewer's workload.

[0039] Among them, the detailed processing logic of the data processing module refers to the homework learning situation collection method introduced below.

[0040] Next, from the perspective of the data processing module, the homework learning situation collection method of this application is introduced. Figure 2 As shown, the homework learning situation collection method of the present application may include the following steps:

[0041] Step S100: Obtain a video captured by a camera of a scene in which a reviewer annotates a paper work document.

[0042] The paper homework document contains the identity information of the respondent.

[0043] like Figure 3 , which illustrates a paper homework document annotated by a reviewer, which includes printed test questions, answers of the respondents, identity information of the respondents, such as name M1, and handwritten comments of the reviewer M2.

[0044] The identity information of the respondent is used to distinguish different respondent, and may be the respondent's name, student ID, serial number, etc.

[0045] The reviewer's handwritten comments can generally be distinguished by using a pen of a different color from the respondent's. For example, in the scenario of a teacher correcting homework, it is common for students to use black or blue pens to answer, and teachers to use red pens to make comments.

[0046] Step S110: Identify each video frame in the video to obtain the reviewer's handwritten annotation content and the identity information of the respondent contained in each video frame.

[0047] It is understandable that the reviewer may need to review paper homework documents completed by multiple respondents with different identities. In order to obtain the homework status of respondents with different identities, each video frame contained in the video taken by the camera can be identified in this step to obtain the reviewer's handwritten comments contained in each video frame, as well as the respondent's identity information.

[0048] The process of recognizing the video frame may use OCR recognition technology, which supports distinguishing and recognizing printed text and handwriting of different colors, so as to obtain the reviewer's handwritten annotation content contained in the video frame and the identity information of the respondent.

[0049] In addition, a pre-trained neural network model can be used to extract the reviewer's handwritten comments and the identity information of the respondent in the video frame respectively.

[0050] Step S120: segment the video based on the identity information of the respondent to obtain a video frame sequence corresponding to each respondent's identity.

[0051] Specifically, the identity information of the respondent corresponding to each video frame has been analyzed in the previous step. Based on this, the identity information of the respondent can be used as the classification basis to segment the video. Video frames with the same identity information are classified into one category to form a video frame sequence. For example, during the entire video shooting process, the reviewer annotated the paper homework documents of three respondent. Then, through the video segmentation in this step, the video can be divided into three video frame sequences, corresponding to three different respondent: respondent 1 corresponds to video frame sequence A, respondent 2 corresponds to video frame sequence B, and respondent 3 corresponds to video frame sequence C.

[0052] Step S130, based on the reviewer's handwritten annotations contained in each video frame in the video frame sequence corresponding to each identity of the answerer, the homework learning status of each identity of the answerer is obtained.

[0053] Specifically, the reviewer's handwritten annotations contained in each video frame have been analyzed in the aforementioned step S110. In this step, for each identity of the respondent, the homework learning status of each identity of the respondent can be obtained based on the reviewer's handwritten annotations contained in each video frame in the corresponding video frame sequence.

[0054] Among them, the homework learning situation can be the correctness of the answer of the test taker to each test question, the accuracy rate and error rate of the whole homework document, etc. The specific items that need to be counted in the homework learning situation can be set according to actual needs.

[0055] The homework learning situation collection method provided in the embodiment of the present application does not require changing the reviewer's annotation process and habits. It is sufficient to annotate the paper homework document after the respondent's answer in a normal annotation manner. The present application uses a camera to shoot the scene of the reviewer annotating the paper homework document, obtains the shot video, and then recognizes each video frame in the video to obtain the reviewer's handwritten annotation content contained in each video frame and the respondent's identity information (the paper homework document may contain the respondent's identity information). It can be understood that the reviewer can sequentially annotate the paper homework documents of multiple respondent's different identities in the annotation scenario. In order to conduct homework learning situation statistics for different respondent's different identities, the shot video can be segmented based on the identity information of the respondent of each identified video frame to obtain a video frame sequence corresponding to each respondent's identity. Based on the reviewer's handwritten annotation content contained in each video frame in the video frame sequence corresponding to each respondent's identity, the homework learning situation of each respondent's identity can be analyzed.

[0056] The homework learning situation collection method provided in this application is imperceptible to the reviewer, that is, there is no need to change the reviewer's annotation process and habits, nor is there a need for the reviewer to manually take photos of the annotated paper homework documents. Therefore, the reviewer's operation content in the homework learning situation collection process is reduced, reducing the reviewer's workload.

[0057] Further optionally, after obtaining the homework and learning status of each respondent with each identity, the homework and learning status of the respondent with each identity can be summarized to obtain the overall homework and learning status.

[0058] In real-life scenarios, the reviewer's process of annotating paper homework documents has several different forms. One is the test paper form, that is, all paper homework documents are included in the scope of homework learning information collection. The other is the book form, such as an exercise book, which contains multiple pages of documents, of which only some pages may belong to the scope of homework learning information collection. For example, in an exercise book, the respondent only needs to answer pages n1-n2, and the corresponding homework learning information collection scope is also pages n1-n2.

[0059] In the second case, in order to ensure effective collection of homework learning information, the present application embodiment provides another homework learning information collection method, combined with Figure 4 As shown, the method may include:

[0060] Step S200, receiving an input collection range instruction, wherein the collection range instruction includes a document range for homework learning situation collection.

[0061] Specifically, when using the homework learning situation collection solution of this application, the user can enter a collection range instruction before starting the camera to shoot, and the instruction contains the document range for the homework learning situation collection this time. Among them, the document range can be the target page number identifier of the document to be collected, such as pages n1-n2, or it can also be the title, keywords, etc. of the document to be collected.

[0062] Step S210: Obtain a video captured by a camera of a scene in which a reviewer annotates a paper work document.

[0063] Step S220: Identify each video frame in the video to obtain the reviewer's handwritten annotation content, the respondent's identity information and the printed content contained in each video frame.

[0064] Specifically, compared to Figure 2 In the example process, in this embodiment, in addition to identifying the handwritten comments of the reviewer and the identity information of the respondent contained in the video frame, the printed content is further identified from the video frame. In the subsequent steps, the video frames within the acquisition range are filtered based on the printed content.

[0065] Step S230: segment the video based on the identity information of the respondent to obtain a video frame sequence corresponding to each respondent's identity.

[0066] Step S240: for the video frame sequence corresponding to each respondent, based on the printed content contained in each video frame in the video frame sequence, filter the video frames within the document range specified by the acquisition range instruction to form a filtered video frame sequence.

[0067] Specifically, the previous step obtains the video frame sequence corresponding to each respondent of each identity, which may contain video frames outside the document range specified by the acquisition range instruction. For example, before the teacher turns the pages in the exercise book to locate the document page to be annotated, the camera also captures the images of other pages, and the video frames corresponding to these images do not belong to the scope of this homework learning situation acquisition. For this reason, in this step, in the video frame sequence corresponding to each respondent of each identity, based on the printed content contained in each video frame, identify and filter out the video frames within the document range specified by the acquisition range instruction to form a filtered video frame sequence, thereby avoiding interference from the remaining video frames.

[0068] Step S250, based on the reviewer's handwritten annotation content contained in each video frame in the filtered video frame sequence corresponding to each identity of the answerer, the homework learning status of each identity of the answerer is obtained.

[0069] The homework learning situation collection method provided in this embodiment supports users to input homework learning situation collection range instructions to specify the document range to be collected. On this basis, for the video frame sequence corresponding to each identity of the respondent obtained by segmentation, the video frames within the specified document range are screened out to form a screened video frame sequence, and then the homework learning situation of the respondent of each identity is obtained based on the screened video frame sequence, so that the collection of homework learning situation is more accurate.

[0070] In some embodiments of the present application, the process of screening video frames within a specified document range in the above step S240 is introduced.

[0071] In this embodiment, the document range for the homework learning situation collection contained in the collection range instruction is taken as the target page number identifier of the document to be collected as an example for explanation. Regarding the screening method, this embodiment provides two optional implementation methods, which are introduced as follows:

[0072] The first one,

[0073] Based on the printed content contained in each video frame in the video frame sequence, the video frames containing the target page number identifier are screened out to form a screened video frame sequence.

[0074] For example:

[0075] The target page number is: 5 pages - 6 pages. Generally, the page number of each page is marked in printed form in a paper document. Therefore, by identifying the printed content contained in each video frame, the video frames containing the target page number "5 pages" and "6 pages" in the printed content can be filtered out, and the filtered video frames form a filtered video frame sequence.

[0076] The second type

[0077] Considering that the first method described above relies solely on the target page number identification for screening, there may be certain errors. For example, the target page number identification corresponding to the acquisition range specified by the user is 5 pages to 6 pages. On page 2 of the exercise book, the two characters "5 pages" appear in the test question part. If the first method is used, the video frame corresponding to page 2 may be mistakenly screened out and considered to be a video frame within the acquisition range.

[0078] To this end, this embodiment provides a more accurate implementation method as follows:

[0079] S1. Obtaining a text recognition result of an original paper job document corresponding to the target page number identifier stored in a preset database.

[0080] The original paper homework document is a document that does not contain handwritten content of the reviewer and the respondent, that is, the original paper homework document may only contain printed content, such as test question information.

[0081] In the specific implementation process, text recognition can be performed on each page of the original paper work document to obtain the text recognition result of each page, and the page number identification and the corresponding text recognition result can be stored in the database. The text recognition process can use OCR recognition technology or other text recognition technology.

[0082] On this basis, when the collection range instruction issued by the user is obtained, the corresponding text recognition result is searched in the database based on the target page number identifier contained in the instruction.

[0083] S2. Compare the printed content contained in each video frame in the video frame sequence with the text recognition results of the original paper work document, and select the video frames whose comparison results meet the set coincidence condition to form a screened video frame sequence.

[0084] Specifically, in order to filter out the video frames within the acquisition range from the video frame sequence, the printed content contained in each video frame can be compared with the text recognition results of the original paper work document corresponding to the target page number identifier. Considering the recognition error, in this embodiment, a coincidence condition can be set to filter out the video frames whose comparison results meet the set coincidence condition to form a filtered video frame sequence, which is used as the video frame sequence within the acquisition range.

[0085] The set overlap condition may be that the character overlap satisfies a set threshold, such as the character overlap is greater than or equal to 95% or other set values.

[0086] According to the method provided in this embodiment, by comparing the printed content contained in each video frame with the text recognition results of the original paper job document corresponding to the pre-stored target page number identifier, the video frames that meet the set overlap conditions are screened out and identified as video frames that belong to the acquisition range, and the judgment results are more accurate.

[0087] In some embodiments of the present application, the above step S130 is introduced to introduce the process of obtaining the homework learning status of each respondent based on the reviewer's handwritten annotation content contained in each video frame in the video frame sequence corresponding to the respondent of each identity.

[0088] It is understandable that as the video recording time increases, in the video frame sequence corresponding to the same respondent, the later video frames contain more handwritten comments from the reviewer, which are more useful for analyzing the learning status of the homework.

[0089] To this end, in this embodiment, an optional implementation of the above step S130 is provided, including:

[0090] S1. For the video frame sequence corresponding to each respondent, a key frame may be selected, wherein the key frame contains the handwritten annotation content of the reviewer.

[0091] Specifically, the number of reviewer handwritten annotation contents contained in the key frame may be set to meet a set number requirement.

[0092] The set quantity requirement may be a maximum quantity, or a quantity exceeding a preset threshold, etc.

[0093] S2. Based at least on the reviewer's handwritten annotations contained in the key frame, the homework learning status of each respondent is obtained.

[0094] Since the reviewer's handwritten annotation content contained in the key frame meets the set quantity requirement, that is, it contains a certain amount of reviewer's handwritten annotation content, it can provide rich data for the homework learning situation, especially when the set quantity requirement is the largest number, the key frame can contain the most reviewer's handwritten annotation content, which can provide the richest data for the homework learning situation. Therefore, based on the reviewer's handwritten annotation content contained in the key frame, the homework learning situation of each identity of the respondent can be analyzed.

[0095] Regarding how to obtain the learning status of the homework based on the reviewer's handwritten comments, corresponding data processing operations can be taken according to actual user needs, such as counting the number of marks marked as correct / wrong in the reviewer's handwritten comments, etc., which will not be described in detail in this application.

[0096] Furthermore, considering that the reviewer may block the document with his / her body when annotating the paper homework document, it may cause the body occlusion area in some video frames taken by the camera. In particular, if there is a body occlusion area in the key frame, it may affect the subsequent analysis of the homework learning situation.

[0097] In order to avoid the above problem, in this embodiment, another optional implementation of the above step S130 is provided, including:

[0098] S1. For the video frame sequence corresponding to each respondent, a key frame may be selected from the video frame, and the number of the reviewer's handwritten annotation content contained in the key frame meets the set number requirement.

[0099] S2. Perform body occlusion detection on the key frame. If a body occlusion area is detected, execute step S3.

[0100] The process of performing limb occlusion detection on key frames can be implemented by using a pre-trained limb detection model, wherein the limb detection model can be trained using image training data marked with limb occlusion area positions.

[0101] If a body occlusion area is detected in the key frame, in order to avoid affecting the subsequent collection of homework learning status, the following step S3 can be performed to obtain the supplementary frame process. Of course, if no body occlusion area is detected in the key frame, the homework learning status of each identity of the respondent can be obtained directly based on the reviewer's handwritten annotation content contained in the key frame according to the processing method of the above embodiment.

[0102] S3. Searching in the video frame sequence for a video frame where there is no occlusion at the same position as the limb occlusion area as a supplementary frame.

[0103] Specifically, when it is detected in the previous step that there is a body occlusion area in the key frame, a video frame without occlusion at the same position as the occlusion area can be searched in the video frame sequence corresponding to the respondent with the same identity, and the video frame can be used as a supplementary frame.

[0104] S4. Based on the reviewer's handwritten annotations contained in the key frame and the supplementary frame, the homework learning status of each respondent is obtained.

[0105] Since there is no limb occlusion area in the supplementary frame, the homework status of each respondent can be obtained more accurately based on the key frame and the supplementary frame.

[0106] It is understandable that the limb occlusion area in the key frame may be occluded printed content, such as test question information, etc., or it may be occluded handwritten annotation content of the reviewer. If it is the former, this application believes that it has little impact on the collection of homework learning conditions and can be ignored. If it is the latter, the occlusion has a greater impact on the collection of homework learning conditions. Therefore, the above S3 can be a process of searching for a video frame in the video frame sequence that is not occluded at the same position as the limb occlusion area as a supplementary frame only when it is detected that there is a limb occlusion area in the key frame and the limb occlusion area overlaps with the reviewer's handwritten annotation content.

[0107] The body occlusion area overlaps with the handwritten annotation content of the reviewer, indicating that the body occlusion area occludes the handwritten annotation content of the reviewer.

[0108] Of course, if in actual application, the user believes that the impact of obstructed printed content on the collection of homework learning conditions cannot be ignored, then the above S3 can also be a process of searching for supplementary frames as long as a limb obstruction area is detected in the key frame.

[0109] In some embodiments of the present application, in order to ensure that the supplementary frame found contains the reviewer's handwritten annotation content as much as possible, an optional implementation of the above step S3 is provided as follows:

[0110] S31. Searching the video frames for a set of candidate video frames in which no occlusion occurs at the same position as the limb occlusion region.

[0111] S32: Select a frame with the latest shooting time from the candidate video frame set as a supplementary frame.

[0112] It is understandable that the later the shooting time is, the higher the probability that the reviewer's handwritten annotation content is included. Therefore, in this embodiment, the frame with the latest shooting time is selected as the supplementary frame in the candidate video frame set. If the reviewer's handwritten annotation content is blocked in the key frame, it can be guaranteed that the supplementary frame found contains the blocked reviewer's handwritten annotation content.

[0113] Furthermore, the embodiment of the present application provides the above-mentioned step S4, which obtains several different implementation methods of the homework learning status of each respondent based on the reviewer's handwritten annotation content contained in the key frame and the supplementary frame.

[0114] The first one,

[0115] S41a, taking the reviewer's handwritten annotation content contained in the key frame as the first homework learning situation.

[0116] S42a, taking the reviewer's handwritten annotation content at the same position as the limb occluded area in the supplementary frame as the second homework learning situation, and combining the first homework learning situation and the second homework learning situation into the homework learning situation of each identity of the respondent.

[0117] The second type

[0118] S41b, cropping the image at the same position as the limb occlusion area in the supplementary frame, and overlaying the cropped sub-image on the limb occlusion area in the key frame to obtain an edited key frame.

[0119] S42b, based on the reviewer's handwritten annotations contained in the edited key frame, the homework learning status of each respondent is obtained.

[0120] The following is a description of the homework learning situation collection device provided in an embodiment of the present application. The homework learning situation collection device described below and the homework learning situation collection method described above can be referenced to each other.

[0121] See also Figure 5 , Figure 5 This is a schematic diagram of the structure of a homework learning situation collection device disclosed in an embodiment of the present application.

[0122] like Figure 5 As shown, the device may include:

[0123] The video acquisition unit 100 is used to acquire a video shot by a camera of a scene in which a reviewer annotates a paper homework document, wherein the paper homework document contains identity information of the reviewer;

[0124] The video frame recognition unit 110 is used to recognize each video frame in the video and obtain the reviewer's handwritten annotation content and the identity information of the respondent contained in each video frame;

[0125] The video frame segmentation unit 120 is used to segment the video based on the identity information of the respondent to obtain a video frame sequence corresponding to each respondent with the same identity;

[0126] The homework learning status determination unit 130 is used to obtain the homework learning status of each respondent based on the reviewer's handwritten annotation content contained in each video frame in the video frame sequence corresponding to the respondent of each identity.

[0127] In an optional case, the process of obtaining the homework learning status of each respondent based on the handwritten annotations of the reviewer contained in each video frame in the video frame sequence corresponding to each respondent may include:

[0128] For each identity of the respondent corresponding to the video frame sequence, a key frame is selected from the sequence, and the number of the reviewer's handwritten annotations contained in the key frame meets the set number requirement;

[0129] Based at least on the reviewer's handwritten annotations contained in the key frames, the homework learning status of each respondent is obtained.

[0130] In another optional case, the process of obtaining the homework learning status of each respondent based on the handwritten annotations of the reviewer contained in each video frame in the video frame sequence corresponding to each respondent may include:

[0131] For each identity of the respondent corresponding to the video frame sequence, a key frame is selected from the key frame, wherein the key frame contains the reviewer's handwritten annotation content, and the number of the reviewer's handwritten annotation content can meet the set quantity requirement, for example, the key frame contains the largest number of reviewer's handwritten annotation content;

[0132] Performing limb occlusion detection on the key frame, and if a limb occlusion area is detected, searching the video frame sequence for a video frame without occlusion at the same position as the limb occlusion area as a supplementary frame;

[0133] Based on the reviewer's handwritten annotations contained in the key frame and the supplementary frame, the homework learning status of each respondent is obtained.

[0134] Optionally, the device of the present application may further include:

[0135] A collection range instruction receiving unit, used to receive an input collection range instruction, wherein the collection range instruction includes a document range for which homework learning information is to be collected;

[0136] The video frame recognition unit recognizes each video frame in the video and obtains the handwritten annotation content of the reviewer and the identity information of the respondent contained in each video frame, and is also used to recognize the printed content contained in each video frame;

[0137] A video frame sequence screening unit is used to screen the video frames within the document range specified by the acquisition range instruction for the video frame sequence corresponding to each identity of the answerer before obtaining the homework learning situation of each identity of the answerer, based on the printed content contained in each video frame in the video frame sequence, to form a screened video frame sequence;

[0138] The process of obtaining the homework learning status of each respondent based on the handwritten annotations of the reviewer contained in each video frame in the video frame sequence corresponding to each respondent may include:

[0139] Based on the reviewer's handwritten annotations contained in each video frame in the filtered video frame sequence corresponding to each respondent, the homework learning status of each respondent is obtained.

[0140] Optionally, the document range for the homework learning situation collection contained in the above collection range instruction may be the target page number identifier of the document to be collected. On this basis, the embodiment of the present application provides two optional implementation methods of the above video frame sequence screening unit, which are as follows:

[0141] First, the video frame sequence screening unit screens the video frames within the document range specified by the acquisition range instruction based on the printed content contained in each video frame in the video frame sequence to form a screened video frame sequence, which may include:

[0142] Based on the printed content contained in each video frame in the video frame sequence, the video frames containing the target page number identifier are screened out to form a screened video frame sequence.

[0143] Second, the video frame sequence screening unit screens the video frames within the document range specified by the acquisition range instruction based on the printed content contained in each video frame in the video frame sequence to form a screened video frame sequence, which may include:

[0144] Acquire a text recognition result of an original paper homework document corresponding to the target page number identifier and stored in a preset database, wherein the original paper homework document is a document that does not contain handwritten content of a reviewer and an answerer;

[0145] The printed content contained in each video frame in the video frame sequence is compared with the text recognition results of the original paper work document, and the video frames whose comparison results meet the set coincidence conditions are screened out to form a screened video frame sequence.

[0146] Optionally, when the operation learning situation determination unit detects the presence of a limb occlusion area, the process of searching for a supplementary frame may include:

[0147] When a limb occlusion area is detected and the limb occlusion area overlaps with the reviewer's handwritten annotation content, a process is performed to search the video frame sequence for a non-occluded video frame at the same position as the limb occlusion area as a supplementary frame.

[0148] Optionally, the process of obtaining the homework learning status of each respondent based on the handwritten annotations of the reviewer contained in the key frame and the supplementary frame by the homework learning status determination unit may include:

[0149] Using the reviewer's handwritten annotation content contained in the key frame as the first homework learning situation;

[0150] The reviewer's handwritten annotation content at the same position as the limb occluded area in the supplementary frame is used as the second homework learning situation, and the first homework learning situation and the second homework learning situation are combined to form the homework learning situation of each respondent.

[0151] Optionally, the process of obtaining the homework learning status of each respondent based on the handwritten annotations of the reviewer contained in the key frame and the supplementary frame by the homework learning status determination unit may include:

[0152] Cut out the image at the same position as the limb occlusion area in the supplementary frame, and overlay the cut sub-image on the limb occlusion area in the key frame to obtain an edited key frame;

[0153] Based on the reviewer's handwritten annotations contained in the edited key frame, the homework learning status of each respondent is obtained.

[0154] Optionally, the device of the present application may further include:

[0155] The learning status summary unit is used to summarize the homework learning status of the respondents of different identities to obtain the overall homework learning status.

[0156] The embodiment of the present application further provides a storage medium, which may store a program suitable for execution by a processor, wherein the program is used to:

[0157] Acquire a video captured by a camera of a scene in which a reviewer annotates a paper homework document, wherein the paper homework document contains identity information of the respondent;

[0158] Identify each video frame in the video to obtain the reviewer's handwritten annotation content and the identity information of the respondent contained in each video frame;

[0159] Segmenting the video based on the identity information of the respondent to obtain a video frame sequence corresponding to each respondent with an identity;

[0160] Based on the reviewer's handwritten annotations contained in each video frame in the video frame sequence corresponding to each respondent, the homework learning status of each respondent is obtained.

[0161] Optionally, the detailed functions and extended functions of the program may refer to the above description.

[0162] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0163] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can refer to each other.

[0164] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for collecting homework learning information, characterized in that: include: Receive an input collection range instruction, wherein the collection range instruction includes a document range for which homework learning information is to be collected; Acquire a video captured by a camera of a scene in which a reviewer annotates a paper homework document, wherein the paper homework document contains identity information of the respondent; Recognize each video frame in the video to obtain the reviewer's handwritten annotation content and the identity information of the respondent contained in each video frame, and recognize the printed content contained in each video frame; Segmenting the video based on the identity information of the respondent to obtain a video frame sequence corresponding to each respondent with an identity; For the video frame sequence corresponding to each respondent with each identity, based on the printed content contained in each video frame in the video frame sequence, the video frames within the document range specified by the acquisition range instruction are screened to form a screened video frame sequence; Based on the reviewer's handwritten annotations contained in each video frame in the filtered video frame sequence corresponding to each respondent, the homework learning status of each respondent is obtained.

2. The method according to claim 1, characterized in that The method of obtaining the homework learning status of each respondent based on the handwritten annotation content of the reviewer contained in each video frame in the video frame sequence corresponding to each respondent includes: For each identity of the respondent, a key frame is selected from the video frame sequence corresponding to the respondent, wherein the key frame contains the handwritten annotation content of the reviewer; Based at least on the reviewer's handwritten annotations contained in the key frames, the homework learning status of each respondent is obtained.

3. The method according to claim 2, characterized in that Before obtaining the homework learning status of each respondent based at least on the reviewer's handwritten annotation content contained in the key frame, the method further includes: Performing limb occlusion detection on the key frame, and if a limb occlusion area is detected, searching the video frame sequence for a video frame without occlusion at the same position as the limb occlusion area as a supplementary frame; The step of obtaining the homework learning status of each respondent based at least on the reviewer's handwritten annotations contained in the key frame includes: Based on the reviewer's handwritten annotations contained in the key frame and the supplementary frame, the homework learning status of each respondent is obtained.

4. The method according to claim 1, characterized in that The document range for which the homework learning situation collection is to be performed, which is included in the collection range instruction, is the target page number identifier of the document to be collected; Then, based on the printed content contained in each video frame in the video frame sequence, the video frames within the document range specified by the acquisition range instruction are screened to form a screened video frame sequence, including: Based on the printed content contained in each video frame in the video frame sequence, the video frames containing the target page number identifier are screened out to form a screened video frame sequence.

5. The method according to claim 1, characterized in that The document range for which the homework learning situation collection is to be performed, which is included in the collection range instruction, is the target page number identifier of the document to be collected; Then, based on the printed content contained in each video frame in the video frame sequence, the video frames within the document range specified by the acquisition range instruction are screened to form a screened video frame sequence, including: Acquire a text recognition result of an original paper work document corresponding to the target page number identifier and stored in a preset database, wherein the original paper work document is a document that does not contain handwritten content of a reviewer and an answerer; The printed content contained in each video frame in the video frame sequence is compared with the text recognition results of the original paper work document, and the video frames whose comparison results meet the set coincidence conditions are screened out to form a screened video frame sequence.

6. The method according to claim 3, characterized in that When a limb occlusion area is detected, the process of finding a supplementary frame includes: When a limb occlusion area is detected and the limb occlusion area overlaps with the reviewer's handwritten annotation content, a process is performed to search the video frame sequence for a non-occluded video frame at the same position as the limb occlusion area as a supplementary frame.

7. The method according to claim 6, characterized in that The method of obtaining the homework learning status of each respondent based on the reviewer's handwritten annotations contained in the key frame and the supplementary frame includes: Using the reviewer's handwritten annotation content contained in the key frame as the first homework learning situation; The reviewer's handwritten annotation content at the same position as the limb occluded area in the supplementary frame is used as the second homework learning situation, and the first homework learning situation and the second homework learning situation are combined to form the homework learning situation of each respondent.

8. The method according to claim 6, characterized in that The method of obtaining the homework learning status of each respondent based on the reviewer's handwritten annotations contained in the key frame and the supplementary frame includes: Cut out the image at the same position as the limb occlusion area in the supplementary frame, and overlay the cut sub-image on the limb occlusion area in the key frame to obtain an edited key frame; Based on the reviewer's handwritten annotations contained in the edited key frame, the homework learning status of each respondent is obtained.

9. The method according to any one of claims 1 to 8, characterized in that: Also includes: Summarize the homework and learning status of the respondents of each identity to obtain the overall homework and learning status.

10. A homework learning situation collection device, characterized in that: include: A collection range instruction receiving unit, used to receive an input collection range instruction, wherein the collection range instruction includes a document range for which homework learning information is to be collected; A video acquisition unit, used to acquire a video shot by a camera of a scene in which a reviewer annotates a paper homework document, wherein the paper homework document contains identity information of the respondent; A video frame recognition unit, used to recognize each video frame in the video, obtain the reviewer's handwritten annotation content and the identity information of the respondent contained in each video frame, and recognize the printed content contained in each video frame; A video frame segmentation unit, configured to segment the video based on the identity information of the respondent to obtain a video frame sequence corresponding to each respondent with the same identity; A video frame sequence screening unit is used to screen the video frames within the document range specified by the acquisition range instruction for the video frame sequence corresponding to each identity of the respondent, based on the printed content contained in each video frame in the video frame sequence, to form a screened video frame sequence; The homework learning status determination unit is used to obtain the homework learning status of each respondent based on the reviewer's handwritten annotation content contained in each video frame in the filtered video frame sequence corresponding to each respondent.

11. A homework learning situation collection device, characterized in that: include: Camera, data processing module and display module; The camera is used to shoot a scene in which the reviewer annotates a paper homework document to obtain a shot video, wherein the paper homework document contains the identity information of the respondent; The data processing module is used to execute each step of the operation learning situation collection method according to any one of claims 1 to 9; The display module is used to display the video captured by the camera in real time.

12. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, each step of the homework learning situation collection method as described in any one of claims 1 to 9 is implemented.

Images