Learning and accompanying method, device and equipment based on four-corner detection of handheld operation
Through the handheld work four-angle detection method, the frequency domain features and MLSD algorithm are used to accurately identify the work four-angle in the learning and accompanying scenario, solving the problems of high cost and low accuracy in the existing technology, and achieving efficient work area segmentation.
Patent Information
- Application Number
- CN202411232723.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-04
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-09-04
AI Technical Summary
The existing technology has the problem of high cost and low accuracy in the four corners of homework detection in the learning and accompanying scenarios, especially in complex contexts, it is difficult to accurately identify and divide the work area.
Using the method based on handheld job four-angle detection, by acquiring multi-frame real-time images, filtering clear images using frequency domain features, combining pre-trained hand detection model and MLSD algorithm, line segment detection and vertex analysis are performed to identify the related four-angle angles of the job to be submitted. Target vertex.
It improves the accuracy and efficiency of operating area segmentation, reduces costs, and is more robust and applicable than traditional sensor technology.
Smart Images

Figure CN118736626B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of learning and accompanying technology, and in particular to a learning and accompanying method, device and equipment based on four-corner detection of handheld operations. Background Art
[0002] In the process of intelligent learning accompaniment, accurately positioning the homework paper is a key step in realizing the automation process. Students only need to place the homework paper on the scanning or shooting device, and the learning accompaniment robot can automatically complete the positioning, identification and evaluation process, reducing the need for human intervention and improving the automation level of the entire homework processing; at the same time, accurately positioning the homework paper is a prerequisite for realizing automatic homework submission. Only by accurately locating the position of the homework on the paper can the learning accompaniment robot correctly identify and analyze the students' answers, and then conduct subsequent evaluation and feedback.
[0003] The existing precise positioning of homework papers mainly adopts some digital blackboards or smart document cameras to capture students' writing behavior in real time through touch sensors, locate and record students' writing areas on the paper; or some smart pens or writing tools are equipped with pressure sensors to detect the pressure applied by students on the paper and help locate the position of handwritten text or graphics. Touch sensors and pressure sensors have some common disadvantages in learning accompaniment: they usually rely on specific devices, such as digital blackboards or smart document cameras, and sensors embedded in smart pens or writing tools, which leads to device dependence and high costs, limiting the widespread application of technology; at the same time, students need to adapt to these new technologies, resulting in a learning curve, which may affect learning efficiency and comfort; in addition, the sensitivity of these sensor technologies to paper shape and material, as well as the limitations on handwriting diversity and writing details, affect their positioning accuracy and applicability.
[0004] Existing Chinese patent CN114926889A discloses a homework submission method, device, electronic device and storage medium, including: obtaining a first image containing a student's face and a homework to be submitted; performing target detection and image segmentation on the first image to obtain a face block and a homework block in the first image; determining the student's identity information based on the face block, and determining the student's homework information based on the homework block; submitting the homework based on the student's identity information and homework information. Although the above patent also discloses image segmentation of the first image to be submitted to obtain the homework block, the scene captured by the camera usually contains a rich and complex background, including various textures, colors and lighting changes. Conventional image segmentation technology is difficult to accurately identify and segment the homework area because the contrast between the homework and the background is not obvious enough.
[0005] Therefore, how to accurately detect the four corners of the work at low cost in the learning and accompanying scenario, and then segment the work area image is an urgent problem to be solved. Summary of the invention
[0006] In view of this, the present invention provides a learning and accompanying method, device and equipment based on handheld operation four-corner detection, so as to solve the problems of high cost and low accuracy in segmenting the operation area image in the learning and accompanying scenario in the prior art.
[0007] The technical solution adopted by the present invention is:
[0008] In a first aspect, the present invention provides a learning and accompanying method based on four-corner detection of handheld operations, the method comprising:
[0009] S1: Acquire multiple frames of real-time images of students holding homework to be submitted in a learning escort scenario;
[0010] S2: screening the clarity of each frame of the real-time image according to the frequency domain characteristics of each frame of the real-time image, and outputting a clear image;
[0011] S3: inputting the clear image into a pre-trained hand detection model and outputting hand position information;
[0012] S4: using the MLSD algorithm, performing line segment detection on the clear image, and outputting a plurality of initial line segments;
[0013] S5: pre-screening the initial line segments according to the hand position information, and outputting a plurality of target line segments;
[0014] S6: Perform vertex analysis on each of the target line segments to identify multiple target vertices related to the four corners of the job to be submitted.
[0015] Preferably, S2 includes:
[0016] S21: performing fast Fourier transform on each frame of the real-time image, and outputting an amplitude spectrum corresponding to each frame of the real-time image;
[0017] S22: according to each of the amplitude spectra, obtaining the energy proportion of the high frequency component in each frame of the real-time image;
[0018] S23: If the energy proportion is greater than a preset ratio threshold, the real-time image corresponding to the energy proportion is identified as a clear image.
[0019] Preferably, the S5 includes:
[0020] S51: identifying, based on the hand position information, whether the student is lifting the homework to be submitted left and right or the student is lifting the homework to be submitted up and down;
[0021] S52: if it is recognized that the student is holding the homework to be submitted in the left and right hands, the initial line segments are pre-screened, and the first line segments whose horizontal coordinates of the points on the line segments are within the hand area are output;
[0022] S53: If it is recognized that the student is lifting the homework to be submitted up and down, the initial line segments are pre-screened, and the second line segments whose ordinates of the points on the line segments are within the hand area are output;
[0023] S54: acquiring each vertical line segment and each horizontal line segment according to the slope of each first line segment and the slope of each second line segment;
[0024] S55: Determine each of the target line segments according to each of the vertical line segments and each of the horizontal line segments.
[0025] Preferably, the S55 includes:
[0026] S551: averaging the horizontal coordinates of the pixels on the vertical line segments, and obtaining the horizontal distance between the vertical line segments according to the average values corresponding to the vertical line segments;
[0027] S552: averaging the ordinates of the pixels on the horizontal line segments, and obtaining the vertical distances between the horizontal line segments according to the average values corresponding to the horizontal line segments;
[0028] S553: Based on the horizontal distance and the vertical distance, combined with a preset distance threshold, similar horizontal line segments and similar vertical line segments are merged to obtain each of the target line segments.
[0029] Preferably, the S6 includes:
[0030] S61: Extend each target line segment to obtain each initial vertex;
[0031] S62: Classifying the initial vertices according to the width and height of the real-time image to determine the vertices of each region;
[0032] S63: Perform similarity screening on each of the region vertices to identify each of the target vertices.
[0033] Preferably, the S62 includes:
[0034] S621: Acquire the width and height of the real-time image;
[0035] S622: Determine the area corresponding to each initial vertex according to the coordinates of each initial vertex in combination with the width and the height;
[0036] S623: Classify each initial vertex according to the region corresponding to each initial vertex, and determine the vertices in each region.
[0037] Preferably, the S63 includes:
[0038] S631: Obtain the feature vector corresponding to each region vertex;
[0039] S632: Calculate the cosine similarity between the vertices of each region according to each of the feature vectors;
[0040] S633: performing sum processing on the cosine similarities to determine the objective function;
[0041] S634: Identify each of the target vertices according to the target function.
[0042] In a second aspect, the present invention provides a learning accompanying device based on four-corner detection of handheld operations, the device comprising:
[0043] A real-time image acquisition module, used to acquire multiple frames of real-time images of students holding homework to be submitted in a learning escort scenario;
[0044] A clarity screening module is used to screen the clarity of each frame of the real-time image according to the frequency domain characteristics of each frame of the real-time image, and output a clear image;
[0045] A hand detection module, used for inputting the clear image into a pre-trained hand detection model and outputting hand position information;
[0046] A line segment detection module, used to perform line segment detection on the clear image using an MLSD algorithm, and output a plurality of initial line segments;
[0047] A line segment pre-screening module, used for pre-screening the initial line segment according to the hand position information, and outputting a plurality of target line segments;
[0048] The vertex analysis module is used to perform vertex analysis on each of the target line segments to identify multiple target vertices related to the four corners of the job to be submitted.
[0049] In a third aspect, an embodiment of the present invention further provides a learning companion robot, comprising: at least one processor, at least one memory, and computer program instructions stored in the memory, which, when executed by the processor, implement the method of the first aspect in the above-mentioned embodiment.
[0050] In a fourth aspect, an embodiment of the present invention further provides a storage medium on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the method of the first aspect in the above-mentioned embodiment is implemented.
[0051] In summary, the beneficial effects of the present invention are as follows:
[0052] The present invention provides a learning supervision method, device and equipment based on detection of the four corners of hand-held homework. The method includes: obtaining multiple frames of real-time images including students holding homework to be submitted in a learning supervision scenario; screening the clarity of each frame of the real-time image based on the frequency domain characteristics of each frame of the real-time image, and outputting a clear image; inputting the clear image into a pre-trained hand detection model, and outputting hand position information; using the MLSD algorithm, performing line segment detection on the clear image, and outputting multiple initial line segments; based on the hand position information, pre-screening the initial line segments, and outputting multiple target line segments; performing vertex analysis on each of the target line segments, and identifying multiple target vertices related to the four corners of the homework to be submitted. The present invention effectively reduces the interference of blurred or low-definition images by performing clarity screening on the frequency domain features of multiple frames of real-time images, which helps to improve the accuracy of subsequent processing steps; accurately extracts the position information of the student's hand through a pre-trained hand detection model, which helps to determine the position of the homework to be submitted, thereby reducing the segmentation area and improving the accuracy of segmentation; adopts the MLSD (multi-scale line segment detection) algorithm to effectively detect multiple line segments in the image, including the line segments that constitute the homework area. Compared with the traditional line segment detection method, MLSD is more robust and can handle different scales and complex scenes; pre-screens the initial line segments in combination with the hand position information to further reduce irrelevant line segments and improve the accuracy of the target line segments. Such correlation analysis helps to exclude line segments that are not related to the student's hand, thereby more accurately locating the homework area; by performing vertex analysis on the target line segments, the target vertices related to the four corners of the homework to be submitted are identified; In summary, the present invention helps to finally determine the boundary of the homework area and improve the accuracy of segmentation. At the same time, the image processing scheme adopted by the present invention is usually lower in cost than the traditional touch sensor and pressure sensor. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] In order to more clearly illustrate the technical solution of the embodiment of the present invention, the drawings required for use in the embodiment of the present invention will be briefly introduced below. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work, and these are all within the protection scope of the present invention.
[0054] Figure 1 It is a schematic diagram of the overall working process of the learning and accompanying method based on the four-corner detection of handheld operation in Example 1 of the present invention;
[0055] Figure 2 This is a schematic diagram of a process for performing definition screening on each frame of the real-time image in Embodiment 1 of the present invention;
[0056] Figure 3 This is a schematic diagram of the process of pre-screening the initial line segment in Embodiment 1 of the present invention;
[0057] Figure 4 It is a schematic diagram of the process of obtaining each target line segment in Embodiment 1 of the present invention;
[0058] Figure 5 is a schematic diagram of a process of performing vertex analysis on each target line segment in Embodiment 1 of the present invention;
[0059] Figure 6 A schematic diagram of a process for classifying the initial vertices and determining the vertices of each region in Embodiment 1 of the present invention;
[0060] Figure 7 is a schematic diagram of a process for identifying each target vertex in Embodiment 1 of the present invention;
[0061] Figure 8 It is a structural block diagram of a learning accompanying device based on handheld operation four-corner detection in Example 2 of the present invention;
[0062] Fig. 9 This is a schematic diagram of the structure of the learning accompanying robot in Example 3 of the present invention. DETAILED DESCRIPTION
[0063] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution in the embodiment of the present invention will be clearly and completely described in conjunction with the drawings in the embodiment of the present invention. It should be noted that, in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. In the description of the present invention, it should be understood that the orientation or position relationship indicated by the terms "center", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc. is based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation of the present invention. Moreover, the term "include", "comprise" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such a process, method, article or device. In the absence of further restrictions, the elements defined by the phrase "comprising..." do not exclude the existence of other identical elements in the process, method, article or device comprising the elements. If there is no conflict, the embodiments of the present invention and the various features in the embodiments can be combined with each other, all within the protection scope of the present invention.
[0064] Example 1
[0065] See also Figure 1 Embodiment 1 of the present invention discloses a learning and accompanying method based on four-corner detection of handheld operations, the method comprising:
[0066] S1: Acquire multiple frames of real-time images of students holding homework to be submitted in a learning escort scenario;
[0067] Specifically, in the learning escort scenario, multiple frames of real-time images of students holding homework to be submitted during the learning process are obtained. Each frame of the real-time image is decomposed from the real-time video data shot by the close-range camera. By extracting and analyzing the real-time video data, continuous multiple frames of real-time images are obtained. Close-range cameras usually have relatively high resolution and can capture the fine features and details of the target, which is very important for capturing the details of the text and paintings written by students; the camera is close to the shooting target, which can fill the image screen and make the target occupy a larger space in the image, which is very beneficial for more accurate analysis and detection of the target; the close-range camera can better capture the hand movements and subtle movements of the fingers, which is key to identifying gestures, writing movements, etc.; the close-range camera can usually better control the lighting, reduce shadows and improve the clarity of the image, which helps to capture the characteristics of the target more accurately. By utilizing the characteristics of these close-range cameras, the details, gestures and movements of students submitting homework can be better captured, providing more accurate data for subsequent image processing and analysis.
[0068] S2: screening the clarity of each frame of the real-time image according to the frequency domain characteristics of each frame of the real-time image, and outputting a clear image;
[0069] Specifically, based on the frequency domain characteristics of each frame of real-time image, these images are screened for clarity through frequency domain analysis methods, so as to output clear images. For example, images with obvious high-frequency components in the spectrum are retained, while images with weaker high-frequency components and higher blur are filtered out. In this way, a sequence of images that have been screened for clarity can be output, ensuring that subsequent learning care and analysis processes are based on clear and accurate image data, thereby improving the effectiveness of learning care.
[0070] In one embodiment, see Figure 2 , said S2 comprises:
[0071] S21: performing fast Fourier transform on each frame of the real-time image, and outputting an amplitude spectrum corresponding to each frame of the real-time image;
[0072] Specifically, in the homework submission scenario, by converting each frame of real-time image into the frequency domain, the real-time image is represented as an amplitude spectrum of different frequency components, which helps to capture subtle features and structures in the image, such as handwritten text or drawing details in the homework.
[0073] S22: according to each of the amplitude spectra, obtaining the energy proportion of the high frequency component in each frame of the real-time image;
[0074] Specifically, based on each of the amplitude spectra, the energy proportion ratio of the high-frequency components in each frame of the real-time image is obtained. In the specific context of homework submission, the high-frequency components correspond to the details and edge information in the image, such as the details of the strokes in the homework or the edges of the objects. By calculating the energy proportion of the high-frequency components, for each frame of the real-time image, the amplitudes of the high-frequency components selected in the amplitude spectrum are accumulated to obtain the total energy of the high-frequency components. This is achieved through simple mathematical operations, such as summation operations. Similarly, the total energy of the entire amplitude spectrum is calculated, that is, the sum of the amplitudes of all frequency components, and the total energy of the high-frequency components is divided by the total energy of the entire amplitude spectrum to obtain the energy proportion ratio of the high-frequency components. The ratio value ranges from 0 to 1, indicating the proportion of the high-frequency components in the entire amplitude spectrum, thereby quantifying the degree of existence of important details such as the details of the strokes or the edges of the objects in the real-time image.
[0075] S23: If the energy proportion is greater than a preset ratio threshold, the real-time image corresponding to the energy proportion is identified as a clear image.
[0076] Specifically, a clear image means that the content of the homework submitted by the student is more clear and discernible, which helps educators to more accurately evaluate the student's academic performance. By setting the ratio threshold T1 and customizing the clarity standard, it is ensured that only those images containing sufficient detail information are identified as clear images. Such an automatic clarity recognition mechanism can improve the efficiency of the learning accompaniment system in analyzing and providing feedback on student homework.
[0077] S3: inputting the clear image into a pre-trained hand detection model and outputting hand position information;
[0078] Specifically, a clear image obtained by frequency domain clarity screening is used as input, which ensures that the hand detection model has high quality and clarity in the processed image, thereby improving the accuracy of hand detection. YOLO is a real-time target detection algorithm, and YoloV8s represents a small variant of the eighth version of YOLO. The model is pre-trained and can detect the hand area in the image. The clear image is sent to the YoloV8s model for inference, including forward propagation of the clear image through a neural network to obtain the model's prediction of the target position in the image. The YoloV8s model outputs the position information of the left and right hand areas, usually represented by a rectangular box, where S1 (x1, y1, w1, h1) represents the coordinates and size of the left hand area, and S2 (x1, y1, w1, h1) represents the coordinates and size of the right hand area, including the coordinates of the upper left corner of the rectangular box and the width and height of the box. The obtained left and right hand area information is output as hand position information, which is used for further analysis, such as tracking the dynamic changes of students' hands during homework submission, detecting the position of handwriting, or assisting in posture analysis.
[0079] S4: using the MLSD algorithm, performing line segment detection on the clear image, and outputting a plurality of initial line segments;
[0080] Specifically, the MLSD algorithm, whose full name is Multi-Line Segment Detector, is an algorithm for detecting line segments in an image. First, the input clear image is segmented, and the edge information is obtained using a common edge detection algorithm, such as the Canny algorithm. On the basis of image segmentation, edge points in the image are extracted, which are positions in the image that may contain line segments. For each edge point, a candidate line segment is generated according to the gradient direction in its local neighborhood, which usually includes setting an initial direction, selecting an endpoint in the direction, and then generating the candidate line segment according to the gradient direction. In order to vote for the candidate line segments, an accumulator space (Hough space) is constructed, and each point in the accumulator space represents a point in the parameter space of a possible line segment. Each candidate line segment is voted in the accumulator space, and the voting process is realized by adding points related to each candidate line segment in the accumulator space. According to the voting situation in the accumulator, a vote threshold is set to filter out line segments with votes higher than the vote threshold, and extract high-voted line segments as the preliminary test line segments. The MLSD algorithm can effectively capture line segments of different lengths by performing line segment detection at different scales and adapt to different job submission scenarios. At the same time, since MLSD takes the gradient direction into consideration when generating candidate line segments, it can detect line segments in multiple directions in the image, which is very suitable for capturing different postures and directions in hand movements. The MLSD algorithm can effectively detect multiple line segments in job submission scenarios, providing a strong foundation for hand position and movement analysis.
[0081] S5: pre-screening the initial line segments according to the hand position information, and outputting a plurality of target line segments;
[0082] In one embodiment, see Figure 3 , the S5 comprises:
[0083] S51: identifying, based on the hand position information, whether the student is lifting the homework to be submitted left and right or the student is lifting the homework to be submitted up and down;
[0084] Specifically, from the hand position information obtained in the previous steps, the coordinate information of the student's left hand position S1 (x1, y1, w1, h1) and right hand S2 (x2, y2, w2, h2) is obtained. If the y1 and y2 coordinates of the left and right hands are in the middle of the image, it is considered that the left and right hands are lifting the homework left and right, which means that the student's left and right hands are holding the homework left and right; if the x1 and x2 coordinates of the left and right hands are in the middle of the image, it is considered that the homework is lifted up and down, which means that the student's left and right hands are holding the homework up and down.
[0085] S52: If it is recognized that the student holds the assignment to be submitted in a left - right lifting manner, then pre - screen the initial line segments, and output each first line segment whose abscissa of each point on the line segment is within the hand area;
[0086] Specifically, if it is recognized that the student holds the assignment to be submitted in a left - right lifting manner, then pre - screen the initial line segments, and retain the line segments whose abscissas are within the left - hand and right - hand areas. Specifically, for each point (xi, yi) on each initial line segment, determine whether xi satisfies S1(x1) < xi < S1(x1 + w1) or S2(x2) < xi < S2(x2 + w2), and only retain the points whose abscissas meet the above requirements.
[0087] S53: If it is recognized that the student holds the assignment to be submitted in an up - down lifting manner, then pre - screen the initial line segments, and output each second line segment whose ordinate of each point on the line segment is within the hand area;
[0088] Specifically, if it is recognized that the student holds the assignment to be submitted in an up - down lifting manner, then pre - screen the initial line segments, and retain the line segments whose ordinates are within the left - hand and right - hand areas. Specifically, for each point (xi, yi) on each initial line segment, determine whether yi satisfies S1(y1) < yi < S1(y1 + h1) or S2(y2) < yi < S2(y2 + h2), and only retain the points whose ordinates meet the above requirements. Through the above steps, pre - screen the initial line segments according to the student's lifting operation of the assignment, and respectively output the first line segments and the second line segments that meet the conditions. This helps to extract line segment information related to the hand in different lifting operation scenarios and provides more specific data for subsequent analysis.
[0089] S54: Obtain each vertical line segment and each horizontal line segment according to the slopes of each first line segment and each second line segment;
[0090] Specifically, for each first line segment and each second line segment, use the formula for slope , calculate the slopes of each first line segment and each second line segment, where θ represents the slope, xn, xm respectively represent the abscissas of any two points on the first line segment and the second line segment, and yn, ym respectively represent the ordinates of any two points on the first line segment and the second line segment. Retain the line segments in the vertical direction and the horizontal direction according to the following formula: where Fabs represents taking the absolute value. Through the above steps, judge whether the line segment is vertical or horizontal according to the calculated slope. Such a judgment method provides more specific and directional information for subsequent line segment analysis, especially in different action scenarios of the student's lifting operation of the assignment.
[0091] S55: Determine each target line segment according to each of the vertical line segments and each of the horizontal line segments.
[0092] In one embodiment, please refer to Figure 4, the S55 comprises:
[0093] S551: averaging the horizontal coordinates of the pixels on the vertical line segments, and obtaining the horizontal distance between the vertical line segments according to the average values corresponding to the vertical line segments;
[0094] Specifically, for each vertical line segment, calculate the average value xavg of the horizontal coordinates xi of all pixels on it, which is obtained by summing the x coordinates of all points on the line segment and dividing it by the number of points; for any two vertical line segments Xi and Xj, calculate the corresponding xiavg and xjavg, and calculate the horizontal distance D between the vertical line segments according to the following formula: D=fabs(xiavg-xjavg); this process can help determine the horizontal distribution of each vertical line segment in the image, providing more specific information about the position of the line segments.
[0095] S552: averaging the ordinates of the pixels on the horizontal line segments, and obtaining the vertical distances between the horizontal line segments according to the average values corresponding to the horizontal line segments;
[0096] Specifically, for each horizontal line segment, calculate the average value yavg of the horizontal coordinates yi of all pixel points on it, which is obtained by summing the y coordinates of all points on the line segment and dividing it by the number of points; for any two horizontal line segments Yi and Yj, calculate the corresponding yiavg and yjavg, and calculate the vertical distance D between the horizontal line segments according to the following formula: D=fabs(yiavg-yjavg).
[0097] S553: Based on the horizontal distance and the vertical distance, combined with a preset distance threshold, similar horizontal line segments and similar vertical line segments are merged to obtain each of the target line segments.
[0098] Specifically, according to the previous steps, the horizontal distance and vertical distance between each vertical line segment are obtained, and combined with the preset distance threshold δ, the preset distance threshold δ is used to determine whether each of the horizontal distances and vertical distances is less than δ. If less than, the two horizontal line segments are similar line segments or the two vertical line segments are similar line segments, and the similar line segments are fused to obtain each of the target line segments. Fusion of similar horizontal and vertical line segments helps to eliminate small, discontinuous line segments caused by noise or resolution limitations in the image and improve the continuity of the line segments; in the image, there may be adjacent horizontal and vertical line segments, which actually represent the same object edge. By fusing similar line segments, redundant information can be reduced, making the line segment representation more concise.
[0099] S6: Perform vertex analysis on each of the target line segments to identify multiple target vertices related to the four corners of the job to be submitted.
[0100] Specifically, vertex analysis is performed on each of the target line segments to identify multiple target vertices related to the four corners of the job to be submitted, wherein the vertex analysis includes: using shape analysis or geometric relationship analysis. By identifying the target vertices related to the four corners of the job to be submitted, the position of the job to be submitted in the image is accurately located.
[0101] In one embodiment, see Figure 5 , the S6 comprises:
[0102] S61: Extend each target line segment to obtain each initial vertex;
[0103] Specifically, using the target line segment set obtained in the previous step, all vertical line segments are extended until the line segments intersect with the upper and lower boundaries of the image; all horizontal line segments are extended until the line segments intersect with the left and right boundaries of the image, and a preliminary vertex set P1 is obtained, wherein P1 includes: points where all extended line segments intersect with the image boundary. By obtaining the vertices where the extended line segments intersect with the image boundary, a preliminary vertex set P1 is formed, which will help the subsequent refinement and adjustment of the target vertices.
[0104] S62: Classifying the initial vertices according to the width and height of the real-time image to determine the vertices of each region;
[0105] Specifically, the width W and height H of the real-time image are obtained, which is obtained according to the image size in the actual scene. Each initial vertex is classified according to the relative position of its horizontal coordinate x and vertical coordinate y to the width and height of the real-time image. The classification results include: upper left vertex: x and y are both in the upper left corner area of the real-time image; upper right vertex: x is in the right half of the real-time image, y is in the upper half of the real-time image; lower left vertex: x is in the left half of the real-time image, y is in the lower half of the real-time image; lower right vertex: x and y are both in the lower right corner area of the real-time image; vertices in the same area are classified as the vertex set of the area. According to the width and height of the real-time image, the initial vertices are classified into different areas, providing more spatial relationship information for subsequent image analysis and student action recognition.
[0106] In one embodiment, see Figure 6 , the S62 comprises:
[0107] S621: Acquire the width and height of the real-time image;
[0108] S622: Determine the area corresponding to each initial vertex according to the coordinates of each initial vertex in combination with the width and the height;
[0109] Specifically, according to the coordinates of each initial vertex, combined with the width and height, each initial vertex is classified into four region vertices, including: P1 region vertices, P2 region vertices, P3 region vertices, and P4 region vertices. Among them, the coordinates of each region vertex satisfy the following rules: 0 < P1(x) <= 1 / 2 * W, 0 < P1(y) <= 1 / 2 * H; 1 / 2 * W < P2(x) <= W, 0 < P2(y) <= 1 / 2 * H; 0 < P3(x) <= 1 / 2 * W, 1 / 2 * H < P3(y) <= H; 1 / 2 * W < P4(x) <= W, 1 / 2 * H < P3(y) <= H; where * represents the multiplication operator in mathematics. Considering that the camera used is a close-up camera, when the operation is clearly photographed, the area of the operation in the image will occupy most of the image area. Through this step, based on the close-up characteristics of the camera, the vertex set can be divided into four parts according to the coordinate rules, so that the vertices in each region satisfy the specified coordinate range rules.
[0110] S623: Classify each of the initial vertices according to the region corresponding to each initial vertex to determine each region vertex.
[0111] S63: Perform similarity screening on each of the region vertices to identify each target vertex.
[0112] In one embodiment, please refer to Figure 7 , where S63 includes:
[0113] S631: Obtain the feature vectors corresponding to each region vertex;
[0114] Specifically, in this step, for the vertices in each region, their feature vectors need to be extracted. The feature vectors include information related to the consistency of the operation texture, such as local texture features, color histograms, and gradient information; determine the template size for feature extraction, and this template will be used to extract local features around the vertices in each region. For the vertices in each region, with each of them as the center, use the template to extract local features in the corresponding region. The local features include grayscale values, color information, and textures; combine the local features extracted from each region into a feature vector, and this vector is used to represent the features of this region.
[0115] S632: Calculate the cosine similarity between each region vertex according to each of the feature vectors;
[0116] Specifically, by calculating the cosine similarity between feature vectors, the cosine similarity is used to measure the angle between two feature vectors, and its calculation formula is: where ||A|| represents the modulus of vector A, ||B|| represents the modulus of vector B, Cosine Similarity(A, B) represents the cosine similarity between vector A and vector B, and · represents the multiplication operator in mathematics.
[0117] S633: performing sum processing on the cosine similarities to determine the objective function;
[0118] Specifically, the cosine similarities between the vertices of each region are summed to form an objective function Fun, which is expressed as: Where Ci represents the cosine similarity between the vertices of the i-th group of regions. The smaller the value of the objective function Fun, the higher the similarity, that is, the closer to the ideal state.
[0119] S634: Identify each of the target vertices according to the target function.
[0120] Specifically, according to the value of the objective function Fun, the four vertices that minimize the objective function are selected as the four vertices of the job. By minimizing the objective function, the most similar vertex combination in each area can be obtained, thereby determining the location of the job. Through the above steps, the four vertices of the job are effectively determined by feature similarity, thereby accurately locating the target image of the job to be submitted in a scene with consistent texture.
[0121] Example 2
[0122] See also Figure 8 Embodiment 2 of the present invention further provides a learning accompanying device based on four-corner detection of handheld operations, the device comprising:
[0123] A real-time image acquisition module, used to acquire multiple frames of real-time images of students holding homework to be submitted in a learning escort scenario;
[0124] A clarity screening module is used to screen the clarity of each frame of the real-time image according to the frequency domain characteristics of each frame of the real-time image, and output a clear image;
[0125] A hand detection module, used for inputting the clear image into a pre-trained hand detection model and outputting hand position information;
[0126] A line segment detection module, used to perform line segment detection on the clear image using an MLSD algorithm, and output a plurality of initial line segments;
[0127] A line segment pre-screening module, used for pre-screening the initial line segment according to the hand position information, and outputting a plurality of target line segments;
[0128] The vertex analysis module is used to perform vertex analysis on each of the target line segments to identify multiple target vertices related to the four corners of the job to be submitted.
[0129] Specifically, a learning accompaniment device based on detection of four corners of hand-held homework provided by an embodiment of the present invention is adopted, and the device includes: a real-time image acquisition module, used to acquire multiple frames of real-time images including students holding homework to be submitted in a learning accompaniment scenario; a clarity screening module, used to perform clarity screening on each frame of the real-time image according to the frequency domain characteristics of each frame of the real-time image, and output a clear image; a hand detection module, used to input the clear image into a pre-trained hand detection model, and output hand position information; a line segment detection module, used to perform line segment detection on the clear image using the MLSD algorithm, and output multiple initial line segments; a line segment pre-screening module, used to pre-screen the initial line segments according to the hand position information, and output multiple target line segments; a vertex analysis module, used to perform vertex analysis on each of the target line segments, and identify multiple target vertices related to the four corners of the homework to be submitted. The device effectively reduces the interference of blurred or low-definition images by performing clarity screening on the frequency domain features of multiple frames of real-time images, which helps to improve the accuracy of subsequent processing steps; accurately extracts the position information of the student's hand through a pre-trained hand detection model, which helps to determine the position of the homework to be submitted, thereby reducing the segmentation area and improving the accuracy of segmentation; adopts the MLSD (multi-scale line segment detection) algorithm to effectively detect multiple line segments in the image, including the line segments that constitute the homework area. Compared with the traditional line segment detection method, MLSD is more robust and can handle different scales and complex scenes; pre-screens the initial line segments in combination with the hand position information to further reduce irrelevant line segments and improve the accuracy of the target line segments. Such correlation analysis helps to exclude line segments that are not related to the student's hand, thereby more accurately locating the homework area; by performing vertex analysis on the target line segment, the target vertices related to the four corners of the homework to be submitted are identified; In summary, the device helps to finally determine the boundary of the homework area and improve the accuracy of segmentation. At the same time, the device is usually lower in cost than the traditional touch sensor and pressure sensor.
[0130] Example 3
[0131] In addition, combined Figure 1 The learning and accompanying method based on the detection of four corners of handheld operations described in Example 1 of the present invention can be implemented by a learning and accompanying robot. Fig. 9 A schematic diagram of the hardware structure of the learning and accompanying robot provided in Example 3 of the present invention is shown.
[0132] The learning companion robot may include a processor and a memory storing computer program instructions.
[0133] Specifically, the processor may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiment of the present invention.
[0134] The memory may include a large capacity memory for data or instructions. By way of example and not limitation, the memory may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. Where appropriate, the memory may include a removable or non-removable (or fixed) medium. Where appropriate, the memory may be inside or outside the data processing device. In a particular embodiment, the memory is a non-volatile solid-state memory. In a particular embodiment, the memory includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM) or a flash memory or a combination of two or more of these.
[0135] The processor implements any one of the learning and accompanying methods based on four-corner detection of handheld operations in the above-mentioned embodiments by reading and executing computer program instructions stored in the memory.
[0136] In one example, the learning companion robot may further include a communication interface and a bus. Fig. 9 As shown, the processor, memory, and communication interface are connected via a bus and communicate with each other.
[0137] The communication interface is mainly used to implement communication between the modules, devices, units and / or equipment in the embodiments of the present invention.
[0138] The bus includes hardware, software or both, and the components of the device are coupled to each other. For example, but not limitation, the bus may include an accelerated graphics port (AGP) or other graphics bus, an enhanced industrial standard architecture (EISA) bus, a front-side bus (FSB), a hypertransport (HT) interconnection, an industrial standard architecture (ISA) bus, an infinite bandwidth interconnection, a low pin count (LPC) bus, a memory bus, a microchannel architecture (MCA) bus, a peripheral component interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a serial advanced technology attachment (SATA) bus, a video electronics standard association local (VLB) bus or other suitable bus or a combination of two or more of these. Where appropriate, the bus may include one or more buses. Although the embodiments of the present invention describe and illustrate a specific bus, the present invention contemplates any suitable bus or interconnection.
[0139] Example 4
[0140] In addition, in combination with the learning and accompanying method based on the detection of four corners of handheld operation in the above embodiment 1, the embodiment 4 of the present invention can also provide a computer-readable storage medium for implementation. The computer-readable storage medium stores computer program instructions; when the computer program instructions are executed by the processor, any one of the learning and accompanying methods based on the detection of four corners of handheld operation in the above embodiments is implemented.
[0141] In summary, the embodiments of the present invention provide a learning and accompanying method, device and equipment based on four-corner detection of handheld operations.
[0142] It should be clear that the present invention is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present invention.
[0143] The functional blocks shown in the structural block diagram described above can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier. "Machine-readable medium" may include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROMs, flash memories, erasable ROMs (EROMs), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.
[0144] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant location, and provide corresponding operation entrances for users to choose to authorize or refuse.
[0145] It should also be noted that the exemplary embodiments mentioned in the present invention describe some methods or systems based on a series of steps or devices. However, the present invention is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiments, or in a different order from the embodiments, or several steps can be performed simultaneously.
[0146] The above is only a specific implementation of the present invention. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the system, module and unit described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. It should be understood that the protection scope of the present invention is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be covered within the protection scope of the present invention.
Claims
1. A learning and accompanying method based on four-corner detection of handheld operations, characterized in that: The method comprises: S1: Acquire multiple frames of real-time images of students holding homework to be submitted in a learning escort scenario; S2: screening the clarity of each frame of the real-time image according to the frequency domain characteristics of each frame of the real-time image, and outputting a clear image; S3: inputting the clear image into a pre-trained hand detection model and outputting hand position information; S4: using the MLSD algorithm, performing line segment detection on the clear image, and outputting a plurality of initial line segments; S5: pre-screening the initial line segments according to the hand position information, and outputting a plurality of target line segments; S6: performing vertex analysis on each of the target line segments to identify multiple target vertices associated with the four corners of the job to be submitted; The S5 includes: S51: identifying, based on the hand position information, whether the student is lifting the homework to be submitted left and right or the student is lifting the homework to be submitted up and down; S52: if it is recognized that the student is holding the homework to be submitted in the left and right hands, the initial line segments are pre-screened, and the first line segments whose horizontal coordinates of the points on the line segments are within the hand area are output; S53: If it is recognized that the student is lifting the homework to be submitted up and down, the initial line segments are pre-screened, and the second line segments whose ordinates of the points on the line segments are within the hand area are output; S54: acquiring each vertical line segment and each horizontal line segment according to the slope of each first line segment and the slope of each second line segment; S55: determining each of the target line segments according to each of the vertical line segments and each of the horizontal line segments; The S6 includes: S61: Extend each target line segment to obtain each initial vertex; S62: Classifying the initial vertices according to the width and height of the real-time image to determine the vertices of each region; The S62 includes: according to the relative positions of the horizontal coordinate x and the vertical coordinate y of each of the initial vertices and the width and height of the real-time image, the classification results include: the upper left vertex, the lower left vertex, the upper right vertex and the lower right vertex; S63: performing similarity screening on each of the region vertices to identify each of the target vertices; The S63 includes: S631: Obtain the feature vector corresponding to each region vertex; S632: Calculate the cosine similarity between the vertices of each region according to each of the feature vectors; S633: performing sum processing on the cosine similarities to determine the objective function; S634: Identify each of the target vertices according to the target function.
2. The learning and accompanying method based on four-corner detection of handheld operations according to claim 1 is characterized in that: The S2 includes: S21: performing fast Fourier transform on each frame of the real-time image, and outputting an amplitude spectrum corresponding to each frame of the real-time image; S22: according to each of the amplitude spectra, obtaining the energy proportion of the high frequency component in each frame of the real-time image; S23: If the energy proportion is greater than a preset ratio threshold, the real-time image corresponding to the energy proportion is identified as a clear image.
3. The learning and accompanying method based on handheld operation four-corner detection according to claim 1 is characterized in that: The S55 includes: S551: averaging the horizontal coordinates of the pixels on the vertical line segments, and obtaining the horizontal distance between the vertical line segments according to the average values corresponding to the vertical line segments; S552: averaging the ordinates of the pixels on the horizontal line segments, and obtaining the vertical distances between the horizontal line segments according to the average values corresponding to the horizontal line segments; S553: Based on the horizontal distance and the vertical distance, combined with a preset distance threshold, similar horizontal line segments and similar vertical line segments are merged to obtain each of the target line segments.
4. The learning and accompanying method based on four-corner detection of handheld operation according to claim 1 is characterized in that: The S62 includes: S621: Acquire the width and height of the real-time image; S622: Determine the area corresponding to each initial vertex according to the coordinates of each initial vertex in combination with the width and the height; S623: Classify each initial vertex according to the region corresponding to each initial vertex, and determine the vertices in each region.
5. A learning and accompanying device based on four-corner detection of handheld operations, characterized in that: The device comprises: A real-time image acquisition module, used to acquire multiple frames of real-time images of students holding homework to be submitted in a learning escort scenario; A clarity screening module is used to screen the clarity of each frame of the real-time image according to the frequency domain characteristics of each frame of the real-time image, and output a clear image; A hand detection module, used for inputting the clear image into a pre-trained hand detection model and outputting hand position information; A line segment detection module, used to perform line segment detection on the clear image using an MLSD algorithm, and output a plurality of initial line segments; A line segment pre-screening module, used for pre-screening the initial line segment according to the hand position information, and outputting a plurality of target line segments; A vertex analysis module, used to perform vertex analysis on each of the target line segments to identify multiple target vertices associated with the four corners of the job to be submitted; The pre-screening of the initial line segments according to the hand position information and outputting a plurality of target line segments comprises: According to the hand position information, identifying whether the student is lifting the homework to be submitted left and right or the student is lifting the homework to be submitted up and down; If it is recognized that the student holds the homework to be submitted in the left and right hands, the initial line segments are pre-screened, and the first line segments whose horizontal coordinates of the points on the line segments are within the hand area are output; If it is recognized that the student is holding up the homework to be submitted, the initial line segment is pre-screened, and each second line segment whose ordinate of each point on the line segment is within the hand area is output; Obtain each vertical line segment and each horizontal line segment according to the slope of each first line segment and the slope of each second line segment; Determine each of the target line segments according to each of the vertical line segments and each of the horizontal line segments; The performing vertex analysis on each of the target line segments to identify multiple target vertices associated with the four corners of the job to be submitted includes: Extending each of the target line segments to obtain each initial vertex; Classifying each of the initial vertices according to the width and height of the real-time image to determine each area vertex; The step of classifying the initial vertices according to the width and height of the real-time image to determine the vertices of each region includes: according to the relative positions of the horizontal coordinate x and the vertical coordinate y of each initial vertex and the width and height of the real-time image, the classification results include: the upper left vertex, the lower left vertex, the upper right vertex and the lower right vertex; Performing similarity screening on each of the region vertices to identify each of the target vertices; The similarity screening of each of the region vertices to identify each of the target vertices includes: Get the feature vectors corresponding to the vertices of each region; Calculate the cosine similarity between the vertices of each region according to each of the feature vectors; Summing the cosine similarities to determine the objective function; According to the target function, each target vertex is identified.
6. A learning and nursing robot, characterized in that: include: At least one processor, at least one memory and computer program instructions stored in the memory, when the computer program instructions are executed by the processor, implement the method according to any one of claims 1 to 4.
7. A storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 4 is implemented.
