Handwritten content extraction method and device, equipment and medium

Abstract

The invention provides a handwritten content extraction method and device, equipment and a medium, and the method comprises the steps: receiving a handwritten operation on a handwritten layer which is a transparent mask layer disposed on a content display layer and overlapped with the content display layer; and in response to the handwriting operation, displaying a handwriting track corresponding to the handwriting operation on the handwriting layer, and extracting the handwriting content from the handwriting layer. According to the handwritten content extraction method and device, the equipment and the medium provided by the invention, separation of the handwritten content and the display content is realized by introducing the transparent mask layer, and the handwritten track is only recorded on the mask layer, so that content confusion and overlapping are avoided. The transparent mask layer ensures accurate extraction of the handwritten content, and the track can still be completely recorded even in a complex scene when the handwritten content is close to or overlapped with the displayed content. Besides, the limitation on a handwriting area is eliminated, the user is allowed to freely write on the whole screen, and immersive experience similar to paper answering is provided for the user.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a handwritten content extraction method, device, equipment and medium. Background Art

[0002] With the popularization of electronic devices, handwriting input on electronic screens is becoming more and more widely used. In scenarios such as education, business, and medical care, handwriting input on electronic screens has become an important way for users to interact with devices.

[0003] The existing technology still has some obvious shortcomings in the extraction of handwritten content. One solution is to obtain handwritten content by adding additional interactive input areas, but this solution cannot provide the immersive feeling in the paper handwriting process, and the writing area is limited, and handwritten content outside the area cannot be extracted. Another solution is to recognize handwritten content through an optical character recognition engine, but this solution can easily misidentify the display content of the electronic screen as handwritten content, especially when the display content is in imitation handwriting fonts or italics, it is difficult to accurately distinguish. In addition, the display content and handwritten content are usually mixed in the same layer. When the handwritten content overlaps with the display content, the extraction accuracy is low. Summary of the invention

[0004] The present invention provides a handwriting content extraction method, device, equipment and medium, which are used to solve the defect of difficulty in handwriting content extraction in related technologies.

[0005] The present invention provides a handwritten content extraction method, comprising: receiving a handwriting operation on a handwriting layer, wherein the handwriting layer is a transparent mask layer disposed above the content display layer and overlapping the content display layer; In response to the handwriting operation, a handwriting track corresponding to the handwriting operation is displayed on the handwriting layer, and handwriting content is extracted from the handwriting layer.

[0006] According to a handwriting content extraction method provided by the present invention, the extracting handwriting content from the handwriting layer includes: Determining a region position of an expected handwriting region on the handwriting layer based on the content layout of the content display layer; Based on the area position of the expected handwriting area, handwriting content is extracted from the handwriting layer.

[0007] According to a handwriting content extraction method provided by the present invention, the determining of the area position of the expected handwriting area on the handwriting layer based on the content layout of the content display layer includes: Based on the content layout of the content display layer, traverse all display contents in the content display layer to obtain a position offset of each display content compared to the content layout; Based on the position offset of each of the displayed contents compared to the content layout and the relative position between each of the displayed contents and the corresponding expected handwriting area, the area position of the expected handwriting area corresponding to each of the displayed contents is determined.

[0008] According to a handwriting content extraction method provided by the present invention, the handwriting content is extracted from the handwriting layer based on the area position of the expected handwriting area, comprising: rasterizing the handwritten layer into a handwritten image, and extracting a connected domain subgraph from the handwritten image; Determine a connected domain subgraph associated with the expected handwriting area based on the area position of the expected handwriting area and the subgraph position of the connected domain subgraph; The handwriting content is extracted from the connected domain subgraph associated with the expected handwriting area.

[0009] According to a handwriting content extraction method provided by the present invention, the method of determining a connected domain subgraph associated with the expected handwriting area based on the area position of the expected handwriting area and the subgraph position of the connected domain subgraph comprises: Based on the area position of the expected handwriting area and the subgraph position of the connected domain subgraph, determining the position intersection between the expected handwriting area and the connected domain subgraph, wherein the position intersection is the number of pixels of the connected domain subgraph in the expected handwriting area, or the area of ​​the rectangular outline of the connected domain subgraph in the expected handwriting area; Based on the position intersection situation, it is determined whether the connected domain subgraph is associated with the expected handwriting area.

[0010] According to a handwriting content extraction method provided by the present invention, the determining whether the connected domain subgraph is associated with the expected handwriting area based on the position intersection situation further includes: When it is determined based on the position intersection that the connected domain subgraph is not associated with the expected handwriting area, it is determined based on the position proximity between the connected domain subgraph and the expected handwriting area whether the connected domain subgraph is associated with the expected handwriting area.

[0011] According to a handwriting content extraction method provided by the present invention, the determining whether the connected domain subgraph is associated with the expected handwriting area based on the positional proximity between the connected domain subgraph and the expected handwriting area includes: In the case where the connected domain subgraph is adjacent to the multiple expected handwriting regions, the connected domain subgraph is segmented to obtain multiple segmented subgraphs as connected domain subgraphs respectively associated with the multiple expected handwriting regions.

[0012] According to a handwriting content extraction method provided by the present invention, the handwriting content is extracted from the handwriting layer based on the area position of the expected handwriting area, comprising: Determine a handwriting track associated with the expected handwriting area based on the area position of the expected handwriting area and the track position of the handwriting track in the handwriting layer; Handwriting content is extracted from the handwriting trajectory associated with the expected handwriting area.

[0013] According to a handwriting content extraction method provided by the present invention, the determining of a handwriting trajectory associated with the expected handwriting area based on the area position of the expected handwriting area and the trajectory position of the handwriting trajectory in the handwriting layer includes: Based on the area position of the expected handwriting area and the track position of the handwriting track in the handwriting layer, determining the intersection of the expected handwriting area and the handwriting track; Based on the position intersection situation, it is determined whether the handwriting trajectory is associated with the expected handwriting area.

[0014] According to a handwriting content extraction method provided by the present invention, the determining whether the handwriting trajectory is associated with the expected handwriting area based on the position intersection situation further includes: When it is determined that the handwriting track is not associated with the expected handwriting area based on the position intersection, it is determined whether the handwriting track is associated with the expected handwriting area based on the position proximity between the handwriting track and the expected handwriting area.

[0015] According to a handwriting content extraction method provided by the present invention, after extracting the handwriting content from the handwriting layer, the method further includes: Based on the area position of the expected handwriting area, the handwriting content is displayed in real time in the content display layer.

[0016] The present invention also provides a handwritten content extraction device, comprising the following modules: A handwriting operation receiving unit, used for receiving a handwriting operation on a handwriting layer, wherein the handwriting layer is a transparent mask layer disposed above the content display layer and overlapping the content display layer; The handwriting content extraction unit is used to display a handwriting track corresponding to the handwriting operation on the handwriting layer in response to the handwriting operation, and extract handwriting content from the handwriting layer.

[0017] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the above-mentioned handwriting content extraction methods is implemented.

[0018] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the handwritten content extraction method described in any one of the above methods is implemented.

[0019] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the handwriting content extraction method described in any one of the above is implemented.

[0020] The handwritten content extraction method, device, equipment and medium provided by the present invention achieve the separation of handwritten content and displayed content by introducing a transparent mask, and the handwritten trajectory is only recorded on the mask, avoiding content confusion and overlap. The transparent mask ensures the accurate extraction of handwritten content, and the trajectory can still be fully recorded even in complex scenes when the handwritten content is close to or overlaps the displayed content. In addition, the restrictions on the handwritten area are removed, allowing users to write freely on the entire screen, providing users with an immersive experience similar to answering questions on paper. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the present invention or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0022] Figure 1 It is a schematic diagram of the first type of solution in the prior art.

[0023] Figure 2 It is a flow chart of the second type of solution in the prior art.

[0024] Figure 3 It is a flow chart of the handwriting content extraction method provided by the present invention.

[0025] Figure 4 It is a schematic diagram of a flow chart of determining the area position of an expected handwriting area provided by the present invention.

[0026] Figure 5 It is a flowchart of the pixel quantity determination method provided by the present invention in a question answering scenario.

[0027] Figure 6 It is a flowchart of the rectangular contour judgment method provided by the present invention in a question-answering scenario.

[0028] Figure 7 It is a flow chart of the handwriting trajectory extraction method provided by the present invention in a question-answering scenario.

[0029] Figure 8 It is a schematic diagram of the process of extracting user answer content provided by the present invention.

[0030] Fig. 9 It is a structural schematic diagram of the handwriting content extraction device provided by the present invention.

[0031] Fig.10 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0033] With the popularity of electronic devices, handwriting input is increasingly used on electronic screens. In education, business, medical and other scenarios, handwriting input has gradually become an important way for users to interact with devices. Whether it is taking notes, signing or answering questions, users can convey information more naturally through handwriting input. However, existing handwriting content extraction technologies still have obvious shortcomings in many application scenarios and cannot meet users' growing demands for accuracy and fluency.

[0034] The existing technical solutions can be roughly divided into two categories. Figure 1 This is a schematic diagram of the first type of solution in the prior art. As shown in the figure, the first type of solution obtains handwritten content by adding an additional answer handwriting area. This method limits users to writing only in a specified area. Once the handwritten content exceeds the preset boundary, it cannot be correctly extracted. At the same time, this solution is difficult to provide the immersive feeling when answering on paper, which affects the user's writing experience.

[0035] Figure 2 It is a flow chart of the second type of solution in the prior art. As shown in the figure, the second type of solution records the displayed content and handwritten content by taking photos or screenshots, and uses an optical character recognition engine to extract the handwritten content. Although this solution can recognize handwritten content by imaging, it has some serious defects. The optical character recognition engine can easily mistake the displayed content on the electronic screen as handwritten content, especially when the displayed content uses imitation handwriting fonts or italics, it is difficult to accurately distinguish between handwritten content and displayed content. In addition, handwritten content and displayed content are usually mixed in the same layer. When the user's handwritten content overlaps with the displayed content, it is difficult to effectively extract the handwritten content, resulting in a decrease in recognition accuracy.

[0036] In view of the above problems, an embodiment of the present invention provides a handwritten content extraction method. Figure 3 FIG. 1 is a flow chart of the handwritten content extraction method provided by the present invention. Figure 3 As shown, the method includes: Step 310, receiving a handwriting operation on a handwriting layer, wherein the handwriting layer is a transparent mask layer disposed above the content display layer and overlapping the content display layer; Specifically, the transparent mask is a layer covering the content display layer, which is used to capture the user's handwriting operations without affecting the display content of the content display layer. The core function of the transparent mask is to receive and record the user's handwriting operations. Every stroke of the user's writing will be recorded on the transparent mask, so that the handwriting content can be accurately extracted in subsequent processing.

[0037] It is understandable that the design of the transparent mask completely separates the handwritten content from the displayed content on the screen. The advantage of this separation mechanism is that the handwritten content will not interfere with the title, picture or other displayed content, ensuring the accurate capture and extraction of the handwriting trajectory. In this way, when the handwritten content is subsequently processed, recognition errors caused by overlapping or confusing displayed content can be avoided. This design greatly improves the accuracy of handwritten content extraction, especially in complex writing scenarios, and can also ensure high-precision extraction effects.

[0038] In addition, the transparent feature of the transparent mask helps improve the user's writing experience. Because the transparent mask covers the display layer, users can input handwriting while seeing the underlying content (such as the title, image, or text prompt). This design not only ensures the smoothness of the user's writing experience, but also provides an interactive experience between the user and the displayed content, providing an immersive feeling in the paper writing process.

[0039] Step 320: In response to the handwriting operation, a handwriting track corresponding to the handwriting operation is displayed on the handwriting layer, and handwriting content is extracted from the handwriting layer.

[0040] Specifically, when the user writes on the handwriting layer, the handwriting track will be presented in real time on the handwriting layer. This process ensures that the user can instantly see their writing content and get a feedback experience similar to writing on paper. Whether using a stylus or a finger, the track of the handwriting operation will be accurately depicted and displayed on the screen, ensuring that the track of each handwriting operation of the user can be clearly displayed.

[0041] Handwriting content extraction is the process of converting the user's handwriting trajectory from the handwriting layer into handwriting content. The handwriting trajectory is the handwriting path left by the user during the writing process, recording the starting point, end point, stroke direction and shape of each stroke. The handwriting content is the overall writing result formed by these trajectories. When extracting handwriting content, you first need to accurately obtain these trajectories from the handwriting layer, and then combine them into complete handwriting content based on the connectivity and stroke structure of the trajectories.

[0042] Handwriting content is inseparable from handwriting trajectory. Each handwriting trajectory represents a single stroke of the user, and handwriting content is a collection of these trajectories. During the extraction process, the accurate recording of handwriting trajectories determines the integrity of the final handwriting content. Therefore, ensuring the accurate capture and separation of handwriting trajectories is a key step in extracting handwriting content. Through this extraction process, the handwriting content can be separated from the handwriting layer and provide basic data for subsequent recognition or storage operations, ensuring that the user's writing content is accurately reflected.

[0043] The handwritten content extraction method provided by the present invention realizes the separation of handwritten content and displayed content by introducing a transparent mask, and the handwritten trajectory is only recorded on the mask, avoiding content confusion and overlap. The transparent mask ensures the accurate extraction of handwritten content, and the trajectory can still be fully recorded even in complex scenes when the handwritten content is close to or overlaps the displayed content. In addition, the restriction on the handwritten area is removed, allowing users to write freely on the entire screen, providing users with an immersive experience similar to answering questions on paper.

[0044] Based on the above embodiment, in step 320, extracting handwritten content from the handwriting layer includes: Determining a region position of an expected handwriting region on the handwriting layer based on the content layout of the content display layer; Based on the area position of the expected handwriting area, handwriting content is extracted from the handwriting layer.

[0045] Specifically, the content layout of the content display layer determines the layout of each display element (such as text, images, etc.). Based on the content layout, the area where the user may write can be accurately inferred. For example, when a user answers a question below a test question, the expected handwriting area (i.e., the answer area) is usually located in the blank area below the question stem or in the answer box, and the content layout of the content display layer will clarify the specific location of the area. For example, in some cases, if the test content is compactly laid out, the expected handwriting area may be close to the bottom of the question or other information. By analyzing these content layout details, the expected handwriting area when the user answers can be determined, and the corresponding area in the handwriting layer can be locked to ensure that subsequent content extraction operations can be accurately located.

[0046] Based on the determined position of the expected handwriting area, the corresponding handwriting content can be extracted from the handwriting layer. In this process, the handwriting trajectory recorded on the handwriting layer will be accurately extracted, and the extraction range is mainly screened according to the position of the expected handwriting area. By identifying the positional relationship between the handwriting trajectory and the expected area, it can be ensured that the extracted handwriting content corresponds to the user's actual answer area. If the user's handwriting trajectory is slightly offset from the predetermined expected handwriting area, the extraction range is adjusted through solutions such as image connectivity algorithms or trajectory line extraction algorithms to ensure that all valid handwriting traces can be correctly extracted. At the same time, the extraction process will filter out traces that are not related to the expected handwriting area, thereby preventing irrelevant content from interfering with the accuracy of the handwriting content.

[0047] Through this handwriting area positioning and trajectory extraction based on content layout, the handwriting content can be accurately captured. Determining the expected handwriting area based on the content layout information of the content display layer can ensure that the extracted handwriting content matches the displayed content. At the same time, no matter the user's handwriting trajectory is close to the edge or adjacent to the display element, it only needs to determine the position and correspondence between the handwriting trajectory and the expected handwriting position. This method can accurately extract the relevant handwriting content and ensure the accuracy and completeness of the extraction.

[0048] Based on the above embodiment, the determining the area position of the expected handwriting area on the handwriting layer based on the content layout of the content display layer includes: Based on the content layout of the content display layer, traverse all display contents in the content display layer to obtain a position offset of each display content compared to the content layout; Based on the position offset of each of the displayed contents compared to the content layout and the relative position between each of the displayed contents and the corresponding expected handwriting area, the area position of the expected handwriting area corresponding to each of the displayed contents is determined.

[0049] Specifically, Figure 4 FIG. 1 is a flow chart of determining the area position of the desired handwriting area provided by the present invention. Figure 4 As shown, first, it is necessary to traverse all display contents in the content display layer. This step includes traversing each item contained in the display content in the content display layer one by one, such as text, test questions, pictures or other elements, based on the content layout of the content display layer. The purpose of this step is to find the specific position of each display content in the content display layer to facilitate subsequent calculations. Next, calculate the position offset of the display content relative to the content layout. This position offset combined with the content layout can reflect the specific position of the display content in the entire page or layout, ensuring that the relevant expected handwriting area can be accurately located later.

[0050] The next step is to extract the expected handwriting area of ​​the displayed content and its location information. The displayed content is often associated with a corresponding expected handwriting area, such as the answer area below the test question or the annotation area next to the image. In this process, the location information of the expected handwriting area extracted includes the specific location and size of the area.

[0051] Finally, add the position offset of the content layout and determine the final position of the expected handwriting area. After obtaining the expected handwriting area corresponding to each displayed content, it is necessary to further determine the specific position of the expected handwriting area in combination with the overall offset information of the content layout. In this way, it can ensure that the expected handwriting area is consistent with the entire page layout, ensure that the handwritten content can be closely associated with the corresponding displayed content, and avoid position deviation.

[0052] Based on the above embodiment, the step of extracting handwritten content from the handwriting layer based on the area position of the expected handwriting area includes: rasterizing the handwritten layer into a handwritten image, and extracting a connected domain subgraph from the handwritten image; Determine a connected domain subgraph associated with the expected handwriting area based on the area position of the expected handwriting area and the subgraph position of the connected domain subgraph; The handwriting content is extracted from the connected domain subgraph associated with the expected handwriting area.

[0053] Specifically, the handwriting trajectory recorded in the handwriting layer initially exists in the form of vectorized digital trajectory data. For subsequent processing, the handwriting layer needs to be rasterized. Rasterization converts the handwriting layer into an image composed of pixels, making the handwriting content easier to further analyze and process. The rasterized image, i.e., the handwriting image, is a pixelated representation of the handwriting content, which can clearly reflect the shape, strokes, and connectivity of the handwriting trajectory, providing a basis for subsequent operations. Through this process, the handwriting trajectory is refined into image pixels, which is convenient for segmentation, analysis, and extraction.

[0054] Next, it is necessary to extract the connected domain subgraph from the rasterized handwritten image. A connected domain subgraph is a continuous area composed of adjacent pixels in a handwritten image, which usually represents a part of the user's handwriting or a single handwritten trace. Connected domain analysis is an image processing technique used to identify connected regions in an image. For example, a character, a letter, or a symbol can constitute a connected domain subgraph. Through this step, the entire handwritten image can be divided into multiple independent subgraphs, each of which represents a part of the user's handwriting. This operation ensures a fine division of the handwritten content, so that each connected region can be analyzed and processed independently. By extracting the connected domain subgraph, the confusion of the handwritten content can be effectively avoided and the integrity of each handwritten trace can be ensured.

[0055] After the connected domain subgraph is extracted, the connected domain subgraph associated with the expected handwriting area is determined based on the position of the expected handwriting area and the subgraph position of the connected domain subgraph. The position of the expected handwriting area has been determined in advance based on the content layout of the content display layer, and the specific position of each connected domain subgraph is calculated when the subgraph is extracted. By comparing the actual position of the connected domain subgraph with the position of the expected handwriting area, it can be determined which subgraphs are associated with the area. For those subgraphs that are within the expected handwriting area or adjacent to the expected handwriting area, they are judged to be related subgraphs. Through this position comparison method, handwritten content that does not belong to the expected handwriting area can be effectively filtered out to ensure that the extracted handwritten content accurately matches the user's operation in the expected handwriting area. In this way, the handwritten content can be avoided from being confused with irrelevant areas, thereby improving the extraction accuracy.

[0056] Once the connected domain subgraphs associated with the expected handwriting area are determined, the handwriting content will be extracted from these subgraphs. By splicing the handwriting trajectories of these subgraphs together, the user's complete handwriting content can be formed. This process ensures the accuracy of handwriting content extraction and avoids the situation of mistakenly extracting irrelevant content. For example, when the user fills in the test answer area, the extracted handwriting content will only correspond to the handwriting trajectories in the test answer area and will not include handwriting trajectories in other areas. This operation ensures that the handwriting content of each expected handwriting area can be extracted independently and accurately.

[0057] Through this comprehensive method based on rasterization, connected domain subgraph extraction and position association, the user's handwriting content can be accurately extracted from the handwriting layer. No matter where the handwriting content is located, as long as it is associated with the expected handwriting area, it can be fully captured. This process not only improves the accuracy of handwriting content extraction, but also enhances the adaptability to complex handwriting scenes, providing a high-quality data foundation for subsequent recognition and processing of handwriting content.

[0058] Based on the above embodiment, the step of determining the connected domain subgraph associated with the expected handwriting area based on the area position of the expected handwriting area and the subgraph position of the connected domain subgraph includes: Based on the area position of the expected handwriting area and the subgraph position of the connected domain subgraph, determining the position intersection between the expected handwriting area and the connected domain subgraph, wherein the position intersection is the number of pixels of the connected domain subgraph in the expected handwriting area, or the area of ​​the rectangular outline of the connected domain subgraph in the expected handwriting area; Based on the position intersection situation, it is determined whether the connected domain subgraph is associated with the expected handwriting area.

[0059] Specifically, determining whether the expected handwritten area is associated with the connected domain subgraph is achieved by judging the positional intersection of the two. The positional intersection can be judged in two ways: the number of pixel hits where the connected domain subgraph is located and the area of ​​the rectangular outline where the connected domain subgraph is located.

[0060] The method for judging the number of hit pixels in the connected domain subgraph is to judge the correlation by calculating the coverage of the pixels of the connected domain subgraph in the expected handwriting area. Specifically, the connected domain subgraph is composed of continuous pixels in the handwriting image, and the number of pixels of each connected domain subgraph in the expected handwriting area falls within the expected handwriting area. If most of the pixels in the connected domain subgraph are located in the expected handwriting area, it means that the connected domain subgraph is highly correlated with the expected area. At the same time, a reasonable threshold is set to judge the correlation. For example, if more than a certain proportion of pixels in the connected domain subgraph hit the expected area, then the connected domain subgraph will be considered to be associated with the expected handwriting area. This method is particularly suitable for handling situations where the handwriting content partially overlaps with the area. By accurately calculating the hit pixels, it can avoid missing some handwriting trajectories.

[0061] The method for determining the area of ​​the rectangular outline of the connected domain subgraph is to use the rectangular outline of the connected domain subgraph as the basis to calculate the intersection between it and the rectangle of the expected handwriting area. Each connected domain subgraph is represented as a rectangle with boundary information (such as the coordinates of the upper left corner, width and height). Through geometric analysis, the overlapping area of ​​the connected domain subgraph rectangle and the expected handwriting area rectangle can be calculated. If the ratio of the intersection area to the total area of ​​the connected domain subgraph rectangle exceeds a preset threshold (such as 50%), the connected domain subgraph is associated with the expected handwriting area; otherwise, the connected domain subgraph is considered irrelevant. The intersection of rectangles is quantified by the overlapping area, which simplifies the process of processing complex handwriting shapes. In order to optimize the extraction accuracy, the connected domain subgraph of the handwriting trajectory is represented in the form of a matrix. The rectangular outline of each connected domain subgraph can be represented by a matrix, the dimension of the matrix corresponds to the width and height of the rectangle, and each element in the matrix represents a pixel point in the handwriting image, 1 indicates that the point is handwriting, and 0 indicates a blank area. Through this matrix representation, the connected domain subgraph can be processed more efficiently, and intersection calculation and position matching can be performed in the matrix, thereby optimizing the accuracy and processing efficiency of handwriting trajectory extraction.

[0062] Based on the judgment results of the two methods for intersection judgment, the relevance of the connected domain subgraph and the expected handwriting area is further determined. When the number of pixel hits or the overlapping area of ​​the rectangular outline exceeds the set threshold, the subgraph is considered to be related to the expected handwriting area and its handwriting content will be extracted. If the association standard is not met, the subgraph will be excluded to avoid the error of extracting irrelevant content. At the same time, through these two judgment methods, the extraction process of handwriting content can not only cope with the situation where the handwriting trajectory is scattered or close to the edge, but also ensure the accuracy of the extraction results, thereby ensuring that the handwriting content is highly matched with the expected handwriting area.

[0063] Based on the above embodiment, the step of determining whether the connected domain subgraph is associated with the expected handwriting area based on the position intersection situation further includes: When it is determined based on the position intersection that the connected domain subgraph is not associated with the expected handwriting area, it is determined based on the position proximity between the connected domain subgraph and the expected handwriting area whether the connected domain subgraph is associated with the expected handwriting area.

[0064] Specifically, if the connected domain subgraph is determined not to belong to any expected handwriting area through the position intersection, that is, the pixel hit rate of the connected domain subgraph is low in the preliminary intersection analysis, or the overlapping area of ​​its rectangular outline and the expected area is not enough to determine its relevance, then it is preliminarily considered that the connected domain subgraph is irrelevant to the expected handwriting area, and the next step of position adjacent analysis is entered. Because even if the connected domain subgraph does not directly intersect with the expected handwriting area, it may be associated due to the proximity of the handwriting traces.

[0065] The judgment of positional proximity is based on whether there is a certain pixel density intersection or rectangular outline overlap area between the connected domain subgraph and the expected handwriting area. This means that even if the two areas do not intersect directly, if the pixel distribution of the connected domain subgraph partially overlaps with the edge of the expected handwriting area, or there is a certain contact or overlap area between their outlines, then these factors can be used to determine that the connected domain subgraph has a potential correlation with the expected handwriting area.

[0066] The pixel density intersection is determined based on the pixel distribution of the border pixels of the connected domain subgraph and the edge of the expected handwriting area. If there is a certain pixel overlap or dense intersection between the edge pixels of the connected domain subgraph and the pixel area of ​​the expected handwriting area (i.e., part of the handwriting track that is close to the edge but not completely in the area), then it can be considered that there is a potential association between the subgraph and the expected handwriting area. This judgment is suitable for handling situations where the handwriting slightly deviates from the area, but may still be related to the handwriting content due to the density of the handwriting or proximity to the answer area.

[0067] The method for determining the overlapping area of ​​rectangular outlines is to calculate the partial overlapping area between the rectangular outline of the connected domain subgraph and the outline of the expected handwriting area. If the rectangular outline of the connected domain subgraph has some contact or overlap with the outline of the expected handwriting area, even if the overlapping area is small (for example, only part of the outline is covered), it may still indicate that the subgraph has a certain correlation with the expected area. This judgment method can effectively capture the associated handwriting trajectory when dealing with irregular handwriting shapes or edge handwriting close to the area.

[0068] If in the analysis of adjacent situations, there is no pixel density intersection and contour overlap area between the connected domain subgraph and the expected handwriting area, that is, the connected domain subgraph does not have any edge pixels touching or overlapping with the expected handwriting area, and cannot be judged as related content by partial overlap of rectangular contours, then it is further confirmed that the subgraph has no association with the expected handwriting area. In this case, the connected domain subgraph will be considered as not belonging to any expected handwriting area, so it will not be extracted. Such subgraphs may be irrelevant handwriting, misoperation, or handwriting that deviates too far from the user, and fail to form sufficient association with the expected area. In order to avoid erroneous extraction of irrelevant content, these unrelated connected domain subgraphs will be discarded to ensure that only handwriting trajectories directly related to or adjacent to the expected handwriting area are extracted.

[0069] This neighboring judgment based on pixel density intersection or rectangular outline overlap area can further ensure the complete extraction of handwritten content. Even if the association is not detected in the initial intersection judgment, the positional neighboring analysis in this step can still capture the handwritten content close to the edge of the expected area, ensuring that those handwritings that are slightly deviated are not missed.

[0070] Based on the above embodiment, the step of determining whether the connected domain subgraph is associated with the expected handwriting area based on the positional proximity between the connected domain subgraph and the expected handwriting area includes: In the case where the connected domain subgraph is adjacent to the multiple expected handwriting regions, the connected domain subgraph is segmented to obtain multiple segmented subgraphs as connected domain subgraphs respectively associated with the multiple expected handwriting regions.

[0071] Specifically, when a connected domain subgraph does not directly intersect with the expected handwriting area and is adjacent to multiple expected handwriting areas at the same time, it is impossible to accurately extract the user's handwriting content by simply relying on association judgment. Therefore, such a connected domain subgraph across multiple areas must be further processed, specifically by using a segmentation ratio algorithm or decomposing the subgraph into multiple segmented subgraphs based on pixel density, and assigning these segmented subgraphs to the corresponding expected handwriting areas.

[0072] The segmentation process is based on the positional relationship between the connected domain subgraph and each expected handwriting area. First, traverse all connected domain subgraphs, analyze the topological position of each connected domain subgraph, and determine whether the connected domain subgraph is associated with multiple expected handwriting areas at the same time. If it is found that the position of the connected domain subgraph crosses the boundaries of multiple areas, use the segmentation ratio algorithm to segment the connected domain subgraph according to the proportion of each area it covers. For example, if most of the handwriting of the connected domain subgraph is located in an expected handwriting area, but a small part extends to another area, the segmentation algorithm will split the subgraph into two segmented subgraphs based on the coverage ratio of the subgraph in each area.

[0073] Another segmentation method is segmentation by pixel density. This method analyzes the pixel density of the connected domain subgraph in each adjacent expected handwriting area to evaluate the correlation with the expected handwriting area. In this way, the connected domain subgraph can be split into multiple segmented subgraphs, and these segmented subgraphs are respectively assigned to the corresponding expected handwriting areas to ensure that each expected handwriting area can accurately receive its related handwriting content. This segmentation process ensures that the handwriting content can be accurately extracted and avoids the confusion of attribution caused by the connected domain subgraph spanning multiple areas. Each segmented subgraph after segmentation will be associated with only one expected handwriting area, which not only avoids the misextraction of content, but also ensures that the handwriting content in different areas will not interfere with each other.

[0074] Based on the above embodiment, the step of extracting handwritten content from the handwriting layer based on the area position of the expected handwriting area includes: Determine a handwriting track associated with the expected handwriting area based on the area position of the expected handwriting area and the track position of the handwriting track in the handwriting layer; Handwriting content is extracted from the handwriting trajectory associated with the expected handwriting area.

[0075] Specifically, the handwriting content extraction process combines each handwriting trajectory of the user, and further determines the relevance of the handwriting content to the expected handwriting area by recording and analyzing these handwriting trajectories. This process starts with each handwriting operation of the user, and records the handwriting trajectory as a handwriting trajectory list. Each handwriting trajectory describes the user's handwriting path, location, starting point, and end point in detail. As the core data of the handwriting content, it provides the basis for subsequent matching and extraction of the expected handwriting area.

[0076] Based on the acquired position of the expected handwriting area, the intersection between the handwriting trajectory and the expected handwriting area is further analyzed. The position of the expected handwriting area is determined by the content layout, indicating the area where the user expects to write. By comparing the position of the handwriting trajectory with the expected handwriting area, it is possible to determine whether the handwriting trajectory intersects or overlaps with the expected handwriting area. The specific analysis may include calculating the overlapping portion of the pixels or path of the handwriting trajectory and the expected handwriting area. If the vast majority of the pixels of the trajectory fall within the expected handwriting area, the trajectory can be considered to be associated with the expected handwriting area. If the handwriting trajectory only partially contacts the expected handwriting area, it is necessary to further analyze the adjacency of the trajectory to ensure that valid writing content is captured.

[0077] After completing the analysis of intersections and adjacencies, the correlation between the handwriting trajectory and the expected handwriting area will be clearly determined. If a handwriting trajectory has a strong correlation with the expected handwriting area, that is, most of the trajectory is located in the expected handwriting area, then the handwriting trajectory will be marked as the handwriting content belonging to the expected handwriting area. If the handwriting trajectory has only a small amount of intersection with the expected handwriting area, it is considered that the handwriting trajectory is adjacent to the expected handwriting area and is also marked as the handwriting content belonging to the expected handwriting area. If the handwriting trajectory is completely unrelated to any expected handwriting area, the trajectory will not be extracted to avoid accidentally extracting irrelevant content.

[0078] Based on the above embodiment, the determining of the handwriting trajectory associated with the expected handwriting area based on the area position of the expected handwriting area and the trajectory position of the handwriting trajectory in the handwriting layer includes: Based on the area position of the expected handwriting area and the track position of the handwriting track in the handwriting layer, determining the intersection of the expected handwriting area and the handwriting track; Based on the position intersection situation, it is determined whether the handwriting trajectory is associated with the expected handwriting area.

[0079] Specifically, the position of the expected handwriting area is usually clearly given by the content layout, representing the user's expected handwriting area, such as the answer box, annotation area, etc. The track position of the handwriting trajectory records the path information of the user's actual handwriting on the handwriting layer. By comparing the handwriting trajectory with the position of the expected handwriting area, the intersection relationship between the two can be analyzed. Regardless of whether the trajectory partially enters the expected area or is close to the area, the overlap or contact between them will be calculated to further determine the relationship between the handwriting trajectory and the expected handwriting area.

[0080] According to the result of intersection analysis, it is determined whether the handwriting track is associated with the expected handwriting area. If the handwriting track is associated with the expected handwriting area, it is considered that the user's handwriting content belongs to the expected handwriting area, and the handwriting track will be extracted and processed. If there is no valid association, the handwriting track will not be included in the extraction range to avoid irrelevant content interfering with the extraction of the handwriting content of the expected handwriting area.

[0081] In addition, it is also considered that in some cases, the handwriting trajectory may be associated with multiple expected handwriting areas. When a handwriting trajectory crosses multiple expected handwriting areas at the same time, it is necessary to further judge its association. At this time, a logic can be adopted: assign the handwriting trajectory to the expected handwriting area passed for the first time. That is to say, according to the path of the handwriting trajectory, identify the expected handwriting area that the handwriting trajectory first enters or passes through, and associate the handwriting trajectory with the expected handwriting area. This allocation method ensures the correct attribution of the handwriting trajectory in multiple adjacent areas, and avoids the confusion caused by the handwriting trajectory being simultaneously classified into multiple expected handwriting areas.

[0082] This process ensures a more precise association between the handwriting trajectory and the expected handwriting area, effectively improving the accuracy and reliability of handwriting content extraction, especially when the handwriting trajectory is close to or overlaps with the area boundary.

[0083] Based on the above embodiment, the step of determining whether the handwriting trajectory is associated with the expected handwriting area based on the position intersection situation further includes: When it is determined that the handwriting track is not associated with the expected handwriting area based on the position intersection, it is determined whether the handwriting track is associated with the expected handwriting area based on the position proximity between the handwriting track and the expected handwriting area.

[0084] Specifically, in order to prevent missing valid handwriting content related to the expected handwriting area, it is necessary to further analyze the proximity of the handwriting trajectory to the expected area. Even if the handwriting trajectory does not directly intersect the expected handwriting area, the handwriting trajectory may be adjacent to or close to the edge of the expected handwriting area. In this case, if the path of the handwriting trajectory is close to the expected handwriting area and there is a certain proximity relationship with the boundary of the area, then it can be inferred that the handwriting trajectory may be associated with the expected handwriting area. The judgment criteria for the adjacent situation are based on the contact between the handwriting trajectory and the edge of the expected area. If some parts of the handwriting trajectory are close to or in contact with the boundary of the area, it indicates that the trajectory may still be valid handwriting content, although the trajectory has not completely entered the expected area.

[0085] By combining the dual judgment of position intersection and position proximity, the extraction of handwriting trajectories is ensured to be more comprehensive and accurate. When the handwriting trajectory is not directly associated with the expected handwriting area, the analysis of the proximity situation can supplement the shortcomings of the intersection judgment, especially when dealing with handwriting content close to the edge of the area, it can capture more potential related handwriting trajectories and ensure the accuracy and completeness of the extraction results.

[0086] Based on the above embodiment, after extracting the handwritten content from the handwritten layer, the method further includes: The handwritten content is displayed in the content display layer based on the area position of the expected handwritten area.

[0087] Specifically, after extracting the user's handwriting content in the handwriting layer, the extracted handwriting content is accurately projected into the content display layer according to the regional position of the expected handwriting area, so that the user can see his or her handwriting content appear at a specific location. For example, when answering questions, the user can directly handwrite the solution process and answer in the answer area on the screen. At this time, the handwriting content is actually on the handwriting layer. After extracting the user's handwriting content, it is displayed at the regional position of the expected handwriting area in the corresponding display layer, so that the user can see his or her handwriting content naturally presented in the answer area, as if writing on actual paper.

[0088] By displaying the extracted handwritten content in real time in the content display layer, the paper-like writing function is realized, ensuring that the user's writing experience on electronic devices is similar to that of paper writing, enhancing the user's sense of naturalness and immersion

[0089] Based on the above embodiments, Figure 5It is a flow chart of the pixel number determination method provided by the present invention in the answering scenario. As shown in the figure, the answering mask image is first rasterized into a whole picture. The purpose of rasterization is to convert the handwriting track from the form of vector data or track points into an image composed of pixel points, which is convenient for subsequent image processing. In the rasterized image, all connected domain subgraphs in the image are extracted by calculating connectivity (i.e., judging whether adjacent pixels constitute a complete area). Each connected domain subgraph represents a part of the handwritten content, and the position of these subgraphs in the image is recorded. Then, all the extracted connected domain subgraphs are traversed, and their relationship with the answering area is analyzed one by one. Then, for each subgraph, the pixel distribution or density in each answering area is calculated. The pixel density indicates how many parts of the subgraph are located in a certain answering area, which helps to judge the relevance of the subgraph to the answering area. Subsequently, by calculating the number of pixels in each answering area, the subgraph is assigned to the answering area with the highest pixel density, indicating that the subgraph is most likely to belong to the answering area. This allocation mechanism ensures that the handwritten content can be accurately classified into the corresponding answering area. For those sub-images that are not associated with any answer area, a list of unassociated sub-images is obtained. These sub-images represent handwritten content that cannot be associated with any area under the current answer area layout.

[0090] Traverse all answer areas and unrelated sub-image lists again. If an answer area is adjacent to an unrelated sub-image, associate the sub-image with the answer area. If an unrelated sub-image is not effectively associated with any answer area, discard the sub-image and consider it to be unrelated content, and do not process it further. For each answer area, extract all sub-images associated with it and combine them into one image. This image represents the user's handwritten content in the answer area.

[0091] Finally, the merged handwritten content image of the answer area is passed to the optical character recognition engine to recognize the user's handwritten text and further process it. The entire process ensures that the user's handwritten content is accurately extracted from the answer area through multiple traversals and image connectivity analysis, avoiding interference from irrelevant handwriting.

[0092] Based on the above embodiments, Figure 6 It is a flow chart of the rectangular contour judgment method provided by the present invention in the answering scenario. As shown in the figure, the process describes how to extract effective answer handwriting from the user's handwriting on the answer mask, and finally identify the user's handwritten answer. First, the handwriting trajectory is converted into a rasterized image form, so that each handwriting content is converted into an image composed of pixels, which is convenient for subsequent analysis and processing. Then, through connectivity calculation, all connected domain subgraphs in the image are identified and extracted, each subgraph represents an independent part of the user's handwriting, and the specific positions of these subgraphs are recorded. Subsequently, each connected domain subgraph is traversed and compared with the position of the answer area to determine whether these subgraphs are adjacent to or intersecting with a certain answer area.

[0093] When judging whether a sub-image intersects with the answer area, the contour area method is used. The sub-image contour and the answer area matrix are extracted, and the overlapping area of ​​the two is calculated, that is, by comparing the matrix elements of the sub-image and the answer area pixel by pixel, their overlap is judged. If there are valid pixels in both the sub-image and the answer area matrix at the same position (for example, both are "1"), the two are considered to intersect. Through this combined method, the association between the handwriting trajectory and the answer area can be determined more accurately. If the overlapping area of ​​the sub-image contour and the answer area matrix reaches a certain threshold, the sub-image is considered to intersect with the answer area, and the sub-image is classified as the answer area.

[0094] If the subgraph does not intersect with the answer area, the subgraph is temporarily not associated with any answer area, forming a list of unrelated subgraphs. Next, all unrelated subgraphs are collected and further judged with the answer area. After that, the list of unrelated subgraphs is traversed to determine whether these subgraphs are adjacent to the answer area. If an unrelated subgraph is adjacent to the answer area, the subgraph is assigned to the answer area to ensure that the missing handwritten content can be associated with the correct area.

[0095] If a sub-image is adjacent to or intersects with multiple answer areas at the same time, it will be segmented based on the proportion or pixel density covered by the sub-image in the answer area and assigned to different answer areas to ensure that each answer area is only associated with relevant handwritten content.

[0096] After all the association operations are completed, the sub-image position list associated with the answer area is merged to generate a minimum bounding rectangle containing all sub-images, which is recorded as the answer position. This rectangle represents the boundary range of all the handwritten content of the user in the answer area. If the merged answer position is associated with multiple answer areas, the answer position is split into multiple parts in proportion and assigned to different answer areas. Based on the position information of the handwritten content, the sub-images are matched with the answer areas, and these contents are used as the final answer results and recorded as the handwritten content of the answer area.

[0097] Finally, the complete handwritten content image of each answer area is passed to the optical character recognition engine to convert the user's handwriting into text format for subsequent processing and analysis. Through this process, the handwritten content can be accurately associated with the corresponding answer area, ensuring that the extraction and recognition of handwritten content can be accurate and without omission in complex scenarios.

[0098] Based on the above embodiments, Figure 7It is a flow chart of the handwriting trajectory extraction method provided by the present invention in the answering scenario. The process describes how to analyze each handwriting trajectory of the user, associate it with a predetermined answer area, and extract valid handwriting content. First, calculate the answer trajectory of each writing of the user, and record these trajectories in the trajectory line list. This step ensures that all handwriting operations are tracked and recorded for subsequent processing. Next, traverse the answer area list and the trajectory line list, compare each answer area and trajectory line, and determine whether the trajectory line intersects with the answer area. If the trajectory intersects with the answer area, the trajectory line is attributed to the answer area. If there is no intersection, the trajectory line is temporarily not processed.

[0099] For tracks that are not associated with any answer area, they are included in the unrelated track list. At this time, all answer areas and unrelated tracks are further judged to check whether these tracks are adjacent to the answer area. If it is confirmed that an unrelated track is adjacent to the answer, the track line will be assigned to the answer area; otherwise, the track line will be considered irrelevant and discarded.

[0100] When a track line passes through multiple answering areas, the track line will first identify the intersection or proximity of multiple answering areas. The track line will be assigned to the answering area it first enters. Even if the track line passes through other answering areas later, it will still be assigned to the answering area that it passed first. This processing method ensures that each track line is associated with only one answering area.

[0101] Then, all the track line lists associated with the answer area are combined into complete handwriting track data, and then synthesized into a complete answer track list in order. This can reproduce the user's writing order and ensure that the handwriting content is fully recorded.

[0102] Finally, the combined answer track image is passed to the optical character recognition engine to extract the user's handwritten text content and complete the recognition. This process ensures that each user's answer track can be correctly identified, associated and processed.

[0103] The handwritten content extraction device provided by the present invention is described below. The handwritten content extraction device described below and the handwritten content extraction device described above can be referred to each other.

[0104] Based on the above embodiments, Figure 8It is a flow chart of extracting the user's answer content provided by the present invention. As shown in the figure, the figure shows the whole process from test paper rendering and typesetting to handwriting trajectory extraction and recognition. First, the content of the test paper will be rendered and typeset. The test paper typesetting may include various elements, such as text, pictures, tables, formulas, etc., which will be displayed on the electronic screen. Above the typeset test paper content, a transparent answer mask is added. This mask is used to capture the user's handwriting operation and record the handwriting trajectory without affecting the display of the underlying test paper content. When the user starts to answer, the user's handwriting trajectory is recorded, that is, the path when the finger or stylus touches the screen. As the user writes, the handwriting trajectory is drawn on the transparent answer mask. This trajectory is the writing content actually input by the user. After the user completes the answer, the correction stage is entered. At this time, all handwriting trajectories have been recorded on the answer mask.

[0105] In order to perform marking, the position of each question and the corresponding answer area will first be extracted from the content of the test paper rendering layout. Different processing methods are used according to different device types. If the device supports image connectivity, the answer result will be rasterized into an image, and the handwriting will be extracted by calculating the connectivity of the image. If the device supports trajectory point line extraction, the trajectory line position of the user's handwriting trajectory will be extracted.

[0106] In devices that support image connectivity, after the answer result is rasterized into an image, the image connectivity calculation is performed to determine the positional relationship between the connected area and the answer area. In devices that support trajectory point and line extraction, the coordinates of the handwritten trajectory are extracted, and the relationship between the trajectory line and the answer area is determined by these coordinates. By traversing the position of each sub-image or trajectory line, it is analyzed whether they intersect or are adjacent to a certain answer area.

[0107] Next, it is necessary to determine whether the subgraph or trajectory is associated with multiple answer areas at the same time. If the trajectory intersects with multiple answer areas at the same time, its distribution in different answer areas will be further analyzed to determine the answer area to which the trajectory ultimately belongs. For subgraphs or trajectory lines that are only associated with a single answer area, they are directly assigned to that answer area. When a trajectory line or subgraph is related to multiple answer areas at the same time, the answer area to which it ultimately belongs will be determined based on their distribution ratio in different answer areas. Subsequently, all trajectory lines or subgraphs associated with the answer area are extracted and integrated into a whole.

[0108] Finally, all sub-graphs or track lines related to the answer area are summarized and input into the optical character recognition engine or handwriting track recognition engine for recognition. The optical character recognition engine or handwriting track recognition engine converts the sub-graphs or track lines into text, and finally obtains the user's handwritten answer. Through this process, it can ensure that the user's handwriting content is accurately matched to the corresponding answer area, and whether the device supports connectivity calculation or track coordinate extraction, it can achieve accurate extraction and recognition of the user's answer.

[0109] Fig. 9 is a schematic diagram of the structure of the handwritten content extraction device provided by the present invention, such as Fig. 9 As shown, the device comprises: A handwriting operation receiving unit 910, configured to receive a handwriting operation on a handwriting layer, wherein the handwriting layer is a transparent mask layer disposed above the content display layer and overlapping the content display layer; The handwriting content extraction unit 920 is used to display the handwriting track corresponding to the handwriting operation on the handwriting layer in response to the handwriting operation, and extract the handwriting content from the handwriting layer.

[0110] The device provided by the embodiment of the present invention achieves separation of handwritten content from displayed content by introducing a transparent mask, and the handwritten trajectory is only recorded on the mask, avoiding content confusion and overlap. The transparent mask ensures accurate extraction of handwritten content, and the trajectory can still be fully recorded even in complex scenarios when the handwritten content is close to or overlaps the displayed content. In addition, the restrictions on the handwriting area are removed, allowing users to write freely on the entire screen, providing users with an immersive experience similar to answering questions on paper.

[0111] Based on any of the above embodiments, the handwritten content extraction unit is specifically used for: Determining a region position of an expected handwriting region on the handwriting layer based on the content layout of the content display layer; Based on the area position of the expected handwriting area, handwriting content is extracted from the handwriting layer.

[0112] Based on any of the above embodiments, the handwritten content extraction unit is further used for: Based on the content layout of the content display layer, traverse all display contents in the content display layer to obtain a position offset of each display content compared to the content layout; Based on the position offset of each of the displayed contents compared to the content layout and the relative position between each of the displayed contents and the corresponding expected handwriting area, the area position of the expected handwriting area corresponding to each of the displayed contents is determined.

[0113] Based on any of the above embodiments, the handwritten content extraction unit is specifically used for: rasterizing the handwritten layer into a handwritten image, and extracting a connected domain subgraph from the handwritten image; Determine a connected domain subgraph associated with the expected handwriting area based on the area position of the expected handwriting area and the subgraph position of the connected domain subgraph; The handwriting content is extracted from the connected domain subgraph associated with the expected handwriting area.

[0114] Based on any of the above embodiments, the handwritten content extraction unit is specifically used for: Based on the area position of the expected handwriting area and the subgraph position of the connected domain subgraph, determining the position intersection between the expected handwriting area and the connected domain subgraph, wherein the position intersection is the number of pixels of the connected domain subgraph in the expected handwriting area, or the area of ​​the rectangular outline of the connected domain subgraph in the expected handwriting area; Based on the position intersection situation, it is determined whether the connected domain subgraph is associated with the expected handwriting area.

[0115] Based on any of the above embodiments, the handwritten content extraction unit is specifically used for: When it is determined based on the position intersection that the connected domain subgraph is not associated with the expected handwriting area, it is determined based on the position proximity between the connected domain subgraph and the expected handwriting area whether the connected domain subgraph is associated with the expected handwriting area.

[0116] Based on any of the above embodiments, the handwritten content extraction unit is specifically used for: In the case where the connected domain subgraph is adjacent to the multiple expected handwriting regions, the connected domain subgraph is segmented to obtain multiple segmented subgraphs as connected domain subgraphs respectively associated with the multiple expected handwriting regions.

[0117] Based on any of the above embodiments, the handwritten content extraction unit is specifically used for: Determine a handwriting track associated with the expected handwriting area based on the area position of the expected handwriting area and the track position of the handwriting track in the handwriting layer; Handwriting content is extracted from the handwriting trajectory associated with the expected handwriting area.

[0118] Based on any of the above embodiments, the handwritten content extraction unit is specifically used for: Based on the area position of the expected handwriting area and the track position of the handwriting track in the handwriting layer, determining the intersection of the expected handwriting area and the handwriting track; Based on the position intersection situation, it is determined whether the handwriting trajectory is associated with the expected handwriting area.

[0119] Based on any of the above embodiments, the handwritten content extraction unit is specifically used for: When it is determined that the handwriting track is not associated with the expected handwriting area based on the position intersection, it is determined whether the handwriting track is associated with the expected handwriting area based on the position proximity between the handwriting track and the expected handwriting area.

[0120] Based on any of the above embodiments, the handwritten content extraction unit is specifically used for: Based on the area position of the expected handwriting area, the handwriting content is displayed in real time in the content display layer.

[0121] Fig.10 An example of a physical structure diagram of an electronic device is shown in FIG. Fig.10 As shown, the electronic device may include: a processor 1010, a communication interface 1020, a memory 1030 and a communication bus 1040, wherein the processor 1010, the communication interface 1020 and the memory 1030 communicate with each other through the communication bus 1040. The processor 1010 may call the logic instructions in the memory 1030 to execute the handwriting content extraction method, which includes: receiving a handwriting operation on a handwriting layer, wherein the handwriting layer is a transparent mask layer disposed above the content display layer and overlapping the content display layer; In response to the handwriting operation, a handwriting track corresponding to the handwriting operation is displayed on the handwriting layer, and handwriting content is extracted from the handwriting layer.

[0122] In addition, the logic instructions in the above-mentioned memory 1030 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the relevant technology or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0123] On the other hand, the present invention further provides a computer program product, the computer program product comprising a computer program, the computer program can be stored in a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the handwritten content extraction method provided by the above methods, the method comprising: receiving a handwriting operation on a handwriting layer, wherein the handwriting layer is a transparent mask layer disposed above the content display layer and overlapping the content display layer; In response to the handwriting operation, a handwriting track corresponding to the handwriting operation is displayed on the handwriting layer, and handwriting content is extracted from the handwriting layer. In another aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the handwriting content extraction method provided by the above methods is implemented, and the method includes: receiving a handwriting operation on a handwriting layer, wherein the handwriting layer is a transparent mask layer disposed above the content display layer and overlapping the content display layer; In response to the handwriting operation, a handwriting track corresponding to the handwriting operation is displayed on the handwriting layer, and the handwriting content is extracted from the handwriting layer. The above-described device embodiments are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.

[0124] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.

[0125] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for extracting handwritten content, characterized in that: include: receiving a handwriting operation on a handwriting layer, wherein the handwriting layer is a transparent mask layer disposed above the content display layer and overlapping the content display layer; In response to the handwriting operation, a handwriting track corresponding to the handwriting operation is displayed on the handwriting layer, and handwriting content is extracted from the handwriting layer.

2. The handwritten content extraction method according to claim 1, characterized in that: The extracting handwritten content from the handwritten layer includes: Determining a region position of an expected handwriting region on the handwriting layer based on the content layout of the content display layer; Based on the area position of the expected handwriting area, handwriting content is extracted from the handwriting layer.

3. The handwritten content extraction method according to claim 2, characterized in that: The determining the area position of the expected handwriting area on the handwriting layer based on the content layout of the content display layer includes: Based on the content layout of the content display layer, traverse all display contents in the content display layer to obtain a position offset of each display content compared to the content layout; Based on the position offset of each of the displayed contents compared to the content layout and the relative position between each of the displayed contents and the corresponding expected handwriting area, the area position of the expected handwriting area corresponding to each of the displayed contents is determined.

4. The handwritten content extraction method according to claim 2, characterized in that: The step of extracting handwritten content from the handwriting layer based on the area position of the expected handwriting area includes: rasterizing the handwritten layer into a handwritten image, and extracting a connected domain subgraph from the handwritten image; Determine a connected domain subgraph associated with the expected handwriting area based on the area position of the expected handwriting area and the subgraph position of the connected domain subgraph; The handwriting content is extracted from the connected domain subgraph associated with the expected handwriting area.

5. The handwritten content extraction method according to claim 4, characterized in that: The step of determining a connected domain subgraph associated with the expected handwriting area based on the area position of the expected handwriting area and the subgraph position of the connected domain subgraph comprises: Based on the area position of the expected handwriting area and the subgraph position of the connected domain subgraph, determining the position intersection between the expected handwriting area and the connected domain subgraph, wherein the position intersection is the number of pixels of the connected domain subgraph in the expected handwriting area, or the area of ​​the rectangular outline of the connected domain subgraph in the expected handwriting area; Based on the position intersection situation, it is determined whether the connected domain subgraph is associated with the expected handwriting area.

6. The handwriting content extraction method according to claim 5, characterized in that: The step of determining whether the connected domain subgraph is associated with the expected handwriting area based on the position intersection situation further includes: When it is determined based on the position intersection that the connected domain subgraph is not associated with the expected handwriting area, it is determined based on the position proximity between the connected domain subgraph and the expected handwriting area whether the connected domain subgraph is associated with the expected handwriting area.

7. The handwritten content extraction method according to claim 6, characterized in that: The determining whether the connected domain subgraph is associated with the expected handwriting area based on the positional proximity between the connected domain subgraph and the expected handwriting area includes: In the case where the connected domain subgraph is adjacent to the multiple expected handwriting regions, the connected domain subgraph is segmented to obtain multiple segmented subgraphs as connected domain subgraphs respectively associated with the multiple expected handwriting regions.

8. The handwriting content extraction method according to claim 2, characterized in that: The step of extracting handwritten content from the handwriting layer based on the area position of the expected handwriting area includes: Determine a handwriting track associated with the expected handwriting area based on the area position of the expected handwriting area and the track position of the handwriting track in the handwriting layer; Handwriting content is extracted from the handwriting trajectory associated with the expected handwriting area.

9. The handwritten content extraction method according to claim 8, characterized in that: The determining of the handwriting track associated with the expected handwriting area based on the area position of the expected handwriting area and the track position of the handwriting track in the handwriting layer includes: Based on the area position of the expected handwriting area and the track position of the handwriting track in the handwriting layer, determining the intersection of the expected handwriting area and the handwriting track; Based on the position intersection situation, it is determined whether the handwriting trajectory is associated with the expected handwriting area.

10. The handwriting content extraction method according to claim 9, characterized in that: The step of determining whether the handwriting track is associated with the expected handwriting area based on the position intersection situation further includes: When it is determined that the handwriting track is not associated with the expected handwriting area based on the position intersection, it is determined whether the handwriting track is associated with the expected handwriting area based on the position proximity between the handwriting track and the expected handwriting area.

11. The handwritten content extraction method according to any one of claims 2 to 10, characterized in that: After extracting the handwritten content from the handwritten layer, the method further includes: Based on the area position of the expected handwriting area, the handwriting content is displayed in real time in the content display layer.

12. A handwritten content extraction device, characterized in that: include: A handwriting operation receiving unit, used for receiving a handwriting operation on a handwriting layer, wherein the handwriting layer is a transparent mask layer disposed above the content display layer and overlapping the content display layer; The handwriting content extraction unit is used to display the handwriting track corresponding to the handwriting operation on the handwriting layer in response to the handwriting operation, and extract the handwriting content from the handwriting layer.

13. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the handwritten content extraction method according to any one of claims 1 to 11 is implemented.

14. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the handwritten content extraction method according to any one of claims 1 to 11 is implemented.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the handwritten content extraction method according to any one of claims 1 to 11 is implemented.

Images