Subject tool generation method and device, equipment, storage medium and program product

Abstract

The invention relates to a subject tool generation method and device, equipment, a storage medium and a program product, and relates to the technical field of computers. The method comprises: displaying a first user interface; receiving a writing operation on the first user interface; in response to the writing operation of the user, displaying handwriting corresponding to the writing operation on the first user interface; after it is confirmed that the writing operation is finished, determining a handwriting region and an element type corresponding to the handwriting, and performing text recognition on the handwriting to obtain text content; generating an interactive region based on the handwriting region, and in response to a trigger operation for the interactive region, generating a subject tool corresponding to the handwriting according to the element type and the text content; a subject tool is displayed on a first user interface. By adopting the method, the subject tool generation efficiency can be improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, device, storage medium, and program product for generating subject-specific tools. Background Technology

[0002] In teaching scenarios, it is often necessary to generate subject-specific tools to assist teaching and improve teaching effectiveness and efficiency. Traditionally, this involves opening the corresponding tool in an application that supports subject-specific tool generation and inputting the relevant content via keyboard. However, this method is cumbersome, especially inconvenient when using a keyboard on a large teaching screen, resulting in low efficiency in generating subject-specific tools. Summary of the Invention

[0003] Therefore, it is necessary to provide a method, apparatus, device, storage medium, and program product for generating subject-specific tools that can improve the efficiency of generating subject-specific tools, in response to the above-mentioned technical problems.

[0004] Firstly, this application provides a method for generating subject-specific tools, the method comprising:

[0005] Display the first user interface;

[0006] Receive writing operations on the first user interface;

[0007] In response to the user's writing operation, the handwriting corresponding to the writing operation is displayed on the first user interface;

[0008] After confirming that the writing operation is completed, determine the handwriting area and element type corresponding to the handwriting, and perform text recognition on the handwriting to obtain the text content;

[0009] An interactive area is generated based on the handwriting area, and in response to a trigger operation on the interactive area, a subject-specific tool corresponding to the handwriting is generated according to the element type and the text content.

[0010] The subject-specific tools are displayed on the first user interface.

[0011] In one embodiment, determining the handwriting region and element type corresponding to the handwriting includes:

[0012] The handwriting is rendered to obtain a handwriting image;

[0013] The handwriting image is input into a trained target detection model to perform target detection on the handwriting image, thereby obtaining the handwriting region and element type corresponding to the handwriting.

[0014] In one embodiment, the object detection model is trained in the following manner:

[0015] Obtain first training data; the first training data includes historical handwriting, and the first reference handwriting area and reference element type corresponding to the historical handwriting;

[0016] Obtain second training data synthesized based on the first training data; the second training data includes historical handwriting combinations, and a second reference handwriting region and reference element type corresponding to the historical handwriting combinations, wherein the historical handwriting combinations are composed of the historical handwriting.

[0017] In one embodiment, obtaining the second training data synthesized based on the first training data includes:

[0018] Multiple types of writing layouts are randomly generated; each writing layout contains at least one text paragraph, each text paragraph contains at least one text line, and each text line contains at least one text area; the text paragraphs, text lines, and text areas are arranged differently in different types of writing layouts;

[0019] According to a preset text filling strategy, the historical handwriting in the first training data is filled into the text area to obtain the second training data; the second training data includes the historical handwriting, and the second reference handwriting area and reference element type corresponding to the historical handwriting; the second reference handwriting area corresponding to each historical handwriting is the text area filled by the historical handwriting.

[0020] In one embodiment, training the target detection model to be trained using the first training data and the second training data to obtain a trained target detection model includes:

[0021] The target detection model to be trained is trained using the second training data to obtain the first detection model;

[0022] The first detection model is trained using the first training data to obtain the second detection model;

[0023] After fixing the backbone network parameters of the second detection model, the second detection model is trained using the first training data to obtain the third detection model;

[0024] The model parameters of the second detection model and the model parameters of the third detection model are fused to obtain a trained target detection model.

[0025] In one embodiment, the step of performing text recognition on the handwriting to obtain text content includes:

[0026] From multiple candidate text recognition models, a target text recognition model corresponding to the element type is selected; the multiple candidate text recognition models correspond to multiple candidate element types respectively; the element type is one of the multiple candidate element types.

[0027] The handwriting is input into the target text recognition model to perform text recognition on the handwriting and obtain the text content.

[0028] In one embodiment, generating a subject-specific tool corresponding to the handwriting based on the element type and the text content in response to a trigger operation on the interactive area includes:

[0029] In response to a trigger operation on the interactive area, an element corresponding to the element type is generated, and the element content of the element is generated according to the text content to obtain the subject tool corresponding to the handwriting.

[0030] In one embodiment, the step of generating an element corresponding to the element type in response to a trigger operation on the interactive area, and generating element content of the element based on the text content to obtain the subject-specific tool corresponding to the handwriting includes:

[0031] When the element type corresponds to at least two candidate elements, in response to a trigger operation on the interactive area, the element identifiers of the at least two candidate elements are displayed on the first user interface;

[0032] In response to an element selection operation, the target element identifier indicated by the element selection operation is determined from the element identifiers of the at least two candidate elements;

[0033] Generate the element corresponding to the target element identifier, and generate the element content of the element according to the text content to obtain the subject tool corresponding to the handwriting.

[0034] In one embodiment, the triggering operation includes a long press operation; the step of generating an element corresponding to the element type in response to the triggering operation on the interactive area, and generating element content of the element based on the text content to obtain the subject tool corresponding to the handwriting, includes:

[0035] In response to a long press operation on the interactive area, the interactive area is highlighted on the first user interface;

[0036] In response to the end of the long press operation, an element corresponding to the element type is generated, and the element content of the element is generated according to the text content to obtain the subject tool corresponding to the handwriting.

[0037] Displaying the subject-specific tools on the first user interface includes:

[0038] The text content corresponding to the handwriting and the subject-specific tools are displayed on the first user interface.

[0039] In one embodiment, the method further includes:

[0040] If an error is found in the element type based on the text content, the element type is corrected according to the text content to obtain the corrected element type.

[0041] The step of generating a subject-specific tool corresponding to the handwriting based on the element type and the text content in response to a trigger operation on the interactive area includes:

[0042] In response to a trigger operation on the interactive area, a subject-specific tool corresponding to the handwriting is generated based on the corrected element type and the text content.

[0043] Secondly, this application provides a subject-specific tool generation device, the device comprising:

[0044] The display module is used to display the first user interface.

[0045] A receiving module is used to receive writing operations on the first user interface;

[0046] The display module is also used to respond to the user's writing operation and display the writing handwriting corresponding to the writing operation on the first user interface;

[0047] The determination module is used to confirm that the writing operation has ended, determine the handwriting area and element type corresponding to the handwriting, and perform text recognition on the handwriting to obtain the text content;

[0048] The generation module is used to generate an interactive area based on the handwriting area, and in response to a trigger operation on the interactive area, generate a subject-specific tool corresponding to the handwriting according to the element type and the text content.

[0049] The display module is also used to display the subject-specific tools on the first user interface.

[0050] Thirdly, this application provides an interactive device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the various method embodiments of this application.

[0051] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in the various method embodiments of this application.

[0052] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the steps in the various method embodiments of this application.

[0053] The aforementioned method, apparatus, device, storage medium, and program product for generating subject-specific tools involve: displaying a first user interface; receiving a writing operation on the first user interface; responding to the user's writing operation by displaying the corresponding handwriting on the first user interface; confirming the completion of the writing operation, determining the handwriting area and element type corresponding to the handwriting, and performing text recognition on the handwriting to obtain text content; generating an interactive area based on the handwriting area, and responding to a trigger operation on the interactive area by generating a subject-specific tool corresponding to the handwriting based on the element type and text content; and displaying the subject-specific tool on the first user interface. Compared to the traditional method of generating subject-specific tools by inputting content via keyboard in an application, this application generates an interactive area based on the handwriting area by recognizing the corresponding handwriting area and element type. Users can directly trigger the interactive area to quickly generate subject-specific tools based on the element type and text content. This method of generating subject-specific tools is convenient to operate, eliminating the need for users to input content via keyboard in an application, thus improving the efficiency of subject-specific tool generation. Attached Figure Description

[0054] Figure 1 This is a flowchart illustrating a method for generating subject-specific tools in one embodiment;

[0055] Figure 2 This is a schematic diagram illustrating the process of obtaining text content through handwriting recognition in one embodiment.

[0056] Figure 3 This is a schematic diagram illustrating how to highlight an interactive area by long-pressing it in one embodiment.

[0057] Figure 4 This is a schematic diagram illustrating the generation of subject-specific tools based on element type and text content in one embodiment;

[0058] Figure 5 This is a schematic diagram showing the element identifiers of the two candidate elements in one embodiment;

[0059] Figure 6 This is a structural block diagram of a subject tool generation device in one embodiment;

[0060] Figure 7 This is a structural block diagram of the subject tool generation device in another embodiment;

[0061] Figure 8 This is a diagram of the internal structure of an interactive device in one embodiment. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0063] Traditional methods for generating subject-specific tools typically involve opening the corresponding tool within an application that supports this feature, and then inputting the desired content via the keyboard. For example, if a teacher wants to display a word card (i.e., a subject-specific tool) for the word "cat" in class, the traditional method requires the teacher to click on the corresponding control in the menu to access the subject-specific tool, and then input "cat" via the keyboard to generate the word card. However, this method is cumbersome, especially inconvenient for keyboard input on a large teaching screen. Furthermore, the entry point for this method is often hidden, and not all teachers are familiar with the software and can quickly locate the control to generate the tool, resulting in low efficiency in subject-specific tool generation.

[0064] In this application, a first user interface is displayed; a writing operation is received on the first user interface; in response to the user's writing operation, the writing handwriting corresponding to the writing operation is displayed on the first user interface; after confirming the completion of the writing operation, the handwriting area and element type corresponding to the handwriting are determined, and text recognition is performed on the handwriting to obtain text content; an interactive area is generated based on the handwriting area, and in response to a trigger operation on the interactive area, a subject-specific tool corresponding to the handwriting is generated according to the element type and text content; the subject-specific tool is displayed on the first user interface. Compared to the traditional method of generating subject-specific tools by inputting content through the keyboard in an application, this application generates an interactive area based on the handwriting area by recognizing the handwriting area and element type corresponding to the handwriting. Users can directly trigger the interactive area to quickly generate subject-specific tools according to the element type and text content. This method of generating subject-specific tools is convenient to operate, eliminating the need for users to input content through the keyboard in an application to generate subject-specific tools, thus improving the efficiency of subject-specific tool generation. For example, if a teacher wants to display a word card (i.e., a subject tool) for the word "cat" in class, the teacher can handwrite the word "cat" on the interactive device used for teaching. By directly triggering the interactive area where the word "cat" is located, the word card for "cat" can be generated and displayed directly on the interactive device. This eliminates the need for the teacher to painstakingly find the control for the subject tool and perform the corresponding generation operation, and also eliminates the need to input "cat" through the keyboard to generate the subject tool, thus improving the generation efficiency of the subject tool.

[0065] In one embodiment, such as Figure 1 As shown, a method for generating subject-specific tools is provided, which can be applied to interactive devices, and includes the following steps:

[0066] Step 102: Display the first user interface.

[0067] Specifically, the interactive device may display a first user interface, which is used to support user interaction with the interactive device.

[0068] Step 104: Receive writing operations on the first user interface.

[0069] Step 106: In response to the user's writing operation, display the writing handwriting corresponding to the writing operation on the first user interface.

[0070] In one embodiment, the interactive device includes a display screen, which may have interactive functionality, allowing the user to write on the display screen. The interactive device can respond to the user's writing operation, acquire the handwriting generated based on the writing operation, and display the handwriting corresponding to the writing operation on a first user interface.

[0071] In one embodiment, the display screen is a touch-sensitive display screen used to detect the position information of the touched object. The touch-sensitive display screen is a touch-enabled display screen, such as an infrared display screen, capacitive display screen, resistive display screen, electromagnetic display screen, or pressure-sensitive display screen. On the touch-sensitive display screen, users can perform writing or other operations by touching the display screen with their fingers, pens, or other objects. Correspondingly, the interactive device detects the position information when the finger or pen performs the touch operation and responds accordingly to achieve touch-based human-computer interaction.

[0072] In one embodiment, a toolbar may be displayed on the screen of the interactive device, and the toolbar may include a writing tool. The interactive device may activate the writing tool in response to a trigger action on the writing tool, allowing the user to write on the screen of the interactive device. The interactive device may also acquire the handwriting generated based on the user's writing action using the writing tool.

[0073] In one embodiment, handwriting may contain at least one stroke, and each stroke may contain multiple handwriting coordinate points.

[0074] Step 108: After confirming that the writing operation is completed, determine the writing area and element type corresponding to the writing, and perform text recognition on the writing to obtain the text content.

[0075] The handwriting area is the region containing the written handwriting. This area can be the smallest bounding rectangle containing the handwriting, the smallest bounding rectangle plus margins, or any other preset shape containing the smallest area of ​​the handwriting. The element type is the type of teaching element associated with the handwriting. Element types can include at least one of the following: Chinese characters, Pinyin, English words, four-line grid, and formulas. For example, if the handwriting is "y=x+1", then the type of teaching element that can be associated with it is a formula; that is, the element type corresponding to this handwriting is a formula. By defining the types of teaching elements associated with the handwriting, the potential meanings of different handwritings can be obtained, thus quickly determining the types of teaching elements associated with them.

[0076] In one embodiment, the interactive device can determine the handwriting region to which the handwriting belongs based on the handwriting itself. This means that the handwriting region contains the handwriting. The interactive device can also determine the handwriting region and element type corresponding to the handwriting. For example, if the handwriting is the result of a user writing "cat," then the handwriting region corresponding to the handwriting of "cat" needs to completely cover "cat," and the element type corresponding to the handwriting of "cat" is an English word.

[0077] In one embodiment, the interactive device can input handwriting into a unified text recognition model to perform text recognition on the handwriting and obtain the text content. It can be understood that if the element type includes Chinese characters, pinyin, English words, and formulas, then the text recognition model can simultaneously support the recognition of Chinese characters, pinyin, English words, and formulas.

[0078] In one embodiment, such as Figure 2 As shown, the interactive device's display screen can show a toolbar, which may contain a writing tool, namely a "pen". The interactive device can respond to a trigger operation on the "pen" writing tool, enabling the user to write on the device's display screen. The interactive device can respond to the user's writing operation based on the "pen" writing tool, acquire the handwriting generated based on the writing operation, and perform text recognition on the handwriting to obtain the text content, namely "y=x+1".

[0079] Step 110: Generate an interactive area based on the handwriting area, and in response to the triggered operation on the interactive area, generate subject-specific tools corresponding to the handwriting according to the element type and text content.

[0080] Interactive areas are areas with user interaction capabilities. Subject-specific tools are tools used to assist in subject teaching within a teaching scenario.

[0081] In one embodiment, when the element type corresponds to one element, the interactive device can respond to a trigger operation on the interactive area and directly generate a subject-specific tool corresponding to the handwriting based on the element type and text content.

[0082] In one embodiment, the subject-specific tools may include at least one of the following: functions, geometry, Chinese character flashcards, word flashcards, pinyin, and four-line grid.

[0083] In one embodiment, such as Figure 3 As shown, the triggering operation is a long press. The interactive device can respond to a long press on the interactive area 301 by highlighting the interactive area 301. Furthermore, as... Figure 4 As shown, the interactive device can respond to the end of the long press operation and generate the subject tool 401 corresponding to the handwriting based on the element type and text content.

[0084] Step 112: Display the subject tools on the first user interface.

[0085] In the aforementioned method for generating subject-specific tools, a first user interface is displayed; a writing operation is received on the first user interface; in response to the user's writing operation, the corresponding handwriting is displayed on the first user interface; after confirming the completion of the writing operation, the handwriting area and element type are determined, and text recognition is performed on the handwriting to obtain text content; an interactive area is generated based on the handwriting area, and in response to a trigger operation on the interactive area, a subject-specific tool corresponding to the handwriting is generated according to the element type and text content; the subject-specific tool is displayed on the first user interface. Compared to the traditional method of generating subject-specific tools by inputting content via keyboard in an application, this application generates an interactive area based on the handwriting area by recognizing the handwriting area and element type. Users can directly trigger the interactive area to quickly generate subject-specific tools based on the element type and text content. This method of generating subject-specific tools is convenient to operate, eliminating the need for users to input content via keyboard in an application, thus improving the efficiency of subject-specific tool generation.

[0086] In one embodiment, determining the handwriting region and element type corresponding to the handwriting includes: rendering the handwriting to obtain a handwriting image; inputting the handwriting image into a trained object detection model to perform object detection on the handwriting image through the object detection model to obtain the handwriting region and element type corresponding to the handwriting.

[0087] Among them, the object detection model is a neural network model with object detection capabilities.

[0088] Specifically, the interactive device can render handwriting to obtain a handwriting image. This handwriting image can contain the corresponding text content. The interactive device can then input the handwriting image into a trained object detection model to perform object detection on the handwriting image, obtaining the corresponding handwriting region and element type. Again, the handwriting region contains the corresponding text content.

[0089] In the above embodiments, by using a pre-trained target detection model to perform target detection on the handwriting image, the handwriting region and element type corresponding to the handwriting can be obtained, which can improve the detection accuracy of the handwriting region and the recognition accuracy of the element type.

[0090] In one embodiment, the target detection model is trained by: acquiring first training data; the first training data includes historical handwriting, and a first reference handwriting region and reference element type corresponding to the historical handwriting; acquiring second training data synthesized based on the first training data; the second training data includes combinations of historical handwriting, and combinations of historical handwriting composed of historical handwriting; a corresponding second reference handwriting region and reference element type; and training the target detection model to be trained using the first training data and the second training data to obtain a trained target detection model.

[0091] In the first training data, the first reference handwriting region and reference element type corresponding to the historical handwriting were obtained through manual annotation. The second training data was synthesized based on the first training data. It is understood that training the object detection model requires a large amount of training data, but obtaining a large amount of training data through manual annotation is costly. Therefore, this application first obtains a small amount of first training data through manual annotation, and then automatically synthesizes a large amount of second training data based on the first training data for use in training the object detection model. This reduces the cost of obtaining training data while ensuring the model training effect. It is understood that the amount of data in the second training data is greater than that in the first training data.

[0092] In one embodiment, the reference element type may include at least one of Chinese characters, pinyin, English words, and formulas.

[0093] In the above embodiments, a small amount of first training data is first obtained through manual annotation, and then a large amount of second training data is automatically synthesized based on the first training data to be used to train the target detection model to be trained. This can reduce the cost of obtaining training data while ensuring the training effect of the model.

[0094] In one embodiment, obtaining second training data synthesized based on first training data includes: randomly generating multiple types of writing layouts; each writing layout contains at least one text paragraph, each text paragraph contains at least one text line, and each text line contains at least one text region; the text paragraphs, text lines, and text regions in different types of writing layouts have different layouts; according to a preset text filling strategy, historical handwriting from the first training data is filled into the text regions to obtain the second training data; the second training data includes historical handwriting, and second reference handwriting regions and reference element types corresponding to the historical handwriting; the second reference handwriting region corresponding to each historical handwriting is the text region filled by the historical handwriting.

[0095] In one embodiment, the element type may include at least one of Chinese characters, Pinyin, English words, and formulas. The preset text filling strategy may be a combination of the following: a text line may contain a certain probability of containing all Chinese characters, a certain probability of containing all Pinyin, a certain probability of containing all English words, a certain probability of containing all formulas, or a certain probability of containing at least one of Chinese characters, Pinyin, English words, and formulas.

[0096] In the above embodiments, by randomly generating various types of writing layouts and automatically generating a large amount of second training data based on the second training data, the quality of the second training data generation can be guaranteed, thereby improving the training effect of the object detection model.

[0097] In one embodiment, training a target detection model to be trained using first training data and second training data to obtain a trained target detection model includes: training the target detection model to be trained using the second training data to obtain a first detection model; training the first detection model using the first training data to obtain a second detection model; after fixing the backbone network parameters of the second detection model, training the second detection model using the first training data to obtain a third detection model; and fusing the model parameters of the second detection model and the model parameters of the third detection model to obtain the trained target detection model.

[0098] It is understandable that the second detection model includes backbone network parameters and non-backbone network parameters. After fixing the backbone network parameters of the second detection model, training the second detection model with the first training data to obtain the third detection model means fine-tuning the second detection model using the first training data.

[0099] In the above embodiments, a first detection model is first trained using a large amount of automatically synthesized second training data to generalize the target detection model. Then, a second detection model is further optimized using a small amount of labeled first training data from real-world scenarios. After fixing the backbone network parameters of the second detection model, it is fine-tuned using a small amount of labeled first training data from real-world scenarios to obtain a third detection model. Finally, the model parameters of the second and third detection models are fused to obtain the trained target detection model. This improves the detection performance of the final target detection model.

[0100] In one embodiment, performing text recognition on handwriting to obtain text content includes: selecting a target text recognition model corresponding to an element type from multiple candidate text recognition models; each of the multiple candidate text recognition models corresponds to a multiple candidate element type; the element type is one of the multiple candidate element types; and inputting the handwriting into the target text recognition model to perform text recognition on the handwriting through the target text recognition model to obtain the text content.

[0101] In one embodiment, if the candidate element types include Chinese characters, pinyin, English words, and formulas, then the multiple candidate text recognition models can include a Chinese character recognition model that supports Chinese character recognition, a pinyin recognition model that supports pinyin recognition, an English word recognition model that supports English word recognition, and a formula recognition model that supports formula recognition. For example, if the element type corresponding to the handwriting area is an English word, the interactive device can select an English word recognition model as the target text recognition model and input the handwriting into the English word recognition model to perform text recognition on the handwriting and obtain the text content.

[0102] In the above embodiments, the target text recognition model corresponding to the element type has a better recognition effect on the handwriting in the handwriting area. By performing text recognition on the handwriting through the target text recognition model corresponding to the element type and obtaining the text content, the recognition accuracy of the text content can be further improved.

[0103] In one embodiment, in response to a trigger operation on an interactive area, a subject-specific tool corresponding to the handwriting is generated based on the element type and text content, including: in response to a trigger operation on an interactive area, generating an element corresponding to the element type, and generating element content of the element based on the text content, to obtain the subject-specific tool corresponding to the handwriting.

[0104] In the above embodiments, by triggering the interactive area, elements corresponding to the element type are generated, and the element content of the elements is generated again according to the text content, so as to obtain the subject tool corresponding to the handwriting, which can improve the accuracy of generating subject tools.

[0105] In one embodiment, in response to a trigger operation on an interactive area, generating an element corresponding to an element type and generating element content based on text content to obtain a subject-specific tool corresponding to the handwriting, includes: when the element type corresponds to at least two candidate elements, in response to a trigger operation on an interactive area, displaying element identifiers of at least two candidate elements on a first user interface; in response to an element selection operation, determining a target element identifier indicated by the element selection operation from the element identifiers of the at least two candidate elements; generating an element corresponding to the target element identifier and generating element content based on text content to obtain a subject-specific tool corresponding to the handwriting.

[0106] In one embodiment, when the element type corresponds to at least two candidate elements, the interactive device may, in response to a trigger operation on the interactive area, highlight the interactive area and display the element identifiers of the at least two candidate elements on the first user interface. The interactive device may, in response to an element selection operation, determine the target element identifier indicated by the element selection operation from the element identifiers of the at least two candidate elements, generate the element corresponding to the target element identifier, and generate the element content of the element based on the text content, thereby obtaining the subject-specific tool corresponding to the handwriting.

[0107] In one embodiment, the triggering operation includes a long press operation; in response to the triggering operation on the interactive area, generating an element corresponding to the element type, and generating element content of the element based on the text content to obtain the subject tool corresponding to the handwriting, includes: in response to the long press operation on the interactive area, highlighting the interactive area on the first user interface; in response to the end of the long press operation, generating an element corresponding to the element type, and generating element content of the element based on the text content to obtain the subject tool corresponding to the handwriting; displaying the subject tool on the first user interface includes: displaying the text content corresponding to the handwriting and the subject tool on the first user interface.

[0108] In one embodiment, the interactive device can display text content corresponding to the handwriting and subject-specific tools on a first user interface according to a preset position or layout. It is understood that simultaneously displaying text content corresponding to the handwriting and subject-specific tools on the first user interface facilitates a better understanding of the corresponding text content by combining the subject-specific tools, thereby improving teaching effectiveness.

[0109] For example, such as Figure 5As shown, the English word "hello" corresponds to two candidate elements: a word card and a four-line grid. The interactive device can respond to a long press on the interactive area 501, highlighting the interactive area and displaying the element identifiers for both the word card and the four-line grid, namely "word card" and "four-line grid." Upon the end of the long press and in response to an element selection operation, the interactive device can determine the target element identifier from at least two candidate element identifiers, generate the element indicated by the target element identifier, and generate the element content based on the text content, thus obtaining the subject-specific tool corresponding to the handwriting. For example, if the user selects "word card," the interactive device can generate a word card containing "hello," thus creating a subject-specific tool.

[0110] In the above embodiments, when there are at least two candidate elements corresponding to the element type, the element identifiers of each candidate element corresponding to the element type are displayed to the user for selection, so that the generated subject tool meets the user's expectations and can improve the generation accuracy of the subject tool.

[0111] In one embodiment, the method further includes: when there is an error in determining the element type based on the text content, correcting the element type according to the text content to obtain the corrected element type; wherein, in response to a trigger operation on the interactive area, generating a subject-specific tool corresponding to the handwriting based on the element type and the text content includes: in response to a trigger operation on the interactive area, generating a subject-specific tool corresponding to the handwriting based on the corrected element type and the text content.

[0112] Specifically, the interactive device can respond to a user's writing operation, acquire the handwriting generated based on the writing operation, and determine the corresponding handwriting area and element type. The interactive device can perform text recognition on the handwriting to obtain text content. The interactive device can determine whether the element type is correct based on the text content; if the element type is determined incorrectly based on the text content, it can correct the element type according to the text content to obtain the corrected element type. The interactive device can generate an interactive area based on the handwriting area, and in response to a trigger operation on the interactive area, generate subject-specific tools corresponding to the handwriting based on the corrected element type and text content.

[0113] For example, if the originally identified handwriting area corresponds to the element type of English words, but the text content obtained from handwriting recognition contains pinyin characters, the interactive device can determine that the originally identified element type (i.e., English words) is incorrect. Furthermore, the interactive device can correct the element type based on the recognized text content to obtain the corrected element type (i.e., pinyin).

[0114] In the above embodiments, if the element type is determined to be incorrect based on the text content, the element type can be corrected according to the text content, which can further improve the accuracy of subsequent generation of subject-specific tools.

[0115] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially, these steps are not necessarily executed in that order. Unless otherwise expressly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the above embodiments may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0116] In one embodiment, such as Figure 6 As shown, a subject-specific tool generation device 600 is provided, which specifically includes:

[0117] Display module 602 is used to display the first user interface;

[0118] The receiving module 604 is used to receive writing operations on the first user interface;

[0119] The display module 602 is also used to respond to the user's writing operation and display the writing handwriting corresponding to the writing operation on the first user interface;

[0120] The determination module 606 is used to confirm that the writing operation has ended, determine the writing area and element type corresponding to the writing, and perform text recognition on the writing to obtain the text content;

[0121] The generation module 608 is used to generate an interactive area based on the handwriting area, and in response to the trigger operation on the interactive area, generate subject tools corresponding to the handwriting according to the element type and text content.

[0122] Display module 602 is also used to display subject tools on the first user interface.

[0123] In one embodiment, the determining module 606 is further configured to render the handwriting to obtain a handwriting image; input the handwriting image into a trained target detection model to perform target detection on the handwriting image through the target detection model, and obtain the handwriting region and element type corresponding to the handwriting.

[0124] In one embodiment, such as Figure 7 As shown, the subject tool generation device 600 also includes:

[0125] The training module 610 is used to acquire first training data; the first training data includes historical handwriting, and the first reference handwriting region and reference element type corresponding to the historical handwriting; acquire second training data synthesized based on the first training data; the second training data includes historical handwriting, and the second reference handwriting region and reference element type corresponding to the historical handwriting; train the target detection model to be trained using the first training data and the second training data to obtain the trained target detection model.

[0126] In one embodiment, the training module 610 is further configured to randomly generate multiple types of writing layouts; each writing layout contains at least one text paragraph, each text paragraph contains at least one text line, and each text line contains at least one text region; the text paragraphs, text lines, and text regions in different types of writing layouts have different layouts; according to a preset text filling strategy, historical handwriting from the first training data is filled into the text region to obtain second training data; the second training data includes historical handwriting, and the second reference handwriting region and reference element type corresponding to the historical handwriting; the second reference handwriting region corresponding to each historical handwriting is the text region filled by the historical handwriting.

[0127] In one embodiment, the training module 610 is further configured to train the target detection model to be trained using the second training data to obtain a first detection model; train the first detection model using the first training data to obtain a second detection model; after fixing the backbone network parameters of the second detection model, train the second detection model using the first training data to obtain a third detection model; and fuse the model parameters of the second detection model and the model parameters of the third detection model to obtain a trained target detection model.

[0128] In one embodiment, the determining module 606 is further configured to select a target text recognition model corresponding to an element type from multiple candidate text recognition models; the multiple candidate text recognition models correspond to multiple candidate element types respectively; the element type is one of the multiple candidate element types; and input the handwriting into the target text recognition model to perform text recognition on the handwriting through the target text recognition model to obtain the text content.

[0129] In one embodiment, the generation module 608 is further configured to generate an element corresponding to the element type in response to a trigger operation on the interactive area, and generate the element content of the element according to the text content to obtain the subject tool corresponding to the handwriting.

[0130] In one embodiment, the generation module 608 is further configured to, in response to a trigger operation targeting an interactive area, display the element identifiers of the at least two candidate elements on a first user interface when the element type corresponds to at least two candidate elements; in response to an element selection operation, determine the target element identifier indicated by the element selection operation from the element identifiers of the at least two candidate elements; generate the element corresponding to the target element identifier, and generate the element content of the element according to the text content, thereby obtaining the subject tool corresponding to the handwriting.

[0131] In one embodiment, the triggering operation includes a long press operation; the generation module 608 is further configured to, in response to a long press operation on the interactive area, highlight the interactive area on the first user interface; in response to the end of the long press operation, generate an element corresponding to the element type, and generate the element content of the element according to the text content, thereby obtaining the subject tool corresponding to the handwriting; the display module 602 is further configured to, on the first user interface, display the text content and subject tool corresponding to the handwriting.

[0132] In one embodiment, the determining module 606 is further configured to correct the element type based on the text content if the element type determination is incorrect based on the text content, and obtain the corrected element type; the generating module 608 is further configured to generate a subject tool corresponding to the handwriting based on the corrected element type and the text content in response to a trigger operation on the interactive area.

[0133] The aforementioned subject-specific tool generation device displays a first user interface; receives writing operations on the first user interface; responds to the user's writing operations by displaying the corresponding handwriting on the first user interface; after confirming the completion of the writing operation, determines the handwriting area and element type corresponding to the handwriting, performs text recognition on the handwriting to obtain text content; generates an interactive area based on the handwriting area, and responds to a trigger operation on the interactive area, generates a subject-specific tool corresponding to the handwriting based on the element type and text content; and displays the subject-specific tool on the first user interface. Compared to the traditional method of generating subject-specific tools by inputting content via keyboard in an application, this application generates an interactive area based on the handwriting area by recognizing the corresponding handwriting area and element type. Users can directly trigger the interactive area to quickly generate subject-specific tools based on the element type and text content. This method of generating subject-specific tools is convenient to operate, eliminating the need for users to input content via keyboard in an application, thus improving the efficiency of subject-specific tool generation.

[0134] Each module in the aforementioned subject tool generation device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of the interactive device in hardware form or independent of it, or stored in the memory of the interactive device in software form, so that the processor can call and execute the operations corresponding to each module.

[0135] In one embodiment, an interactive device is provided, the internal structure of which can be shown as follows: Figure 7 As shown, the interactive device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a method for generating subject-specific tools. The display unit of the interactive device is used to form a visually visible image. It can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the interactive device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the interactive device, or external keyboards, touchpads, or mice, etc.

[0136] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the interactive device to which the present application is applied. Specific interactive devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0137] In one embodiment, an interactive device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0138] In one embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0139] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0140] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0141] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0142] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0143] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for generating subject-specific tools, characterized in that, The method includes: Display the first user interface; Receive writing operations on the first user interface; In response to the user's writing operation, the handwriting corresponding to the writing operation is displayed on the first user interface; After confirming that the writing operation is completed, determine the handwriting area and element type corresponding to the handwriting, and perform text recognition on the handwriting to obtain the text content; An interactive area is generated based on the handwriting area, and in response to a trigger operation on the interactive area, a subject-specific tool corresponding to the handwriting is generated according to the element type and the text content. The subject-specific tools are displayed on the first user interface.

2. The method according to claim 1, characterized in that, Determining the handwriting region and element type corresponding to the handwriting includes: The handwriting is rendered to obtain a handwriting image; The handwriting image is input into a trained target detection model to perform target detection on the handwriting image, thereby obtaining the handwriting region and element type corresponding to the handwriting.

3. The method according to claim 2, characterized in that, The target detection model is trained in the following way: Obtain first training data; the first training data includes historical handwriting, and the first reference handwriting area and reference element type corresponding to the historical handwriting; Obtain second training data synthesized based on the first training data; the second training data includes historical handwriting combinations, and a second reference handwriting region and reference element type corresponding to the historical handwriting combinations, wherein the historical handwriting combinations are composed of the historical handwritings; The target detection model to be trained is trained using the first training data and the second training data to obtain a trained target detection model.

4. The method according to claim 3, characterized in that, The step of obtaining the second training data synthesized based on the first training data includes: Multiple types of writing layouts are randomly generated; each writing layout contains at least one text paragraph, each text paragraph contains at least one text line, and each text line contains at least one text area; the text paragraphs, text lines, and text areas are arranged differently in different types of writing layouts. According to a preset text filling strategy, the historical handwriting in the first training data is filled into the text area to obtain the second training data; the second training data includes the historical handwriting, and the second reference handwriting area and reference element type corresponding to the historical handwriting; the second reference handwriting area corresponding to each historical handwriting is the text area filled by the historical handwriting.

5. The method according to claim 3, characterized in that, The step of training the target detection model to be trained using the first training data and the second training data to obtain a trained target detection model includes: The target detection model to be trained is trained using the second training data to obtain the first detection model; The first detection model is trained using the first training data to obtain the second detection model; After fixing the backbone network parameters of the second detection model, the second detection model is trained using the first training data to obtain the third detection model; The model parameters of the second detection model and the model parameters of the third detection model are fused to obtain a trained target detection model.

6. The method according to claim 1, characterized in that, The step of performing text recognition on the handwriting to obtain text content includes: From multiple candidate text recognition models, a target text recognition model corresponding to the element type is selected; the multiple candidate text recognition models correspond to multiple candidate element types respectively; the element type is one of the multiple candidate element types. The handwriting is input into the target text recognition model to perform text recognition on the handwriting and obtain the text content.

7. The method according to claim 1, characterized in that, The method of generating a subject-specific tool corresponding to the handwriting based on the element type and the text content in response to a trigger operation on the interactive area includes: In response to a trigger operation on the interactive area, an element corresponding to the element type is generated, and the element content of the element is generated according to the text content to obtain the subject tool corresponding to the handwriting.

8. The method according to claim 7, characterized in that, The method of generating an element corresponding to the element type in response to a trigger operation on the interactive area, and generating element content of the element based on the text content to obtain the subject-specific tool corresponding to the handwriting, includes: When the element type corresponds to at least two candidate elements, in response to a trigger operation on the interactive area, the element identifiers of the at least two candidate elements are displayed on the first user interface; In response to an element selection operation, the target element identifier indicated by the element selection operation is determined from the element identifiers of the at least two candidate elements; Generate the element corresponding to the target element identifier, and generate the element content of the element according to the text content to obtain the subject tool corresponding to the handwriting.

9. The method according to claim 7, characterized in that, The triggering operation includes a long press operation; the step of generating an element corresponding to the element type in response to the triggering operation on the interactive area, and generating the element content of the element according to the text content to obtain the subject tool corresponding to the handwriting, includes: In response to a long press operation on the interactive area, the interactive area is highlighted on the first user interface; In response to the end of the long press operation, an element corresponding to the element type is generated, and the element content of the element is generated according to the text content to obtain the subject tool corresponding to the handwriting. Displaying the subject-specific tools on the first user interface includes: The text content corresponding to the handwriting and the subject-specific tools are displayed on the first user interface.

10. The method according to any one of claims 1 to 9, characterized in that, The method further includes: If an error is found in the element type based on the text content, the element type is corrected according to the text content to obtain the corrected element type. The step of generating a subject-specific tool corresponding to the handwriting based on the element type and the text content in response to a trigger operation on the interactive area includes: In response to a trigger operation on the interactive area, a subject-specific tool corresponding to the handwriting is generated based on the corrected element type and the text content.

11. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 10.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 10.

Images