Note illustrating method and device, equipment and storage medium

Abstract

The invention discloses a note illustration method and device, equipment and a storage medium, and belongs to the technical field of software auxiliary application. The method comprises the steps of displaying a note interface, and determining at least one handwriting of a to-be-illustrated picture based on the note interface; acquiring an illustration image matched with the at least one handwriting; and displaying the illustration image on the note interface. According to the method, the illustration image can be automatically obtained and displayed through the at least one handwriting of the to-be-illustrated image, the note illustration operation process is simplified, and the note illustration efficiency is improved. Moreover, the automatically acquired illustration image is matched with at least one handwriting, so that the accuracy of the illustration image is relatively high.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of software-assisted applications, and particularly to a method, device, equipment and storage medium for note illustration. Background Art

[0002] With the development of software-assisted application technology, note-taking software can insert pictures to illustrate notes, thereby improving the quality of notes. In the related art, it is necessary to manually search for pictures according to the note content and then manually insert the found pictures into the notes. However, in the related art, the process of manually searching for pictures and inserting the found pictures into the notes requires switching between different software and the note-taking software, making the method of note illustration cumbersome and inefficient. Summary of the Invention

[0003] The embodiments of the present application provide a method, device, equipment and storage medium for note illustration, which can be used to solve the problems existing in the related art. The technical solutions are as follows:

[0004] In a first aspect, a method for note illustration is provided. The method includes:

[0005] Display a note interface, and determine at least one handwriting to be illustrated based on the note interface;

[0006] Obtain an illustration image that matches the at least one handwriting;

[0007] Display the illustration image on the note interface.

[0008] In another aspect, a device for note illustration is provided. The device includes:

[0009] A display module for displaying a note interface;

[0010] A determination module for determining at least one handwriting to be illustrated based on the note interface;

[0011] An obtaining module for obtaining an illustration image that matches the at least one handwriting;

[0012] The display module is further configured to display the illustration image on the note interface.

[0013] In another aspect, a computer device is further provided. The computer device includes a processor and a memory. At least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor so that the computer device implements the method for note illustration described in any of the above aspects.

[0014] On the other hand, a computer-readable storage medium is also provided. At least one computer program is stored in the computer-readable storage medium. The at least one computer program is loaded and executed by a processor to enable a computer to implement the note illustration method described in any of the above aspects.

[0015] On the other hand, a computer program product or a computer program is also provided. The computer program product or the computer program includes computer instructions. The computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the note illustration method described in any of the above aspects.

[0016] The technical solutions provided in the embodiments of the present application at least bring the following beneficial effects:

[0017] The technical solution provided in the present application automatically obtains and displays an illustration image through at least one handwriting to be illustrated, simplifies the operation process of note illustration, and improves the efficiency of note illustration. Moreover, the automatically obtained illustration image is matched with at least one handwriting, so that the accuracy of the illustration image is relatively high. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following described drawings are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained according to these drawings.

[0019] Figure 1 is a schematic diagram of an implementation environment of a note illustration method provided in an embodiment of the present application;

[0020] Figure 2 is a flowchart of a note illustration method provided in an embodiment of the present application;

[0021] Figure 3 is a schematic diagram of a note interface provided in an embodiment of the present application;

[0022] Figure 4 is another schematic diagram of a note interface provided in an embodiment of the present application;

[0023] Figure 5 is another schematic diagram of a note interface provided in an embodiment of the present application;

[0024] Figure 6 is a schematic diagram of a function image provided in an embodiment of the present application;

[0025] Figure 7It is a schematic flowchart of a note illustration provided by an embodiment of the present application;

[0026] Figure 8 It is another schematic flowchart of a note illustration provided by an embodiment of the present application;

[0027] Figure 9 It is another schematic diagram of a note interface provided by an embodiment of the present application;

[0028] Figure 10 It is another schematic diagram of a note interface provided by an embodiment of the present application;

[0029] Figure 11 It is another schematic diagram of a note interface provided by an embodiment of the present application;

[0030] Figure 12 It is another schematic diagram of a note interface provided by an embodiment of the present application;

[0031] Figure 13 It is a schematic diagram of a template interface provided by an embodiment of the present application;

[0032] Figure 14 It is a schematic structural diagram of a note illustration device provided by an embodiment of the present application;

[0033] Figure 15 It is a schematic structural diagram of a terminal provided by an embodiment of the present application. Detailed implementation manners

[0034] To make the objectives, technical solutions, and advantages of the present application clearer, the following will further describe the embodiments of the present application in detail with reference to the accompanying drawings.

[0035] It should be noted that the terms "first", "second", etc. (if any) in the specification of the present application are used to distinguish similar objects and do not necessarily describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application.

[0036] With the development of digitization, note-taking software has become an indispensable tool in learning and work. However, the note content presented by recording text content is limited. Furthermore, with the development of software-assisted application technologies, users' requirements for note content are constantly increasing. For example, in addition to recording text content, note content also needs to record picture content.

[0037] In the related art, users need to search for pictures through external image search tools and then manually insert them into the notes that record text. However, during the process of searching for and inserting pictures, users need to frequently switch between different software, which increases the complexity of operations. In addition, due to compatibility issues between different software, the inserted picture format may not match or the position may not be accurate.

[0038] An embodiment of the present application provides a method for adding pictures to notes, which is used to simplify the process of adding pictures to notes and improve the accuracy of adding pictures to notes. Please refer to Figure 1 , Figure 1 FIG. shows a schematic diagram of the method implementation environment provided by the embodiment of the present application. The implementation environment may include: a computer device 11.

[0039] Among them, the embodiment of the present application does not limit the product form of the computer device 11. No matter what product form it is, a note software is installed on the computer device 11, and the note software can use the method for adding pictures to notes provided by the embodiment of the present application to add pictures to notes.

[0040] For example, the computer device 11 may be a terminal. Optionally, the terminal may be any electronic product that can perform human-computer interaction with the user in one or more ways such as a keyboard, a touchpad, a touch screen, a remote control, voice interaction, or a handwriting device. For example, a PC (Personal Computer), a mobile phone, a smart phone, a PDA (Personal Digital Assistant), a wearable device, a PPC (Pocket PC), a tablet computer, a smart vehicle machine, a smart TV, etc.

[0041] Those skilled in the art should understand that the above terminals are only examples, and other existing or future terminals that are applicable to the present application should also be included within the protection scope of the present application and are hereby incorporated herein by reference.

[0042] Taking the computer device executing this method as an example, refer to Figure 2 , Figure 2 is a flowchart of a method for adding pictures to notes provided by an embodiment of the present application. This method can be applied to Figure 1 the implementation environment shown in Figure 1 For example, the computer device 11 shown in Figure 2 executes this method. As shown in

[0043] Step 201, display a note interface, and determine at least one handwriting to be paired with a picture based on the note interface.

[0044] In an embodiment of the present application, the computer device includes a display screen, and the computer device displays a note interface through the display screen. Optionally, a note software is running on the computer device, and the note interface is an interface for the note software to interact with the user. The user can record, organize, and view note content through the note interface. For example, the user can record note content in the note interface, and recording note content includes inputting text, drawing graphics, or adding pictures, etc.

[0045] When the user records note content in the handwriting mode, the note interface can display handwriting, which is the note content written by the user on the note interface. Among them, the types of multiple handwritings include but are not limited to text, graphics, lines, or shapes, etc. Refer to Figure 3 , Figure 3 FIG. is a schematic diagram of a note interface provided by an embodiment of the present application. The note interface includes multiple handwritings, and any handwriting can be text, graphics, lines, or shapes, etc. recorded by handwriting.

[0046] Exemplarily, when the user writes a paragraph of text in the note software, the paragraph of text appears on the note interface in the form of handwriting. Or, the user can draw a mind map, a flowchart, or an organizational chart, etc. on the note interface, so as to help the user better record note content. The note interface also provides a handwriting adjustment component, so as to facilitate the user to adjust the color, thickness, and style of the handwriting according to the needs, and facilitate the user to create visually appealing notes, thereby improving the readability and comprehension of the notes.

[0047] Optionally, the note interface includes multiple recorded handwritings; the method for determining at least one handwriting to be paired with a picture based on the note interface includes, in response to a selection operation on the note interface, displaying a selection area corresponding to the selection operation on the note interface, and determining at least one handwriting to be paired with a picture based on at least one handwriting included in the selection area.

[0048] For the handwriting that needs to generate a paired picture, the user can select the handwriting that needs to generate a paired picture through a selection operation on the note interface, and the note software, in response to the selection operation, displays a selection area corresponding to the selection operation on the note interface. The embodiment of the present application does not limit the manner of displaying the selection area on the note interface, and it can be displayed in forms such as highlighting, dotted lines, shadows, or colors. The selection area includes at least one handwriting, that is, the handwriting selected by the user that needs to generate a paired picture.

[0049] Among them, the selection operation includes but is not limited to a selection operation, a click operation, or a marquee selection operation, etc. In the case where the selection operation is a marquee selection operation, the selection area is the marquee area. Among them, the marquee area can be of any shape and size, for example, rectangular, elliptical, or triangular, etc., specifically depending on the dragging and selection of the user on the note interface.

[0050] Refer to Figure 4, Figure 4 is the note interface displayed after a selection operation based on the note interface shown. Taking the case where the selection area is displayed in a dotted line form as an example, Figure 3 it includes a selection area 401, and the selection area 401 includes a handwriting. Figure 4

[0051] In a possible implementation manner, the selection operation is also used for the user to organize and sort note content. For example, by selecting relevant handwriting or a group of handwriting together, the user can move, adjust the position or merge the selection area and the content in the selection area by dragging the selection area, etc., to make the note more orderly and easy to manage.

[0052] Optionally, the manner of determining at least one handwriting to be paired with a picture based on the note interface may further include: in response to a trigger operation for the picture-pairing function, displaying a picture-pairing interface; determining at least one handwriting to be paired with a picture based on at least one handwriting input in the picture-pairing interface. In this case, the handwriting to be paired with a picture is not displayed in the note interface, but in the picture-pairing interface, and the picture-pairing interface is displayed in response to a trigger operation for the picture-pairing function.

[0053] In the embodiments of the present application, the trigger operation for the picture-pairing function may be clicking the soft switch of the picture-pairing function button on the note interface, or pressing the combination key corresponding to the picture-pairing function on the keyboard connected to the computer device running the note software, etc. After the picture-pairing interface is displayed, the user can input handwriting in the picture-pairing interface, and at least one handwriting input in the picture-pairing interface is at least one handwriting to be paired with a picture.

[0054] Referring to Figure 5 the schematic diagram of the note interface shown, the note interface includes a picture-pairing interface 501 displayed in response to a trigger operation for the picture-pairing function, and the picture-pairing interface 501 is used to obtain the handwriting input by the user. Figure 5 The position range of the picture-pairing interface 501 in is only for illustrative purposes. The picture-pairing interface 501 can be located at any position of the note interface, and the picture-pairing interface 501 can also be the same size as the note interface.

[0055] Step 202, obtaining a picture image that matches at least one handwriting.

[0056] For the optional solution in step 201, if at least one handwriting is determined according to the selection area, then in response to a trigger operation for the selection area, obtain a picture image that matches at least one handwriting. If at least one handwriting is determined according to the picture-pairing interface, then in response to a trigger operation for the picture-pairing interface, obtain a picture image that matches at least one handwriting.

[0057] ​The embodiments of the present application do not limit the triggering operation, and the triggering operation can trigger the computer device to obtain the matching image. For example, the triggering operation can be a click operation of the user on the soft switch of the note interface; or, the voice command operation of the user; or, the hardware triggering operation of the computer device running the note software, etc. Among them, the matching image corresponds to at least one handwriting, so as to facilitate the user to better understand and remember the note content and improve the quality of the note.

[0058] In a possible implementation manner, the process of obtaining the matching image corresponding to at least one handwriting includes but is not limited to the following steps 2021-step 2023.

[0059] Step 2021, identify at least one handwriting, and obtain the first text content corresponding to at least one handwriting according to the identification result.

[0060] When the computer device receives the triggering operation of the user, the computer device obtains at least one handwriting included in the selected area or at least one handwriting included in the matching interface. The computer device identifies each handwriting, and by identifying the handwriting, the computer device can extract the text content corresponding to the handwriting. That is, the handwriting is the user's handwritten text, and the computer device needs to identify the handwritten text as the standard text stored in the computer device.

[0061] In a possible implementation manner, at least one handwriting includes at least one sub-handwriting, and any sub-handwriting is a continuous trajectory, representing a specific part or stroke written by the user. The process of identifying the handwriting and obtaining the first text content corresponding to the handwriting according to the identification result includes determining the trajectory point coordinates of at least one sub-handwriting, obtaining the generation time of at least one sub-handwriting, and obtaining the first text content based on the trajectory point coordinates and the generation time.

[0062] In order to identify the text content corresponding to the handwriting, the note software needs to obtain the trajectory point coordinates of each sub-handwriting on the note interface or the matching interface. The trajectory points can include the starting point, the intermediate turning point, the ending point, etc. The coordinates represent the position of the trajectory point on the interface. For example, if the user writes a character "two" on the note interface or the matching interface, the sub-handwriting corresponding to "two" includes the upper horizontal stroke and the lower horizontal stroke. The note software will record the trajectory point coordinates of the upper horizontal stroke and the lower horizontal stroke. The trajectory point coordinates of the upper horizontal stroke can include the starting point coordinates and the ending point coordinates of the upper horizontal stroke, and the trajectory point coordinates of the lower horizontal stroke can include the starting point coordinates and the ending point coordinates of the lower horizontal stroke.

[0063] The note-taking software also records when each sub-handwriting was written, i.e., the generation time of the sub-handwriting, which is used to identify continuous strokes, determine the writing order of sub-handwritings, determine the order of text, and analyze the semantic relationships between handwritings, so as to extract text content more accurately. Optionally, the generation time can be represented by a timestamp.

[0064] By analyzing the trajectory point coordinates and generation time of sub-handwritings, the note-taking software can identify the text content corresponding to the handwriting. This process may involve techniques such as shape matching and stroke recognition. For example, the note-taking software may compare continuous trajectory point coordinates with predefined character shapes to identify corresponding letters or Chinese characters. At the same time, combined with the generation time, the note-taking software can understand the relative order between sub-handwritings, thus constructing the complete text content.

[0065] Step 2022, determine the type of the first text content based on the first text content.

[0066] After obtaining the first text content corresponding to at least one handwriting, the type of the first text content can be determined by analyzing the first text content. The embodiments of the present application do not limit the type of the first text content. Optionally, the type of the first text content may include a term type, a formula type, or a text type, etc.

[0067] In a possible implementation manner, the process of determining the type of the first text content based on the first text content includes: if the first text content includes professional terms or proper nouns, determine that the type of the first text content is the term type; if the first text content includes a mathematical formula, determine that the type of the first text content is the formula type; if the first text content does not include professional terms, proper nouns, and mathematical formulas, determine that the type of the first text content is the text type.

[0068] Next, two scenarios of determining that the type of the first text content is the term type and the formula type are described respectively. In the case where the type of the first text content is not determined to be the term type or the formula type, it is determined that the type of the first text content is the text type.

[0069] Scenario 1, determine that the type of the first text content is the term type.

[0070] Optionally, the process of determining that the type of the first text content is a term type includes: preprocessing the first text content, obtaining an initial text according to the preprocessing result; extracting semantic features from the initial text, obtaining a semantic feature vector according to the semantic feature extraction result, inputting the semantic feature vector into a term recognition model, determining that the first text content includes at least one term according to the output result of the term recognition model, where the term recognition model is used to predict the term corresponding to the input semantic feature vector; and based on the fact that the first text content includes at least one term, determining that the type of the first text content is a term type.

[0071] Among them, the preprocessing includes at least one of removing special characters, word segmentation, or removing stop words. Removing special characters can remove irrelevant characters, formats, or marks in the first text content. Word segmentation can split the first text content into independent words or phrases. Removing stop words can simplify the first text content. For example, stop words refer to common words such as "de" (of), "he" (and), etc. Exemplarily, if the first text content is "@#Cell structure diagram", after preprocessing, the obtained initial text may be "Cell structure diagram".

[0072] After obtaining the initial text, semantic features can be further extracted from the initial text. Semantic feature extraction is to convert the initial text into a vector representation that can be recognized by a computer device. For example, the vector representation corresponding to "cell" is 01, and the vector representation corresponding to "structure diagram" is 02. Then, semantic feature extraction is performed on the initial text "Cell structure diagram", and the obtained semantic feature vector is 0102.

[0073] Input the semantic feature vector 0102 into the term recognition model. This term recognition model can recognize whether the term corresponding to the input semantic feature vector exists in the term image library. If it is a term existing in the term image library, the note software can determine that the first text content contains a term and determine what term the first text content contains. If the first text content contains a term, the note software will determine the type of the first text content as the term type.

[0074] Exemplarily, the establishment process of the term recognition model can be as follows: collect a corpus of professional vocabulary, and the corpus of professional vocabulary can include texts from related fields, definitions of professional terms, research papers, etc. For the collected corpus data of professional vocabulary, perform a series of the above-mentioned preprocessing operations. Through the above semantic feature extraction algorithm, convert the text into a vector representation that can be processed by a computer device. The embodiments of the present application do not limit the semantic feature extraction algorithm. For example, the semantic feature extraction algorithm can include the bag-of-words model, TF-IDF (Term Frequency-Inverse Document Frequency), word embedding, etc.

[0075] The corpus of professional vocabulary is divided into a training set and a test set according to a certain proportion. The training set is used to train a term recognition model, and the test set is used to evaluate the performance of the term recognition model. The embodiments of this application do not limit the type of the term recognition model. For example, SVM (Support Vector Machine), LR (Logistic Regression), decision tree, deep neural network, etc.

[0076] To ensure the accuracy of the term recognition model, the trained term recognition model is evaluated using the test set, and indicators such as the accuracy rate, precision rate, and recall rate of the model are calculated to measure the performance of the model. According to the evaluation results, the term recognition model is optimized. The embodiments of this application do not limit the optimization method. For example, parameter adjustment, feature selection, algorithm replacement, etc. Thus, the note software can accurately determine whether the first text content belongs to the term type through the term recognition model.

[0077] Scenario 2: Determine that the type of the first text content is a formula type.

[0078] Optionally, the process of determining that the type of the first text content is a formula type includes: determining at least one formula start character in the first text content; determining at least one formula end character corresponding to each of the at least one formula start character in the first text content; determining that the first text content includes at least one formula based on the at least one formula end character corresponding to each of the at least one formula start character; and determining that the type of the first text content is a formula type based on the fact that the first text content includes at least one formula.

[0079] Among them, the at least one formula start character includes the first mathematical symbol in the first text content, or the at least one formula start character further includes the first mathematical symbol after any character other than a mathematical symbol after the first mathematical symbol in the first text content. There is no character other than a mathematical symbol between any formula end character and the formula start character corresponding to the formula end character, and the first character after any formula end character is any character other than a mathematical symbol or the formula end character is the last character in the first text content.

[0080] The formula start character refers to the mathematical character that marks the start of a mathematical formula. For example, the formula start character can be an algebraic symbol, an operation symbol, a special symbol, a relation symbol, an ellipsis symbol, etc., and the special symbol can be Π. Taking the first text content as "y1 = 2x1, formula y2 = 3x2" as an example, the formula start characters are y1 and y2, y1 corresponds to the first mathematical symbol in the first text content, and y2 corresponds to the first mathematical symbol after any character other than a mathematical symbol after the first mathematical symbol in the first text content.

[0081] For any formula, any formula includes a formula start character and a formula end character, that is, the content of a mathematical formula is delimited by a pair of formula start characters and formula end characters. As the above example, for the formula start character y1, the formula end character is x1, that is, there is no character other than mathematical symbols between any formula end character and the corresponding formula start character, and the first character after any formula end character is any character other than mathematical symbols; for the formula start character y2, the formula end character is x2, that is, there is no character other than mathematical symbols between any formula end character and the corresponding formula start character, and any formula end character is the last character in the first text content.

[0082] Thus, by identifying at least one formula start character and its corresponding formula end character, it is possible to determine whether the first text content contains a formula. Extracting and delimiting the scope of a mathematical formula based on the formula start character and its corresponding formula end character can ensure the integrity and accuracy of the formula. Exemplarily, if the first text content contains at least one formula, the type of the first text content is determined as the formula type.

[0083] Step 2023, obtain a matching illustration image for at least one handwriting according to the type of the first text content and the first text content.

[0084] The computer device searches for the corresponding illustration image in the internal database of the note software or external resources outside the note software according to the type of the first text content and the first text content. For example, if the first text content of the text type is the name of a certain plant, the computer device calls the internal database of the note software to search for pictures related to the certain plant. If there are no pictures related to the certain plant in the internal database, external resources can be called. Exemplarily, the example of first searching the internal database and then searching external resources mentioned in the embodiments of the present application is only for illustration. For different first text contents, different user usage habits, the networking situation of the computer device, etc., the search method can be different.

[0085] The computer device automatically matches the illustration image according to the handwriting selected by the user, simplifies the process for the user to open other software or other computer devices to find the illustration image, and improves the efficiency and accuracy of note illustration.

[0086] Optionally, different methods can be adopted to obtain the illustration image according to different types of the first text content. Below, taking the types of the first text content including the term type, the formula type, and the text type as an example for illustration.

[0087] In Method 1, when the type of the first text content is a term type, a matching illustration image is obtained from a term image library. The term image library includes images corresponding to multiple terms respectively.

[0088] If the type of the first text content is a term type, for example, the first text content includes specific scientific names, technical terms or professional vocabulary, the note-taking software will access the term image library. Multiple terms and images corresponding to the multiple terms are stored in the term image library. For example, if the term "cell" is included in the first text content, the note-taking software will search for an image related to "cell" in the term image library and select the image that best matches the definition of the term as the illustration image.

[0089] To ensure the consistency and similarity in color and tone between the illustration image and the handwriting content, the process of obtaining a matching illustration image from the term image library for at least one term in the first text content includes: obtaining at least one first image corresponding to at least one term in the term image library; collecting the R (Red), G (Green), and B (Blue) values corresponding to each of the at least one first image, and collecting the RGB values corresponding to the note interface, and obtaining the similarity of the RGB values between each of the at least one first image and the note interface; determining the first image whose similarity of RGB values meets the similarity condition as the illustration image. Optionally, the similarity condition may be that the similarity of RGB values is greater than 80%.

[0090] Exemplarily, the note-taking software retrieves multiple first images corresponding to a term in the first text content from the term image library. For each retrieved first image, the note-taking software collects the RGB values of each first image, that is, the intensity values of the three color channels of red, green, and blue. To ensure the acquisition speed and thus improve the response speed of the note-taking software, when collecting the RGB values of each image, the RGB values of some pixel points can be selected to reduce the calculation amount. In this embodiment of the application, taking 100 pixel points as an example for illustration. The note-taking software records the RGB values of each pixel point as (Rx, Gx, Bx).

[0091] The RGB values corresponding to the note interface collected by the note-taking software can be the average of the RGB values of each pixel point of the note interface. The note-taking software records the RGB values corresponding to the note interface as (R0, G0, B0). By calculating the similarity of the RGB values between each pixel point of each first image and the note interface, it is ensured that the selected illustration image matches the color and tone of the note interface.

[0092] Exemplarily, the formula for calculating the RGB values of any pixel point of a first image and the RGB values of the note interface can be: similarity = (R0 - Rx) 2+(G0 - Gx) 2 +(B0 - Bx) 2 In the embodiment of the present application, the similarity between any pixel point of a first image and the RGB values of the note interface being less than or equal to 100 is used as a screening condition. At least one first image that meets the screening condition is screened out through the screening condition, and at least one first image that meets the screening condition is sorted from small to large according to the average value of the similarity between any pixel point and the RGB values of the note interface. The image with the smallest average value of the similarity between any pixel point and the RGB values of the note interface has the smallest color difference degree from the note interface. The pictures are arranged in the browsing area according to the color difference degree, and in response to the user's selection operation, the first image that meets the similarity condition is determined as the accompanying picture image. In this case, the similarity condition is the first image that meets the screening condition and is indicated by the user's selection operation.

[0093] Thus, it is possible to determine an accompanying picture image that matches or is similar to the color of the note interface and meets the user's needs, thereby enhancing the overall visual effect and consistency of the note, further improving the user experience, and ensuring the visual coherence and beauty of the note content.

[0094] In the second method, when the type of the first text content is a formula type, at least one formula in the first text content is input into a function generation tool, and an accompanying picture image that matches at least one handwriting is obtained according to the function image output by the function generation tool based on at least one formula.

[0095] If the type of the first text content is a formula type, for example, the first text content includes a mathematical formula, the note software will input the mathematical formula into a function generation tool. The function generation tool can generate a corresponding function image according to the input formula. For example, if the handwriting content is the mathematical formula y = x 2 , the function generation tool will generate an image representing the function, such as the shape of a parabola. Among them, the function generation tool can be a tool integrated in the note software, or the note software can call the relevant functions of the function generation tool of other software. For example, for the function y = x, a schematic diagram of a function image as shown in Figure 6 is generated.

[0096] In the third method, when the type of the first text content is a text type, at least one text keyword in the first text content is input into an AI (Artificial Intelligence) painting tool, and an accompanying picture image that matches at least one handwriting is obtained according to the semantic image generated by the AI painting tool based on at least one text keyword.

[0097] If the type of the first text content is text type, for the text keywords in the first text content, the note-taking software will process them using an AI painting tool, which can generate corresponding semantic images according to the text keywords. For example, if the text keyword is sunrise, the AI painting tool will generate an image of sunrise. The note-taking software matches a corresponding illustration image for the first text content according to the type of the first text content. This meets the actual needs of the note content and ensures the accuracy and consistency between the illustration image and the note content.

[0098] In a possible implementation manner, the process of obtaining at least one illustration image corresponding to at least one handwriting based on the semantic image generated by the AI painting tool according to at least one text keyword includes: obtaining at least one second image corresponding to at least one text keyword through the AI painting tool; receiving indication information for the second image, and when the indication information indicates that the second image meets the illustration condition, determining the second image as the illustration image.

[0099] When the indication information indicates that the second image does not meet the illustration condition, receiving modification information, where the modification information includes at least one of modification word information and image style information, obtaining at least one modified text keyword based on the modification information; obtaining at least one modified second image corresponding to the at least one modified text keyword through the AI painting tool, until the indication information received for the modified second image indicates that the modified second image meets the illustration condition, and determining the modified second image as the illustration image.

[0100] The note-taking software uses the AI painting tool to process the text keywords and generate corresponding semantic images, that is, the second images. For example, if the text keyword is sunrise, the AI painting tool will generate a corresponding image reflecting sunrise according to sunrise.

[0101] The note-taking software receives feedback or indication information from the user for the second image. The feedback or indication information is used to determine whether the second image meets the condition of being an illustration image. If the indication information shows that the second image meets the requirements of being an illustration image, the note-taking software will directly determine the second image as the illustration image for the handwriting. The indication information can be information corresponding to the user's operation of clicking on the second image. Meeting the indication information can be that the user clicks the OK button, and not meeting the indication information can be that the user clicks the Cancel button, etc.

[0102] If the user believes that the second image does not meet the criteria for being an illustration image, the note-taking software can prompt the user to provide modification information. The modification information can include modification word information, image style information, etc., to provide a basis for further adjustment by the AI painting tool. Based on the modification information provided by the user, the note-taking software will adjust the original text keywords or add new keywords to guide the AI painting tool to generate a new semantic image. Through the new text keywords, the AI painting tool will generate a new semantic image again. Through multiple adjustments and feedback, the note-taking software finally determines a semantic image that meets the user's needs as the illustration image for the handwriting.

[0103] The note-taking software can continuously optimize the generated semantic image according to the user's feedback and guidance until it meets the criteria for being an illustration image. This ensures that the final illustration image not only meets the semantic requirements of the handwriting content but also satisfies the user's aesthetics and preferences, improving the quality and readability of the notes and enhancing the user's note-taking experience.

[0104] Thus, through the above methods 1 to 3, illustration images corresponding to different types of first text content can be obtained.

[0105] Combined Figure 7 Taking the flowchart of note illustration shown as an example to illustrate the process of obtaining the illustration image. For the note interface, in response to selecting the handwriting by selecting the area and receiving the information that the user clicks the AI illustration button, that is, the trigger operation for the selected area; transmit the handwriting coordinate points in the selected area to the handwriting recognition algorithm, and obtain the text content corresponding to the handwriting in the selected area through the handwriting recognition algorithm; judge whether the type of the text content is a term type or a formula type. If it is a term type or a formula type, perform the corresponding operations for the term type or the formula type. If it is not a term type or a formula type, send the text keywords in the text content to Open-Ai, and generate a picture corresponding to the text content through Open-Ai. Open-Ai corresponds to the AI painting tool; judge whether the picture generated by Open-Ai meets the user's expectations. If it meets the user's expectations, use the picture generated by Open-Ai as the illustration image. If it does not meet the user's expectations, modify the text keywords in response to the user's operation of selecting the picture and clicking the modification button, select the picture style, and generate more pictures that meet the conditions.

[0106] See Figure 8Another process schematic diagram of note illustration is shown. For the note interface, in response to selecting handwriting in the selected area and receiving the information that the user clicks the illustration button, that is, in response to the trigger operation for the selected area, the handwriting in the selected area is input into the handwriting recognition algorithm. The handwriting recognition algorithm recognizes the handwriting as the first text content, and determines whether it is a professional term through the term recognition model, that is, whether there is a term in the first text content. Among them, the process of building the term recognition model includes collecting the corpus of professional terms, data preprocessing, feature extraction algorithm, dividing the corpus of professional terms into a training set and a test set according to a certain ratio, training the term recognition model, evaluating the performance of the term recognition model, and tuning and optimizing. For the relevant description of step 202, it will not be elaborated here.

[0107] If there is no term, call the AI painting tool to generate an image, extract the RGB values of 100 pixel points of the generated image, and record the Xth as (Rx, Gx, Bx). Calculate the sum of the squares of the differences between the RGB of each pixel point of the image and the RGB of the screenshot, that is, the similarity. Filter the images with a similarity less than 100. The images with a similarity less than 100 are sorted according to the size of the similarity, and the smaller the result value, the more forward the recommended position. The first 4 images with the smallest similarity are displayed in the default browsing area. In response to the user's selection operation, determine the selected image as the illustration image. In addition, after selecting the image in the browsing area and clicking More, other images with a similarity less than 100 can be displayed.

[0108] Exemplarily, for any first text content and any type of the first text content, at least one browsing image corresponding to the first text content can be obtained through the above methods 1 to 3, and the at least one browsing image is displayed in the browsing area of the note interface; the user can perform a selection operation on the at least one browsing image displayed in the browsing area, and the browsing image corresponding to the selection operation is the illustration image matching at least one handwriting.

[0109] See Figure 9 , in Figure 9 In another schematic diagram of the note interface shown, in response to the trigger operation for the selected area 401, the note interface displays at least one image in the browsing area 901 corresponding to the selected area 401. The browsing area 901 is used for the user to browse at least one browsing image corresponding to the handwriting in the selected area 401, so that the user can select the illustration image in the browsing area 901 according to the needs, and the illustration image matches the handwriting in the selected area 401.

[0110] Or, see Figure 10 , in Figure 10In another schematic diagram of the note interface shown, in response to a trigger operation on the illustration interface 501, at least one image is displayed in the browsing area 1001 within the illustration interface 501. The browsing area 1001 is used for the user to browse at least one browsing image corresponding to the handwriting within the illustration interface 501, so that the user can select an illustration image in the browsing area 1001 according to needs, and the illustration image matches the handwriting within the illustration interface 501. In addition, in the embodiments of the present application, the position of the browsing area 1001 is only for illustrative purposes, and the browsing area 1001 can also be located in the note interface.

[0111] For example, taking the first text content as the term type, and taking the determination of at least one handwriting to be illustrated through a selection operation as an example, the user records content about plants on the note interface, and this content includes different kinds of flowers. The user can select the handwriting containing the names of different kinds of flowers through a selection operation, and then perform a trigger operation. The note software will automatically search for browsing images related to the names of different kinds of flowers and display the relevant browsing images in the browsing area corresponding to the selection area. The note software can also provide various types of browsing images according to the user's preferences and needs. For example, hand-drawn illustrations, physical photos, and stick figures, etc., to meet the needs and styles of different users. The user can select a suitable browsing image as the illustration image according to their own preferences, making the note more in line with the user's usage habits and improving the quality of the note.

[0112] Step 203, display the illustration image on the note interface.

[0113] After obtaining the illustration image that matches at least one handwriting, the illustration image can be displayed on the note interface. In the embodiments of the present application, the position where the illustration image is displayed is not limited. For example, it can be displayed at any position on the note interface, or displayed in a designated image area, and the image area can be an area that can accommodate the size of the illustration image. Refer to Figure 11 Another schematic diagram of the note interface shown, this note interface includes a selection area 401 and an image area 1101. The image area 1101 can Figure 9 coincide with the browsing area 901 shown, so as to make full use of the usage space of the note interface. In addition, after the illustration image is displayed in the image area corresponding to the selection area, the size of the illustration image can also be adjusted in response to the user's stretching operation, and the embodiments of the present application do not limit this.

[0114] Optionally, refer to Figure 12 Another schematic diagram of the note interface shown, this note interface is Figure 10 the interface after the illustration image is selected in the note interface shown. After the illustration image is selected Figure 10The accompanying picture interface 501 therein is automatically closed or closed by the user's triggering operation, and the accompanying picture image is displayed on the note interface. The accompanying picture image can be displayed at any position on the note interface. In addition, the displayed accompanying picture image can change its display position in response to the user's dragging operation. The embodiments of the present application do not limit the dragging operation. For example, the user triggers and drags on the display screen of the computer device, or the user drags through the hardware connected to the computer device, etc.

[0115] In a possible implementation manner, the note interface is generated based on a note template, and the note template can be a default one or selected before taking notes. After generating the note interface based on the note template, the handwriting template of the note interface can also be changed. Optionally, the process of determining the note template may include obtaining the second text content in the template interface; determining the note template based on the second text content.

[0116] The note interface is generated based on a note template. The note template is a preset framework that determines the elements and layout to be included in the note. By using the note template, the user can quickly create a note with a consistent structure and format. When generating or changing the note interface, the note software extracts the second text content from the template interface. The template interface can be an interface different from the note interface and is used for the user to select the note template.

[0117] In the embodiments of the present application, the second text content includes, but is not limited to, the name of the reference template or descriptive text, etc. Optionally, the note software includes a template library that contains various different reference templates, such as Cornell notes. By comparing the second text content with the content in the template library, it can be determined whether the template library includes the template corresponding to the second text content.

[0118] In a possible implementation manner, the process of determining the note template based on the second text content includes, when the second text content indicates any reference template in the template library, determining any reference template as the note template, and the template library includes multiple reference templates. For example, if the second text content indicates Cornell notes in the template library, then determine the Cornell notes template in the template library as the note template.

[0119] When the second text content does not indicate any reference template in the template library, at least one template image is generated based on the second text content, and the size of any template image matches the size of the note interface. For any template image in the at least one template image, the RGB values of a reference number of pixel points in any template image are collected. Based on the variance between the RGB values of the reference number of pixel points, the reference value corresponding to any template image is obtained, and the template image in the at least one template image whose reference value meets the reference condition is determined as the note template.

[0120] Among them, the size of the template image matches the size of the note interface, providing users with different recording layout options. For each generated template image, the note software collects the RGB values of a certain number of pixels for evaluating the color distribution and variation of each template image. The embodiments of the present application do not limit the number of pixels collected. For example, the number of pixels is 100.

[0121] By analyzing the RGB values of the collected pixels, the note software calculates the variance between the RGB values of the collected pixels. The variance can reflect the range of pixel color changes, thereby evaluating the diversity or complexity of the template image. According to the calculated reference value, the note software filters out the template images whose reference values meet the reference conditions. The reference conditions can be the range of color changes, the complexity of the template image, etc., to ensure that the selected note template matches the user's recording needs. For the note templates whose calculated variances meet the variance conditions, they are sorted and displayed in ascending order of variance, and the note template is determined based on the user's template selection indication. The embodiments of the present application do not limit the variance conditions. For example, the variance condition is less than or equal to 200.

[0122] See Figure 13 A schematic diagram of a template interface is shown. The template interface includes a description input box for the user to input the second text content, and a template recommendation part for providing templates or reference templates that meet the variance conditions. Thus, according to the user's needs and preferences, note templates can be flexibly obtained. Generating a note interface through the note template simplifies the note recording process, ensures the structuring of note content, and improves the user experience.

[0123] In summary, this method automatically obtains and displays a matching image through at least one handwriting to be matched with an image, simplifies the operation process of note matching with an image, and improves the efficiency of note matching with an image. Moreover, the automatically obtained matching image is matched with at least one handwriting, making the accuracy of the matching image relatively high.

[0124] See Figure 14 , the embodiments of the present application provide a note matching with an image device, and the device includes:

[0125] A display module 1401, configured to display a note interface;

[0126] A determination module 1402, configured to determine at least one handwriting to be matched with an image based on the note interface;

[0127] An acquisition module 1403, configured to acquire a matching image that matches at least one handwriting;

[0128] The display module 1401 is further configured to display the matching image on the note interface.

[0129] In a possible implementation, the note interface includes multiple handwritings recorded; a determination module 1402, configured to, in response to a selection operation on the note interface, display a selection area corresponding to the selection operation on the note interface, and determine at least one handwriting to be paired with an image based on at least one handwriting included in the selection area.

[0130] In a possible implementation, the determination module 1402 is configured to, in response to a trigger operation for the image pairing function, display an image pairing interface; and determine at least one handwriting to be paired with an image based on at least one handwriting input in the image pairing interface.

[0131] In a possible implementation, an acquisition module 1403 is configured to recognize at least one handwriting, obtain first text content corresponding to the at least one handwriting according to the recognition result; determine the type of the first text content based on the first text content; and obtain a paired image matching the at least one handwriting according to the type of the first text content and the first text content.

[0132] In a possible implementation, the acquisition module 1403 is configured to, when the type of the first text content is a term type, obtain a paired image matching at least one term in the first text content in a term image library, where the term image library includes images corresponding to multiple terms respectively; when the type of the first text content is a formula type, input at least one formula in the first text content into a function generation tool, and obtain a paired image matching the at least one handwriting according to a function image output by the function generation tool based on the at least one formula; when the type of the first text content is a text type, input at least one text keyword in the first text content into an AI painting tool, and obtain a paired image matching the at least one handwriting according to a semantic image generated by the AI painting tool based on the at least one text keyword.

[0133] In a possible implementation, the acquisition module 1403 is configured to obtain at least one first image corresponding to at least one term in the term image library; collect RGB values corresponding to the at least one first image respectively, and collect RGB values corresponding to the note interface; obtain the similarity of the RGB values between the at least one first image and the note interface respectively; and determine a first image whose similarity of the RGB values meets the similarity condition among the at least one first image as the paired image.

[0134] In a possible implementation, the acquisition module 1403 is configured to obtain at least one second image corresponding to text keywords through an AI painting tool; receive indication information for the second image; determine the second image as a matching image when the indication information indicates that the second image meets the matching condition; when the indication information indicates that the second image does not meet the matching condition, receive modification information, where the modification information includes at least one of modification word information and image style information; obtain at least one modified text keyword based on the modification information; obtain at least one modified second image corresponding to the at least one modified text keyword through the AI painting tool; until the indication information received for the at least one modified second image indicates that the at least one modified second image meets the matching condition, determine the at least one modified second image as a matching image.

[0135] In a possible implementation, the acquisition module 1403 is configured to preprocess the first text content, obtain an initial text according to the preprocessing result, where the preprocessing includes at least one of removing special characters, word segmentation, or removing stop words; extract semantic features from the initial text, and obtain a semantic feature vector according to the semantic feature extraction result; input the semantic feature vector into a term recognition model, and determine that the first text content includes at least one term according to the output result of the term recognition model, where the term recognition model is used to predict the term corresponding to the input semantic feature vector; based on the fact that the first text content includes at least one term, determine that the type of the first text content is a term type.

[0136] In a possible implementation, the acquisition module 1403 is configured to determine at least one formula start character in the first text content, where the at least one formula start character includes the first mathematical symbol in the first text content, or the at least one formula start character further includes the first mathematical symbol after any character other than a mathematical symbol after the first mathematical symbol in the first text content; determine the formula end character corresponding to each of the at least one formula start character in the first text content, where there is no character other than a mathematical symbol between any formula end character and the formula start character corresponding to the formula end character, and the first character after any formula end character is any character other than a mathematical symbol or the formula end character is the last character in the first text content; determine that the first text content includes at least one formula based on the formula end character corresponding to each of the at least one formula start character; based on the fact that the first text content includes at least one formula, determine that the type of the first text content is a formula type.

[0137] In a possible implementation, at least one handwriting includes at least one sub-handwriting, and any sub-handwriting is a continuous trajectory. The acquisition module 1403 is configured to determine the trajectory point coordinates of the at least one sub-handwriting; obtain the generation time of the at least one sub-handwriting; and obtain the first text content based on the trajectory point coordinates and the generation time.

[0138] In a possible implementation, the note interface is generated based on a note template. The acquisition module 1403 is further configured to acquire the second text content in the template interface; the determination module 1402 is further configured to determine the note template based on the second text content.

[0139] In a possible implementation, the determination module 1402 is configured to determine any reference template in the template library as the note template when the second text content indicates any reference template in the template library, and the template library includes multiple reference templates.

[0140] When the second text content does not indicate any reference template in the template library, at least one template image is generated based on the second text content, and the size of any template image matches the size of the note interface; for any template image in the at least one template image, the RGB values of the reference number of pixel points in the any template image are collected, and the reference value corresponding to the any template image is obtained based on the variance between the RGB values of the reference number of pixel points; the template image in the at least one template image whose reference value meets the reference condition is determined as the note template.

[0141] In summary, the device automatically acquires and displays the illustration image through at least one handwriting to be illustrated, simplifies the operation process of note illustration, and improves the efficiency of note illustration. Moreover, the automatically acquired illustration image is matched with at least one handwriting, making the accuracy of the illustration image relatively high.

[0142] It should be noted that when the device provided in the above embodiment implements its functions, only the above-mentioned division of each functional module is used for illustration. In actual functions, the above functions can be allocated to different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device provided in the above embodiment and the method embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0143] Figure 15 It is a schematic structural diagram of a terminal provided by an embodiment of the present application. The terminal may be, for example: a smart phone, a tablet computer, a player, a notebook computer or a desktop computer. The terminal may also be referred to by other names such as user equipment, portable terminal, laptop terminal, desktop terminal, etc.

[0144] Generally, the terminal includes: a processor 1501 and a memory 1502.

[0145] The processor 1501 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. The processor 1501 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), or PLA (Programmable Logic Array). The processor 1501 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the wake state, also known as the CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 1501 may be integrated with a GPU (Graphics Processing Unit), and the GPU is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1501 may further include an AI (Artificial Intelligence) processor, and the AI processor is used to process computational operations related to machine learning.

[0146] The memory 1502 may include one or more computer-readable storage media, and the computer-readable storage media may be non-transitory. The memory 1502 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash storage devices. In some embodiments, the non-transitory computer-readable storage media in the memory 1502 is used to store at least one instruction, and the at least one instruction is used to be executed by the processor 1501 so that the terminal implements the note illustration method provided in the method embodiments of the present application.

[0147] In some embodiments, the terminal may also optionally include: a peripheral device interface 1503 and at least one peripheral device. The processor 1501, the memory 1502, and the peripheral device interface 1503 may be connected through a bus or signal lines. Each peripheral device may be connected to the peripheral device interface 1503 through a bus, signal lines, or a circuit board. Specifically, the peripheral devices include at least one of a radio frequency circuit 1504, a display screen 1505, a camera assembly 1506, an audio circuit 1507, and a power supply 1508.

[0148] The peripheral device interface 1503 can be used to connect at least one I / O (Input / Output) related peripheral device to the processor 1501 and the memory 1502. In some embodiments, the processor 1501, the memory 1502, and the peripheral device interface 1503 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1501, the memory 1502, and the peripheral device interface 1503 can be implemented on a separate chip or circuit board, and this embodiment does not limit this.

[0149] The radio frequency circuit 1504 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 1504 communicates with a communication network and other communication devices through electromagnetic signals. The radio frequency circuit 1504 converts an electrical signal into an electromagnetic signal for transmission, or converts the received electromagnetic signal into an electrical signal. Optionally, the radio frequency circuit 1504 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a subscriber identity module card, and so on. The radio frequency circuit 1504 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes but is not limited to: metropolitan area network, generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area network, and / or WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 1504 may further include a circuit related to NFC (Near Field Communication), and this application does not limit this.

[0150] The display screen 1505 is used to display the UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 1505 is a touch display screen, the display screen 1505 also has the ability to collect touch signals on or above the surface of the display screen 1505. The touch signals can be input as control signals to the processor 1501 for processing. At this time, the display screen 1505 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, there can be one display screen 1505, which is provided on the front panel of the terminal; in some other embodiments, there can be at least two display screens 1505, which are respectively provided on different surfaces of the terminal or in a folding design; in some other embodiments, the display screen 1505 can be a flexible display screen, which is provided on the curved surface or the folding surface of the terminal. Even, the display screen 1505 can also be set to an irregular non-rectangular shape, that is, a special-shaped screen. The display screen 1505 can be prepared using materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0151] The camera module 1506 is used to collect images or videos. Optionally, the camera module 1506 includes a front camera and a rear camera. Generally, the front camera is provided on the front panel of the terminal, and the rear camera is provided on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth camera, a wide-angle camera, and a telephoto camera respectively, so as to implement functions such as the combination of the main camera and the depth camera to achieve the background blurring function, the combination of the main camera and the wide-angle camera to achieve panoramic shooting and VR (Virtual Reality) shooting functions or other combined shooting functions. In some embodiments, the camera module 1506 can also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to the combination of a warm light flash and a cold light flash, which can be used for light compensation under different color temperatures.

[0152] The audio circuit 1507 may include a microphone and a speaker. The microphone is used to collect sound waves of the user and the environment, and convert the sound waves into electrical signals for input to the processor 1501 for processing, or input to the radio frequency circuit 1504 to implement voice communication. For the purpose of stereo collection or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the terminal. The microphone may also be an array microphone or an omnidirectional collection microphone. The speaker is used to convert the electrical signals from the processor 1501 or the radio frequency circuit 1504 into sound waves. The speaker may be a traditional thin film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert electrical signals into sound waves audible to humans, but also convert electrical signals into sound waves inaudible to humans for uses such as ranging. In some embodiments, the audio circuit 1507 may further include a headphone jack.

[0153] The power supply 1508 is used to supply power to each component in the terminal. The power supply 1508 may be alternating current, direct current, a disposable battery or a rechargeable battery. When the power supply 1508 includes a rechargeable battery, the rechargeable battery may support wired charging or wireless charging. The rechargeable battery may also be used to support fast charging technology.

[0154] In some embodiments, the terminal further includes one or more sensors 1509. The one or more sensors 1509 include but are not limited to: an acceleration sensor 1510, a gyroscope sensor 1511, a pressure sensor 1512, an optical sensor 1513, and a proximity sensor 1514.

[0155] The acceleration sensor 1510 can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established by the terminal. For example, the acceleration sensor 1510 can be used to detect the components of the gravitational acceleration on the three coordinate axes. The processor 1501 can control the display screen 1505 to display the user interface in a landscape view or a portrait view according to the gravitational acceleration signal collected by the acceleration sensor 1510. The acceleration sensor 1510 can also be used for collecting game or user's motion data.

[0156] The gyroscope sensor 1511 can detect the body direction and rotation angle of the terminal. The gyroscope sensor 1511 can cooperate with the acceleration sensor 1510 to collect the 3D actions of the user on the terminal. According to the data collected by the gyroscope sensor 1511, the processor 1501 can implement the following functions: motion sensing (such as changing the UI according to the user's tilting operation), image stabilization during shooting, game control, and inertial navigation.

[0157] The pressure sensor 1512 can be disposed on the side frame of the terminal and / or the lower layer of the display screen 1505. When the pressure sensor 1512 is disposed on the side frame of the terminal, it can detect the holding signal of the user on the terminal, and the processor 1501 performs left / right hand recognition or shortcut operations according to the holding signal collected by the pressure sensor 1512. When the pressure sensor 1512 is disposed on the lower layer of the display screen 1505, the processor 1501 controls the operable controls on the UI interface according to the pressure operation of the user on the display screen 1505. The operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

[0158] The optical sensor 1513 is used to collect the ambient light intensity. In one embodiment, the processor 1501 can control the display brightness of the display screen 1505 according to the ambient light intensity collected by the optical sensor 1513. Specifically, when the ambient light intensity is high, the display brightness of the display screen 1505 is increased; when the ambient light intensity is low, the display brightness of the display screen 1505 is decreased. In another embodiment, the processor 1501 can also dynamically adjust the shooting parameters of the camera module 1506 according to the ambient light intensity collected by the optical sensor 1513.

[0159] The proximity sensor 1514, also known as the distance sensor, is usually disposed on the front panel of the terminal. The proximity sensor 1514 is used to collect the distance between the user and the front of the terminal. In one embodiment, when the proximity sensor 1514 detects that the distance between the user and the front of the terminal is gradually decreasing, the processor 1501 controls the display screen 1505 to switch from the lit state to the off state; when the proximity sensor 1514 detects that the distance between the user and the front of the terminal is gradually increasing, the processor 1501 controls the display screen 1505 to switch from the off state to the lit state.

[0160] Those skilled in the art can understand that Figure 15 the structure shown in

[0161] does not constitute a limitation on the terminal, and may include more or fewer components than shown in the figure, or combine some components, or adopt different component arrangements.

[0162] In an exemplary embodiment, there is also provided a computer device, which includes a processor and a memory, and at least one computer program is stored in the memory. The at least one computer program is loaded and executed by one or more processors, so that the computer device implements any of the above note illustration methods.

[0163] In one possible implementation, the above computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, optical data storage device, etc.

[0164] In an exemplary embodiment, a computer program product or a computer program is further provided. The computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes any one of the above note illustration methods.

[0165] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data for analysis, stored data, displayed data, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with the relevant laws, regulations and standards of relevant countries and regions. For example, the relevant data related to note illustration in this application are obtained under full authorization.

[0166] It should be understood that the "plurality" mentioned herein refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist simultaneously, and B exists alone. The character " / " generally represents an "or" relationship between the associated objects before and after.

[0167] The above are only exemplary embodiments of the present application, and are not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for matching pictures with notes, characterized in that, The method includes: Display a note interface, and determine at least one handwriting to be accompanied by a picture based on the note interface; Obtain a picture-matching image that matches the at least one handwriting; Display the picture-matching image on the note interface.

2. The method according to claim 1, wherein The note interface includes a plurality of recorded handwritings; the determining at least one handwriting to be accompanied by a picture based on the note interface includes: In response to a selection operation on the note interface, display a selection area corresponding to the selection operation on the note interface, and determine the at least one handwriting to be accompanied by a picture based on at least one handwriting included in the selection area.

3. The method according to claim 1, characterized in that, The determining at least one handwriting to be accompanied by a picture based on the note interface includes: In response to a trigger operation for the picture accompaniment function, display a picture accompaniment interface; Determine the at least one handwriting to be accompanied by a picture based on at least one handwriting input in the picture accompaniment interface.

4. The method according to claim 1, wherein The obtaining a picture-matching image that matches the at least one handwriting includes: Identify the at least one handwriting, and obtain first text content corresponding to the at least one handwriting according to the identification result; Determine the type of the first text content based on the first text content; Obtain a picture-matching image that matches the at least one handwriting according to the type of the first text content and the first text content.

5. The method according to claim 4, characterized in that, The obtaining a picture-matching image that matches the at least one handwriting according to the type of the first text content and the first text content includes: When the type of the first text content is a term type, obtain a picture-matching image that matches at least one term in the first text content in a term image library, where the term image library includes images corresponding to a plurality of terms respectively; When the type of the first text content is a formula type, input at least one formula in the first text content into a function generation tool, and obtain a picture-matching image that matches the at least one handwriting according to a function image output by the function generation tool based on the at least one formula; When the type of the first text content is a text type, input at least one text keyword in the first text content into an artificial intelligence (AI) painting tool, and obtain a picture-matching image that matches the at least one handwriting according to a semantic image generated by the AI painting tool based on the at least one text keyword.

6. The method according to claim 5, wherein The obtaining a picture-matching image that matches at least one term in the first text content in the term image library includes: Obtain at least one first image corresponding to the at least one term in the term image library; Collect the red, green, and blue (RGB) values corresponding to the at least one first image respectively, and collect the RGB values corresponding to the note interface; Obtain the similarity of the RGB values between the at least one first image and the note interface respectively; Determine that the first image whose similarity of the RGB values in the at least one first image satisfies the similarity condition is the picture-matching image.

7. The method according to claim 5, wherein The obtaining a picture-matching image that matches the at least one handwriting according to a semantic image generated by the AI painting tool based on the at least one text keyword includes: Obtain a second image corresponding to the at least one text keyword through the AI painting tool; Receive indication information for the second image; When the indication information indicates that the second image meets the illustration condition, determine the second image as the illustration image; When the indication information indicates that the second image does not meet the illustration condition, receive modification information, where the modification information includes at least one of modification word information and image style information; obtain at least one modified text keyword based on the modification information; obtain a modified second image corresponding to the at least one modified text keyword through the AI painting tool; until the indication information received for the modified second image indicates that the modified second image meets the illustration condition, determine the modified second image as the illustration image.

8. The method according to claim 4, characterized in that, The determining the type of the first text content based on the first text content includes: Preprocess the first text content, and obtain an initial text according to the preprocessing result. The preprocessing includes at least one of removing special characters, word segmentation, or removing stop words; Extract semantic features from the initial text, and obtain a semantic feature vector according to the semantic feature extraction result; Input the semantic feature vector into a term recognition model, and determine that the first text content includes at least one term according to the output result of the term recognition model. The term recognition model is used to predict the term corresponding to the input semantic feature vector; Based on the fact that the first text content includes at least one term, determine that the type of the first text content is the term type.

9. The method according to claim 4, wherein The determining the type of the first text content based on the first text content includes: Determine at least one formula start character in the first text content. The at least one formula start character includes the first mathematical symbol in the first text content, or the at least one formula start character further includes the first mathematical symbol after any character other than the mathematical symbol after the first mathematical symbol in the first text content; Determine the formula end characters respectively corresponding to the at least one formula start character in the first text content. Any formula end character and the formula start character corresponding to the formula end character do not include any character other than the mathematical symbol therebetween, and the first character after the any formula end character is any character other than the mathematical symbol or the any formula end character is the last character in the first text content; Based on the formula end characters respectively corresponding to the at least one formula start character, determine that the first text content includes at least one formula; Based on the fact that the first text content includes at least one formula, determine that the type of the first text content is the formula type.

10. The method according to any one of claims 4-9, characterized in that, The at least one handwriting includes at least one sub-handwriting, and any sub-handwriting is a continuous trajectory. The recognizing the at least one handwriting and obtaining the first text content corresponding to the at least one handwriting according to the recognition result includes: Determine the trajectory point coordinates of the at least one sub-handwriting; Obtain the generation time of the at least one sub-handwriting; Obtain the first text content based on the trajectory point coordinates and the generation time.

11. According to the method described in any one of claims 1-9, characterized in that, The note interface is generated based on a note template, and the method further includes: Obtain the second text content in the template interface; Determine the note template based on the second text content.

12. The method according to claim 11, wherein The determining the note template based on the second text content includes: When the second text content indicates any reference template in the template library, determine the any reference template as the note template, and the template library includes multiple reference templates; When the second text content does not indicate any reference template in the template library, generate at least one template image based on the second text content, and the size of any template image matches the size of the note interface; For any one of the at least one template images, collect the RGB values of a reference number of pixel points in the any template image, and obtain the reference value corresponding to the any template image based on the variance between the RGB values of the reference number of pixel points; Determine the template image whose reference value in the at least one template image meets the reference condition as the note template.

13. A note illustration device, characterized in that, The device includes: A display module for displaying a note interface; A determination module for determining at least one handwriting to be accompanied by a picture based on the note interface; An acquisition module for acquiring a picture-matching image that matches the at least one handwriting; The display module is further configured to display the picture-matching image on the note interface.

14. A computer device, characterized in that, The computer device includes a processor and a memory, and at least one computer program is stored in the memory. The at least one computer program is loaded and executed by the processor to enable the computer device to implement the note picture-matching method according to any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that, At least one computer program is stored in the computer-readable storage medium. The at least one computer program is loaded and executed by a processor to enable a computer to implement the note picture-matching method according to any one of claims 1 to 12.

Images