A user drawing style quick learning method, system, device and medium of a hand-drawing input method

Abstract

A user drawing style quick learning method, system, device and medium of a hand-drawing input method, the method is: obtaining the recognition network model parameters of the hand-drawing input method trained offline by using a large data set; randomly selecting n hand-drawing image samples for each hand-drawing category supported by the hand-drawing input method from the recognition network model training data set of the offline training hand-drawing input method as the drawing examples of each hand-drawing category; drawing no more than n hand-drawing image samples for each category in all or part of the hand-drawing categories supported by the hand-drawing input method; extracting the source domain and the target domain, and updating the parameters of the basic model of the hand-drawing input method by using a supervised conditional adversarial domain adaptation algorithm; the system, device and medium can enable the user to quickly learn the user drawing style of the hand-drawing input method; the recognition accuracy is high, and the recognition accuracy of the new drawing data of the user of the hand-drawing input method can be quickly improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of human-computer interaction, and particularly relates to a user drawing style quick learning method, system, device and medium for a hand-drawing input method, which can be used for quick learning and personalized adaptation of the hand-drawing input method to different user drawing styles. BACKGROUND

[0002] Hand-drawn sketches are simple in line and simple in drawing method, and can express rich semantic information. As a computer interaction language, hand-drawn sketches are widely used in many interactive scenarios. The development of deep learning has greatly promoted the field of computer vision. Under the condition of sufficient data, after training a deep convolutional network, efficient recognition and classification of data can be achieved. The hand-drawing input method recognition model based on the deep convolutional network needs to be trained using a large amount of hand-drawing data to achieve accurate recognition of hand-drawing data. However, when the drawing style of the user is quite different from the drawing style contained in the large data, the recognition accuracy of the recognition model trained offline using the collected large data will be greatly reduced. This situation will result in poor performance of the developed hand-drawing input method when used by users with different drawing styles. It is unrealistic to let each user collect large data for retraining of the model, and the correct approach should be to quickly optimize the model using a small amount of data drawn by the user during use. This faces the problem of quick learning of the user drawing style under the condition of small sample.

[0003] Patent application CN11510032A discloses a shape recognition and correction method, device and storage medium based on hand-drawing trajectory, which comprises: based on a pre-trained neural network model, performing large classification recognition on the obtained hand-drawing trajectory, so that the hand-drawing trajectory is classified into one of a closed figure and a non-closed figure; when the hand-drawing trajectory is a closed figure, calculating the inflection points existing in the hand-drawing trajectory to determine the specific shape of the hand-drawing trajectory; when the hand-drawing trajectory is a non-closed figure, calculating the curvatures between every three adjacent data points of the hand-drawing trajectory to obtain the number of feature points, and determining the specific shape of the hand-drawing trajectory according to the number of feature points; according to the different specific shapes of the obtained hand-drawing trajectory, and according to the inflection points or / and the feature points, the hand-drawing trajectory is corrected to make the hand-drawing trajectory fit the standard figure. The present application can improve the accuracy of hand-drawing geometric figure recognition, and can quickly and accurately recognize the figure category drawn by the user and correct it. However, since different drawing personnel will show different drawing styles when drawing the same icon, the network model trained based on the data set collected by part of the drawing personnel may not be suitable for other people with large differences in drawing styles. The present application only proposes a hand-drawing trajectory shape recognition method, and does not solve the problem of quick adaptation of different personnel drawing styles. SUMMARY

[0004] To solve the above problems, the application provides a user drawing style quick learning method, system, device and medium of a hand-drawing input method, which can realize quick learning of user data style based on small sample data and combined with domain adversarial idea; test results prove that the application can realize quick improvement of recognition accuracy of new drawing data of users with different drawing styles by using a small amount of data, and improve the adaptability of the hand-drawing input method to new users.

[0005] To achieve the above purpose, the technical scheme adopted by the application is:

[0006] A user drawing style quick learning method of a hand-drawing input method, comprising the following steps:

[0007] S1, obtaining hand-drawing input method recognition network model parameters trained offline using a large data set, including feature extractor parameters and classifier parameters, as basic model parameters of the hand-drawing input method;

[0008] S2, randomly selecting n hand-drawing image samples for each hand-drawing category supported by the hand-drawing input method from the offline training data set of the hand-drawing input method recognition network model as drawing example samples of each hand-drawing category; the hand-drawing image samples are drawn by developers of the hand-drawing input method or specially recruited collection personnel;

[0009] S3, a user with a drawing style different from that of the drawing personnel in step S2, i.e., a hand-drawing input method user, draws no more than n hand-drawing image samples for each category of all or part of the hand-drawing categories supported by the hand-drawing input method;

[0010] S4, taking the n drawing example samples randomly selected for each hand-drawing category in step S2 as the source domain, and taking the no more than n samples drawn by the user for each category in step S3 as the target domain, and using supervised conditions to realize parameter updating of the basic model of the hand-drawing input method by using an adversarial domain self-adaptive algorithm.

[0011] Further, in step S1, the hand-drawing input method recognition network model is obtained offline based on a large data set.

[0012] Further, in step S3, the hand-drawing input method user needs to collect no more than n samples for each category of all or part of the hand-drawing categories supported by the hand-drawing input method as data reflecting the personal drawing style of the user.

[0013] Further, the specific method of step S4 is: taking the samples randomly selected from the large data set in S2 as source domain data taking the samples drawn by the user as target domain data wherein is hand-drawing data, and For category labels, N s and N t Let the number of samples be denoted as ; then, a supervised adversarial algorithm is used to update the parameters of the recognition model. The optimization objective of the algorithm is .

[0014] and in

[0015]

[0016]

[0017] Among them, E (*) Let f be the expectation function, G be the recognition model, D be the domain discriminant model, and f be the domain discriminant model. i s and To identify the features output by the model, and To identify the model's prediction results, Represents the source domain D s The i-th sample in the series, Represents the target domain D t The j-th sample in the dataset, where L is the cross-entropy loss.

[0018] A user-defined drawing style rapid learning system for a hand-drawn input method includes:

[0019] A hand-drawn category recognition network model is used to recognize hand-drawn data created by users of hand-drawn input methods.

[0020] The hand-drawn category recommendation module recommends standard hand-drawn icons to users based on the recognition results of the recognition network, allowing users to select the correct standard hand-drawn category icon corresponding to their hand-drawn icon, similar to selecting the correct Chinese character after entering pinyin in a Chinese pinyin input method.

[0021] The user hand-drawn data storage module is used to record the user's hand-drawn samples and the category tags selected by the user from the hand-drawn categories automatically recommended by the hand-drawn input method;

[0022] The network model optimization module optimizes the parameters of the hand-drawn input method recognition network model using step S4.

[0023] A device for rapid learning of user drawing styles in a hand-drawn input method includes:

[0024] Memory, used to store computer programs;

[0025] A processor is used to implement the user drawing style rapid learning method of a hand-drawn input method described in steps S1 to S4 when executing the computer program.

[0026] A computer readable storage medium stores a computer program, and the computer program is executed by a processor to enable a user to quickly learn a user drawing style of a hand-drawing input method.

[0027] Compared with the prior art, the present application has the following advantages and beneficial effects:

[0028] (1) The user drawing style quick learning method of the hand-drawing input method provided by the present application can realize the quick improvement of the recognition accuracy of the new drawing data of the hand-drawing input method user with different drawing styles by using only a small sample for a small amount of calculation.

[0029] (2) Compared with the existing domain adaptation method, the present application uses a supervised domain adversarial algorithm, and only a small amount of samples are required for the source domain and the target domain. The example data of each hand-drawing category is used as the source domain data, and the actual use process is also feasible. In the use process, the user does not need to collect a large amount of data for each hand-drawing category to optimize the parameters of the recognition network model. The hand-drawing input method will recommend the standard hand-drawing category according to the data being drawn by the user, and the user will select the correct category corresponding to the drawn data (similar to when the user uses the Chinese pinyin input method, the input method will recommend a word or a word according to the letter sequence input by the user, and the user will select the word or word he wants), which provides labeled target domain data. These factors ensure the feasibility of the domain adaptation algorithm based on the supervised domain adversarial algorithm, and greatly improve the accuracy of domain adaptation. BRIEF DESCRIPTION OF DRAWINGS

[0030] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0031] Figure 1 is a flowchart of the embodiment of the present application.

[0032] Figure 2 is a conditional adversarial algorithm structure diagram of the present application.

[0033] Figure 3 is a domain discrimination model network structure in the embodiment of the present application. DETAILED DESCRIPTION

[0034] With reference to the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by a person of ordinary skill in the art without creative effort belong to the scope of protection of the present application.

[0035] The embodiment of the present application provides a user drawing style quick learning method of a hand-drawing input method, and specific embodiments include the following steps:

[0036] S1, obtaining the recognition network model parameters of the hand-drawing input method trained offline by using a large data set, including feature extractor parameters and classifier parameters, as the basic model parameters of the hand-drawing input method; the hand-drawing samples in the large data set are drawn by the developers of the hand-drawing input method or specially recruited collection personnel, and the actual users of the hand-drawing input method are often different from the collection personnel of the large data set, resulting in that the drawing styles of the two groups of people may not be the same. However, it is not necessary to collect a large amount of data for each hand-drawing category to optimize the parameters of the recognition network model, which can be used as the initial parameters of the hand-drawing input method recognition model for normal use by the user.

[0037] In the embodiment, ResNet18 is selected as the recognition model of the hand-drawing input method. In the embodiment, the hand-drawing data comes from NATTO APP-6(B) joint military symbol standard, which contains 374 categories of hand-drawing data drawn by 67 people, and each hand-drawing category contains 100 samples. Part of the data drawn by the people is used as the large data for training the recognition model, and the data of the remaining people is used as the data for verifying the quick learning effect of the user drawing style.

[0038] S2, in step S2, the user cannot obtain the large data in S1 during the use of the hand-drawing input method, but the developers of the hand-drawing input method or specially recruited collection personnel can randomly select 5 samples for each hand-drawing category from the large data as drawing examples of each hand-drawing category for the actual user to refer to. The existing domain adaptation algorithm uses a large amount of sample data of the source domain and the target domain, which is feasible in algorithm research, but has poor feasibility in actual application. For example, in the application of the hand-drawing input method, the developers cannot open source the large data for training the recognition model while releasing the hand-drawing input method, and the user also has no condition to retrain the entire network. However, from the perspective of the application of the hand-drawing input method, it is completely reasonable to provide several examples for each category to guide the user to learn to draw. Therefore, the present application randomly selects 5 samples for each category from the offline training large data. In order to ensure that these samples cover more drawing styles, the coverage of the training data set drawing personnel needs to be considered when selecting these samples, rather than simply selecting the samples of a certain person.

[0039] S3. For the hand-drawn input method, users do not need to collect a large amount of data for each hand-drawn category to optimize the parameters of the recognition network model. Users only need to draw no more than 5 samples for each icon in all or some categories. In this embodiment, we set up test scenarios where users draw 1, 2, and 5 samples for each category, and also set up test scenarios where users draw 2 or 5 samples for only some categories.

[0040] S4 uses five example samples from each hand-drawn category as the source domain and no more than five user-drawn samples from each category as the target domain. A supervised conditional adversarial domain adaptation algorithm is then used to update the parameters of the basic hand-drawn input method model. The structure diagram of the conditional adversarial algorithm is shown below. Figure 2 As shown, samples randomly selected from the large dataset are used as the source domain data. Use user-collected samples as target domain data in and For hand-drawn data, and For category labels, N s and N t Let the number of samples be denoted as ; then, a supervised adversarial algorithm is used to update the parameters of the recognition model. The optimization objective of the algorithm is .

[0041] and in

[0042]

[0043]

[0044] Among them, E (*) Let f be the expectation function, G be the recognition model ResNet18, D be the domain discriminant model, and f be the domain discriminant function. i s and To identify the features output by the model, and To identify the model's prediction results, Represents the source domain D s The i-th sample in the series, Represents the target domain D t The j-th sample in the model is given by L, where L is the cross-entropy loss. The structure of the domain discriminant model D is as follows: Figure 3 As shown, it contains three fully connected layers (FC). The first two layers use the ReLU activation function, with 1024 neurons, and the dropout layer has a 0.5 puncture rate. The last fully connected layer has only one neuron, which, after passing through the Sigmoid activation function, yields the output of the domain discrimination model, i.e., the probability that the input is a domain source.

[0045] In order to verify the effect of the method of the present application, the data used in this embodiment contains 374 hand-drawing categories, and each category in the large data training set contains 100 samples. Five samples are randomly selected for each category as source domain data. The data of 10 new users are used as target domain data, and five test cases are set: (1) one sample is collected by the user for each category, (2) two samples are collected by the user for each category, (3) five samples are collected by the user for each category, (4) only two samples are collected by the user for each of the 118 categories, and (5) only five samples are collected by the user for each of the 118 categories. The above data are used as domain adaptation training data of the target domain, and in addition, five samples are collected by each user for each category as domain adaptation test data. Through test analysis, the recognition accuracy of the domain adaptation test data of the 10 users is 54.9% without domain adaptation (i.e. without user drawing style learning), and the recognition accuracy of the five test cases is (1) 96.7%, (2) 96.9%, (3) 97.0%, (4) 92.2%, and (5) 97.2% with domain adaptation (i.e. user drawing style learning) using the method of the present application. From the test results, the recognition rate after domain adaptation is still 96.7% when only one sample is used for each category, which is a substantial improvement compared to 54.9%. This proves the effectiveness of the method of the present application.

[0046] The above merely describes preferred embodiments of the present application but is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for quickly learning a user drawing style of a hand-drawing input method, characterized by: Specifically comprising the following steps: S1, obtaining the recognition network model parameters of the hand-drawing input method trained offline using a large data set, including feature extractor parameters and classifier parameters, as the basic model parameters of the hand-drawing input method; S2, randomly selecting n hand-drawing image samples for each hand-drawing category supported by the hand-drawing input method from the recognition network model training data set of the offline trained hand-drawing input method, as the drawing example samples of each hand-drawing category; The hand-drawing image samples are drawn by the developers of the hand-drawing input method or specially recruited collectors; S3, a user with a drawing style different from that of the drawing personnel in step S2, i.e., a hand-drawing input method user, draws no more than n hand-drawing image samples for each of all or part of the hand-drawing categories supported by the hand-drawing input method; S4, taking the n drawing example samples randomly selected for each hand-drawn category in step S2 as the source domain, taking the no more than n samples drawn by the user for each category in step S3 as the target domain, and updating the parameters of the basic model of the hand-drawing input method by using a supervised conditional, adversarial domain adaptation algorithm; the specific method is: taking the samples randomly selected from the big data in step S2 as the source domain data taking the samples drawn by the user as the target domain data wherein and are hand-drawn data, and are category labels, and are sample numbers; then updating the parameters of the recognition model by using a supervised domain adversarial algorithm, and the optimization target of the algorithm is and Wherein, wherein, is a desired function, is a recognition model, is a domain discriminative model, and is a feature output by the recognition model, and is a prediction result of the recognition model, denotes the i-th sample in the source domain denotes the j-th sample in the target domain denotes the i-th sample in the source domain denotes the j-th sample in the target domain denotes the i-th sample in the source domain denotes the j-th sample in the target domain is a cross-entropy loss.

2. The method of claim 1, wherein the method further comprises: determining a user drawing style based on the user drawing data; and determining a user drawing style similarity between the user drawing style and the drawing style of the drawing application. Further, in step S1, the recognition network model of the hand-drawing input method is obtained offline based on a large data set.

3. The method for rapid learning of user drawing style in a hand-drawn input method according to claim 1, characterized in that: Further, in step S3, the hand-drawing input method user needs to collect no more than n samples for each of all or part of the hand-drawing categories supported by the hand-drawing input method as data reflecting the user's personal drawing style.

4. A user drawing style quick learning system for a hand-drawing input method, characterized by: A user drawing style fast learning method for implementing any one of claims 1 to 3 of the hand-drawing input method, comprising: a hand-drawing category recognition network model for recognizing hand-drawing data drawn by a hand-drawing input method user; a hand-drawing category recommendation module for recommending standard hand-drawing icons to the user according to the recognition result of the recognition network, so that the user can select the correct standard hand-drawing category icon corresponding to the hand-drawing icon; a user hand-drawing data storage module for recording the user's hand-drawing samples and the category label selected by the user from the hand-drawing category recommended by the hand-drawing input method; an identification network model optimization module for optimizing the parameters of the hand-drawing input method identification network model.

5. A user drawing style quick learning device for a hand-drawing input method, characterized by: Comprising: a memory for storing a computer program; a processor for executing the computer program to implement the user drawing style fast learning method of any one of claims 1 to 3 of the hand-drawing input method.

6. A computer readable storage medium storing a computer program, the computer program comprising instructions which, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 5. When the computer program is executed by the processor, the user can quickly learn the user drawing style of the hand-drawing input method through the learning method of any one of claims 1 to 3.

Images