Lightweight hand-drawn figure recognition and parameter regression method and system based on multi-task learning

Abstract

The application discloses a kind of based on multi-task learning's light hand-drawn figure recognition and parameter regression method and system, including hand-drawn track point sequence is preprocessed into standardization binary image, high-level semantic feature is extracted by feature extraction network, and class and parameter are output in parallel by classification branch and multi-parameter regression branch, call special regression subnetwork refinement regression for pentagon, pentacle, finally output shape type and geometric parameter according to JSON protocol.Model uses two-stage training, mask MSE regression is done to polygon, curve, circular class in joint stage, independent standard MSE regression is done to pentagon, pentacle in special stage, combined with weight cross-entropy classification loss and multi-branch selective back propagation strategy, the application realizes light, efficient hand-drawn figure real-time identification and regular, applicable to mobile terminal image editing class application.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and deep learning technology, specifically to a lightweight hand-drawn graphic recognition and parameter regression method and system based on multi-task learning. Background Technology

[0002] In recent years, touch-based drawing and editing functions have gradually become an important part of mobile image editing software. When users doodle and draw using photo albums, note-taking, or design applications, they often want to be able to quickly generate regular geometric patterns, such as straight lines, rectangles, circles, or stars, to improve drawing efficiency and visual appeal.

[0003] Traditional hand-drawn character recognition algorithms based on trajectory recognition and geometric matching typically extract the spatial distribution and geometric features of trajectory points by sampling and analyzing the user's handwriting trajectory, thereby identifying the category of the drawn shape and correcting it into a regularized standard geometric figure. In recent years, some studies have introduced improved approaches such as stroke segmentation, curvature analysis, or deep learning models: one type of method detects the inflection points or curvature changes in the handwriting to segment irregular strokes into regular line segments or curve fragments, and then performs shape fitting and regularization based on geometric constraints; another type of method uses neural networks or metric learning models to compare the features of hand-drawn graphics with standard templates to improve the recognition ability of incomplete or noisy handwriting.

[0004] However, existing hand-drawn recognition and shape correction algorithms still have the following shortcomings: On the one hand, traditional algorithms based on geometric rules or template matching are highly dependent on the quality of the input trajectory. When the user's handwriting is shaky, missing corners, or not completely closed, the recognition accuracy will drop significantly. On the other hand, the geometric features of different shape categories (such as rectangles, pentagons, and cloud shapes) are quite different. If a unified rule or fixed threshold is used for recognition, shape confusion or ineffective differentiation may easily occur.

[0005] Therefore, how to design an intelligent hand-drawn recognition and shape correction method that can identify and automatically correct user handwriting in real time during touch drawing, and provide a better human-computer interaction experience for applications such as intelligent photo album editing, note drawing, and interaction design, has become a key technical issue in the field of mobile graphic interaction. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a lightweight hand-drawn graphic recognition and parameter regression method and system based on multi-task learning. This invention discloses a lightweight hand-drawn image recognition and parameter regression method based on multi-task learning, comprising: Step 1: Obtain the trajectory point sequence input by the user's hand drawing, preprocess the trajectory point sequence, and generate a standardized binary image; Step 2: Input the standardized binary image into a pre-trained feature extraction network to extract image features; Step 3: Based on the image features, the results are output in parallel through the classification branch and at least two parametric regression branches of the multi-task learning framework. The classification branch outputs the prediction result of the shape category to which the hand-drawn graphic belongs, and each parametric regression branch corresponds to a set of preset shape categories, which are used to output the geometric parameters of the corresponding shape category. Step 4: When the shape category output by the classification branch belongs to the preset complex shape, the corresponding dedicated regression sub-network is called to process the standardized binary image and output the geometric parameters of the corresponding complex shape. Step 5: Based on the shape category output by the classification branch, select the corresponding geometric parameters output by the parameter regression branch or the dedicated regression subnetwork as the final recognition result output.

[0007] As a further improvement of the present invention, in step 1, the trajectory point sequence is preprocessed, specifically including: The trajectory point sequence is parsed, the input point sequence is read in chronological order, abnormal strokes are filtered and the path is smoothed; The coordinates of the trajectory point sequence after trajectory parsing are normalized, and the original trajectory point coordinates are mapped to a unified canvas coordinate system to maintain the consistency of handwriting proportion and relative position. Image generation is performed based on the trajectory point sequence after coordinate normalization. The handwriting path is drawn in a pixel-level manner to generate a binary image. The image is then centered and filled with boundaries to obtain the standardized binary image.

[0008] As a further improvement of the present invention, in step 2, a lightweight convolutional neural network is used as the feature extraction network; specifically including: The top classifier of the lightweight convolutional neural network is removed and replaced with an identity mapping, so that the feature extraction network is used only to extract image features, which are high-level semantic features, to provide input for the classification branch and the parametric regression branch.

[0009] As a further improvement of the present invention, the lightweight convolutional neural network adopts the MobileNetV2 model.

[0010] As a further improvement of the present invention, in step 3, the at least two parameter regression branches include: The first parameter regression branch is used to predict the geometric parameters of the polygon family shape composed of triangles and rectangles; The second parameter regression branch is used to predict the geometric parameters of the family of curves whose shape is being predicted. The third parameter regression branch is used to predict the geometric parameters of the family of circular shapes consisting of circles, ellipses, and clouds.

[0011] As a further improvement of the present invention, in step 4, the preset complex shape includes a pentagon and a pentagram; the dedicated regression sub-network includes: The first dedicated regression subnetwork is used to refine the geometric parameters of the pentagon when the classification branch determination shape is pentagonal; The second dedicated regression subnetwork is used to refine the geometric parameters of the pentagram when the classification branch determines the shape as a pentagram.

[0012] As a further improvement of the present invention, the first dedicated regression subnetwork outputs the two-dimensional coordinates of the five vertices, and the second dedicated regression subnetwork outputs the two-dimensional coordinates of the five outer vertices and the five inner concave points.

[0013] As a further improvement of the present invention, the feature extraction network, classification branch, and all parameter regression branches are trained through a joint training phase. During the training process, for training samples of different categories with different parameter dimensions, a masking mechanism is introduced to shield invalid parameter dimensions, and the loss is calculated only for the parameter regression results of the valid dimensions. At the same time, according to the true shape category of the training sample, the loss is calculated and the parameters are updated only for the parameter regression branch corresponding to that category, and the other parameter regression branches do not participate in the parameter update of this iteration. Among them, line-type samples only participate in the training of the classification branch during the joint training phase and do not participate in the gradient update of any parameter regression branch. All the dedicated regression subnetworks are trained independently through a dedicated training phase, which is implemented separately from the joint training phase.

[0014] As a further improvement of the present invention, in the joint training phase, for training samples of different categories with different parameter dimensions, the geometric parameters of all samples are first filled with zeros to a uniform maximum dimension, and then invalid dimensions are masked by a masking mechanism.

[0015] This invention discloses a lightweight hand-drawn image recognition and parameter regression system based on multi-task learning, which is applied to the aforementioned lightweight hand-drawn image recognition and parameter regression method, including: The data preprocessing module is used to acquire the trajectory point sequence input by the user's hand drawing, perform preprocessing operations, and generate a standardized binary image; The feature extraction module, equipped with a pre-trained feature extraction network, is used to extract image features from the standardized binary image; The multi-task recognition and regression module includes a classification branch, at least two parameter regression branches, and a dedicated shape regression sub-network. It is used to output shape category prediction results and corresponding geometric parameters in parallel based on the image features, and to call the dedicated regression sub-network to output refined geometric parameters according to the classification results. The result output module is used to select the corresponding geometric parameters output by the parameter regression branch or the dedicated regression subnetwork according to the shape category output by the classification branch, encapsulate them in a structured protocol in JSON format, and output the final recognition result.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention employs the MobileNetV2 lightweight network for feature extraction, designs a two-stage training strategy, and a multi-task parallel framework to form a complete technical closed loop of "trajectory preprocessing → feature extraction → multi-task parallelism → result selection and output." This solves the technical problems of existing hand-drawn graphic recognition models, such as redundancy, large memory consumption, sluggish operation, disconnect between recognition and parameter regression, low accuracy in complex shape regression, incomplete shape coverage, and lack of targeted processing for straight lines. It achieves efficient and accurate implementation of hand-drawn graphic recognition and parameter regression, significantly improves inference efficiency, meets the real-time interaction needs of mobile devices, and can cover a variety of common hand-drawn graphics. Attached Figure Description

[0017] Figure 1 This is a flowchart of a lightweight hand-drawn image recognition and parameter regression method based on multi-task learning disclosed in one embodiment of the present invention; Figure 2 This is a result illustration of a lightweight hand-drawn graphic recognition and parameter regression method based on multi-task learning disclosed in an embodiment of the present invention. Detailed Implementation To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] The present invention will now be described in further detail with reference to the accompanying drawings: Example 1 like Figure 1-2 As shown, a lightweight hand-drawn image recognition and parameter regression method based on multi-task learning according to the present invention includes: Step 1: Data Preprocessing The system obtains a sequence of trajectory points input by the user's hand-drawn drawing, wherein the trajectory point sequence is a hand-drawn sequence of trajectory points containing the coordinate information of each point. In a preferred embodiment, the trajectory point sequence may also include timestamp information to assist in trajectory sorting and key point extraction.

[0019] The trajectory point sequence is preprocessed to generate a standardized binary image, specifically including: Trajectory analysis: Perform trajectory analysis on the sequence of trajectory points, read the input point sequence in chronological order, filter out abnormal strokes and smooth the path; Coordinate normalization: The coordinates of the trajectory point sequence after trajectory parsing are normalized, and the coordinates of the original trajectory points are mapped to a unified canvas coordinate system to maintain the consistency of handwriting proportion and relative position. Image generation: Image generation is performed based on the trajectory point sequence after coordinate normalization. The handwriting path is drawn in a pixel-level manner to generate a binary image. The image is then center-aligned and filled with boundaries to obtain a standardized binary image.

[0020] Through the above preprocessing, the system achieves automatic conversion from the original trajectory points to the standardized model input, preserving the spatial structural features of the handwritten handwriting and providing a unified and standardized input data format for the subsequent recognition module.

[0021] Step 2, Feature Extraction A standardized binary image is input into a pre-trained feature extraction network (feature extraction backbone network) to extract image features. This embodiment uses a lightweight convolutional neural network as the feature extraction network, preferably the MobileNetV2 model. Specifically, the top classifier of the MobileNetV2 model is removed and replaced with an identity mapping, so that the feature extraction network is used only for extracting image features. These image features are high-level semantic features, used to provide input for subsequent classification and parametric regression branches.

[0022] Step 3: Parallel output of multiple tasks Based on image features, the classification branch of the multi-task learning framework (corresponding to...) Figure 1 The classification head in the model and at least two parameter regression branches (corresponding to the classification head) and at least two parameter regression branches (corresponding to the classification head in the model) Figure 1 The regression head in the model outputs results in parallel, while the classification branch outputs the predicted shape category of the hand-drawn graphic. Each parameter regression branch corresponds to a set of preset shape categories, which are used to output the geometric parameters of the corresponding shape category.

[0023] In this embodiment, three regression branches with shared parameters are set up as follows: (1) First parameter regression branch: used to predict the geometric parameters of the polygon family shape composed of triangles and rectangles. Its network structure includes dropout layers, fully connected layers to 256 dimensions, ReLU activation function, and fully connected layers to 8 dimensions. The output is normalized to the [0,1] coordinate system by Sigmoid. The output is 8 floating-point numbers, representing the normalized coordinates of up to four vertices of a triangle or rectangle.

[0024] (2) Second parameter regression branch: used to predict the geometric parameters of the family of curves. Its network structure includes dropout layers, fully connected layers to 256 dimensions, ReLU activation function, and fully connected layers to 2 dimensions. The output is normalized to the [0,1] coordinate system by Sigmoid. The output is two floating-point numbers, representing the normalized coordinates of the intermediate bends of the curve.

[0025] (3) Third parameter regression branch: used to predict the geometric parameters of a family of circular shapes consisting of circles, ellipses, and clouds. Its network structure includes dropout layers, fully connected layers up to 128 dimensions, ReLU activation function, and fully connected layers up to 5 dimensions. The output is normalized to the [0,1] coordinate system by Sigmoid. The output consists of 5 floating-point numbers, representing the normalized representation of the center and radius of the circle, as well as the rotation angle parameter.

[0026] Step 4: Specialized regression for complex shapes When the shape category output by the classification branch belongs to the preset complex shape, the corresponding dedicated regression sub-network is called to process the standardized binary image and output the geometric parameters of the corresponding complex shape.

[0027] In this embodiment, the preset complex shapes include pentagons and pentagrams. The dedicated regression subnetwork includes: (1) First dedicated regression subnetwork: used to refine the geometric parameters of the pentagon when the classification branch determines the shape as pentagon. This subnetwork takes a standardized binary image as input, uses a lightweight convolutional backbone and a fully connected regression head, and outputs a 10-dimensional normalized vector, corresponding to the two-dimensional coordinates of the five vertices.

[0028] (2) Second dedicated regression subnetwork: used to refine the geometric parameters of the pentagram when the classification branch determines the shape as a pentagram. This subnetwork takes a standardized binary image as input, uses a lightweight convolutional backbone and a fully connected regression head, and outputs a 20-dimensional normalized vector, corresponding to the two-dimensional coordinates of the five outer vertices and five inner concave points.

[0029] The aforementioned dedicated regression subnetwork is independent of the multi-task learning framework and is only invoked on demand when the classification result is triggered, avoiding the need for all inputs to be processed by a complex model and improving the overall inference efficiency.

[0030] Step 5: Result Selection and Output Based on the shape category output by the classification branch, select the corresponding geometric parameters output by the parametric regression branch or the dedicated regression subnetwork as the final recognition result output.

[0031] For samples classified as "straight lines," since parametric regression is not involved, their geometric parameters are directly generated from the trajectory point information retained during the data preprocessing stage. Specifically, the two points with the earliest and latest timestamps in the original trajectory point sequence are extracted and used as the start and end coordinates of the straight line, respectively, and output along with the classification results.

[0032] The final recognition results are encapsulated in a structured JSON format to uniformly describe the types and key parameters of different geometric shapes.

[0033] Example 2 This embodiment details the training process of the feature extraction network, classification branch, all parameter regression branches, and all dedicated regression subnetworks. The training process is divided into a joint training phase and a dedicated training phase. The feature extraction network, classification branch, and all parameter regression branches are trained through the joint training phase. During training, for training samples of different categories with different parameter dimensions, a masking mechanism is introduced to block invalid parameter dimensions, calculating the loss only for the parameter regression results of valid dimensions. Simultaneously, based on the true shape category of the training sample, the loss is calculated and parameters are updated only for the parameter regression branch corresponding to that category; other parameter regression branches do not participate in the parameter updates of this iteration. Specifically, line-type samples only participate in the training of the classification branch during the joint training phase and do not participate in the gradient updates of any parameter regression branch. All dedicated regression subnetworks are trained independently through a dedicated training phase, which is implemented separately from the joint training phase. Details are as follows: 1) Dataset Construction: We constructed a self-built dataset containing eight categories of hand-drawn shapes, with 3000-10000 images per category. Each sample contains a 256×256 binary image and its corresponding JSON parameter file. The parametric annotation methods for each type of shape are as follows: Straight line sample: Extract the earliest and latest time points in the trajectory as the start and end coordinates; Curve samples: The coordinates of the starting point, bends, and ending points are manually marked; Triangle sample: Extract the three vertices of the outer contour of the trajectory using OpenCV's convex hull function; Rectangular samples: The center, width, height, and rotation angle of the rectangle are extracted by fitting using the OpenCV minimum bounding rectangle algorithm (minAreaRect); Pentagonal sample: The center point, circumcircle radius, and rotation angle are obtained by manually matching a regular pentagon template; Pentagonal sample: The center coordinates, circumcircle radius, and rotation angle are obtained through manual template matching; Circle and ellipse samples: The fitEllipse function of OpenCV was used for fitting to extract the center point, major and minor axis radii and rotation angle; Cloud samples: Approximate fitting with an circumscribed ellipse, and also use OpenCV's fitEllipse function to obtain the descriptive parameters.

[0034] 2) Training samples and symbol definitions: Let the first The input samples are ; Let its corresponding geometric category label be... ,in , Indicates the total number of graphic categories; Let its corresponding true geometric parameter vector be ; Since different graphics categories have different parameter dimensions, the parameter vectors of each category are zero-padded to a uniform length. Let the unified parameter vector be . , This represents the maximum value of the parameter dimension across all categories; Let the effective dimension mask of the parameter be... Among them, when the first When the dimension parameter is a valid parameter for that category, When the first When the dimension parameter is a zero-padded term, , This represents the maximum value of the parameter dimension across all categories; Let the logits output by the classification branch be... The corresponding number Class prediction probability is denoted as ; Let the joint training phase be... Each parameter regression branch affects the sample. The output is ; Let the routing function be , used to represent samples The parameter regression branch number that should participate in the joint training phase.

[0035] 3) Joint Training Phase The joint training phase is used to train the feature extraction network, the classification branch, and the three shared parameter regression branches (first, second, and third parameter regression branches). The classification task is trained using all eight classes of samples, while the joint training for the regression task uses all classes except pentagons and pentagrams (i.e., lines, curves, triangles, rectangles, circles / ellipses, and clouds).

[0036] 3.1) Construction of classification loss The classification task employs a cross-entropy loss function with class weights to address the problem of imbalanced sample size. ; In the formula, Indicates classification loss; Indicates the number of samples in a batch; Indicates the first The true category label of each sample; Indicates the first The category weights corresponding to the true category of each sample; Indicates sample Predicted to be its true category The probability of; where, class weight It can be set according to the reciprocal of the number of samples in the corresponding category and normalized to increase the influence weight of the fewer sample categories during the training process.

[0037] 3.2) Construction of shared regression head and parameter regression loss During the joint training phase, parameter regression is performed only on the samples participating in the parameter regression branch training. Let the samples... The corresponding parameter regression branch is determined by the routing function. If the decision is made, then its prediction parameter vector is denoted as... Since different categories have different parameter dimensions, a mask-constrained mean squared error loss function is used: ; In the formula, Indicates sample Parameter regression loss during the joint training phase; Indicates the first The first sample Validity mask for dimensional parameters; This indicates the first branch output of the regression branch corresponding to the parameter. Dimensional prediction parameters; The first true parameter vector after uniform length is represented by the second true parameter vector. dimension; To prevent numerical stability constants with a denominator of zero.

[0038] The mask normalization method described above can shield the interference of zero-padding parameter dimension on loss calculation and reduce the difference in loss scale caused by different parameter dimensions for different image categories, thereby improving the stability of multi-class joint training.

[0039] For consistent description, routing functions The routing can be defined based on the family of shapes the samples belong to. When a sample belongs to a triangle or rectangle, it is routed to the first parametric regression branch of the polygon family; when a sample belongs to a curve, it is routed to the second parametric regression branch of the curve family; when a sample belongs to a circle (ellipse) or a cloud, it is routed to the third parametric regression branch of the circle family. Pentagons, pentagrams, and lines do not participate in the joint stage parametric regression.

[0040] 3.3) Joint training objective function Let the sample set participating in the joint phase parameter regression be... Then the parameter regression loss of the joint phase is: ; In the formula, This indicates the number of samples participating in the joint phase parameter regression.

[0041] Therefore, the total loss function during the joint training phase can be expressed as: ; In the formula, This indicates the total losses during the joint training phase; and These are the weighting coefficients for classification loss and regression loss.

[0042] During backpropagation, all samples participate in the parameter updates of the classification branch; they belong only to the joint stage regression set. Only a subset of samples participate in the parameter updates of the corresponding shared regression branch, while the remaining branches do not participate in gradient updates in the current iteration. This training method can reduce gradient interference between different geometric categories and improve the convergence efficiency of the model under mixed batch training conditions.

[0043] 4) Specialized training phase The dedicated training phase focuses on the geometric features of pentagons and pentagrams, training the first and second dedicated regression sub-networks respectively. The geometric constraints of these two shapes (vertex order, alternating inner and outer points, etc.) differ significantly from other shapes; therefore, an independent end-to-end training method is adopted, without sharing network parameters with the joint training phase, and both use the standard MSE loss as the optimization objective, without the need for masking.

[0044] 4.1) Specialized training on pentagons The pentagonal-specific regression subnetwork outputs the two-dimensional coordinates of the five vertices, for a total of 10 regression parameters. The loss function is: ; In the formula, This represents the pentagonal specialized regression loss; The output of the pentagonal model represents the first... One prediction parameter; The first pentagon in the true parameter vector represents the pentagonal vector. The pentagonal parameter vector is obtained by concatenating the two-dimensional coordinates of the five vertices in a fixed order.

[0045] 4.2) Special training on pentagrams The pentagram-specific regression subnetwork outputs the two-dimensional coordinates of five outer vertices and five inner concave points, totaling 20 regression parameters. The loss function is: ; In the formula, This indicates that the pentagram represents a specific regression loss; This represents the output of the pentagram model. One prediction parameter; The first element in the true parameter vector of the pentagram represents the... The pentagram parameter vector is obtained by concatenating the two-dimensional coordinates of the five outer vertices and five inner concave points in a fixed order.

[0046] 5) Training Process First, the feature extraction network, its classification branch, and all parametric regression branches are jointly trained to complete feature extraction and geometric parameter regression for most categories. Then, dedicated regression sub-networks are trained separately for pentagons and pentagrams. During the inference phase, the input image is first processed by the feature extraction network to determine its category. When the classification result is pentagon or pentagram, the corresponding dedicated regression sub-network is called to output refined geometric parameters. When the classification result is another jointly trained category, the corresponding parametric regression branch directly outputs the parameter results. This design balances a unified training framework with the ability to model complex categories with precision.

[0047] 6) Overall training objectives The overall training objective is comprised of the losses from the joint training phase and the specialized training phase. The overall system loss function can be expressed as: ; In the formula, Indicates the overall system loss; This indicates the total losses during the joint training phase; This represents the pentagonal specialized regression loss; This indicates that the pentagram represents a specific regression loss; and These represent the weighting coefficients for the training losses specifically for pentagons and pentagrams, respectively.

[0048] In this embodiment, if the joint training phase and the dedicated training phase are implemented separately and independently in actual training, the overall training objective can be understood as the comprehensive optimization objective in the entire model training process; if a unified scheduling method is used for training, the above loss terms can be used to jointly constitute the system-level total loss function according to the set weights, which is used to measure the overall performance of the model in classification tasks and various parameter regression tasks.

[0049] In this embodiment, a lightweight convolutional neural network is used as the backbone feature extractor. The preferred approach is to use MobileNetV2 as the feature extraction network, which, compared to traditional large-scale convolutional networks, significantly reduces the number of model parameters and computational cost while maintaining high feature extraction capabilities. A structural design combining the feature extraction network with multi-task branches is employed. That is, the same feature extraction network is used for multiple types of hand-drawn graphics, with only a classification branch and at least two parametric regression branches added at higher levels, avoiding the need to train a complete network separately for each type of graphics, thereby reducing model redundancy. In this embodiment, only a small number of parameter regression branches are set during the joint training phase. For graphic categories with similar geometric parameter forms, a shared parameter regression branch is used for unified parameter prediction, which reduces the number of branches and deployment complexity.

[0050] In this embodiment, complex categories employ a dedicated sub-model mechanism that is invoked on demand. Only when the classification result is a pentagon or a pentagram is the corresponding dedicated regression sub-network invoked for refined parameter regression, thus avoiding all input samples going through the complex model and improving overall inference efficiency.

[0051] In this embodiment, the input image uses a standardized resolution of 256×256. This input size reduces the amount of forward computation while ensuring the complete representation of the hand-drawn shape outline information.

[0052] Example 3 This embodiment provides a lightweight hand-drawn image recognition and parameter regression system based on multi-task learning, applied to the method described in Embodiment 1, and includes the following modules: (1) Data preprocessing module This tool is used to acquire a sequence of trajectory points input by the user, perform preprocessing operations, and generate a standardized binary image. Specific functions include trajectory parsing, coordinate normalization, image generation, and alignment / filling, as detailed in step 1 of Example 1.

[0053] (2) Feature extraction module It incorporates a pre-trained feature extraction network for extracting image features from standardized binary images. This feature extraction network is a lightweight convolutional neural network, preferably the MobileNetV2 model, whose top classifier has been removed and is used only for extracting high-level semantic features.

[0054] (3) Multi-task recognition and regression module It includes a classification branch, at least two parameter regression branches, and a dedicated shape regression subnetwork, which is used to output shape category prediction results and corresponding geometric parameters in parallel based on the image features, and to call the dedicated regression subnetwork to output refined geometric parameters according to the classification results.

[0055] Classification branch: Fully connected layer, outputs the probability of each class; Three shared parameter regression branches: the structure is as described in step 3 of Example 1; Two dedicated regression subnetworks: the structure is as described in step 4 of Example 1.

[0056] (4) Result Output Module Based on the shape category output by the classification branch, the system selects the corresponding geometric parameters output by the parametric regression branch or a dedicated regression subnetwork, encapsulates them according to a structured protocol in JSON format, and outputs the final recognition result. For line samples, the start and end coordinates are obtained from the preprocessing module.

[0057] Advantages of this invention: This invention employs the MobileNetV2 lightweight network for feature extraction, designs a two-stage training strategy, and a multi-task parallel framework to form a complete technical closed loop of "trajectory preprocessing → feature extraction → multi-task parallelism → result selection and output." This solves the technical problems of existing hand-drawn graphic recognition models, such as redundancy, large memory consumption, sluggish operation, disconnect between recognition and parameter regression, low accuracy in complex shape regression, incomplete shape coverage, and lack of targeted processing for straight lines. It achieves efficient and accurate implementation of hand-drawn graphic recognition and parameter regression, significantly improves inference efficiency, meets the real-time interaction needs of mobile devices, and can cover a variety of common hand-drawn graphics.

[0058] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A lightweight hand-drawn image recognition and parameter regression method based on multi-task learning, characterized in that, include: Step 1: Obtain the trajectory point sequence input by the user's hand drawing, preprocess the trajectory point sequence, and generate a standardized binary image; Step 2: Input the standardized binary image into a pre-trained feature extraction network to extract image features; Step 3: Based on the image features, the results are output in parallel through the classification branch and at least two parametric regression branches of the multi-task learning framework. The classification branch outputs the prediction result of the shape category to which the hand-drawn graphic belongs, and each parametric regression branch corresponds to a set of preset shape categories, which are used to output the geometric parameters of the corresponding shape category. Step 4: When the shape category output by the classification branch belongs to the preset complex shape, the corresponding dedicated regression sub-network is called to process the standardized binary image and output the geometric parameters of the corresponding complex shape. Step 5: Based on the shape category output by the classification branch, select the corresponding geometric parameters output by the parameter regression branch or the dedicated regression subnetwork as the final recognition result output.

2. The lightweight hand-drawn graphic recognition and parameter regression method according to claim 1, characterized in that, In step 1, the trajectory point sequence is preprocessed, specifically including: The trajectory point sequence is parsed, the input point sequence is read in chronological order, abnormal strokes are filtered and the path is smoothed; The coordinates of the trajectory point sequence after trajectory parsing are normalized, and the original trajectory point coordinates are mapped to a unified canvas coordinate system to maintain the consistency of handwriting proportion and relative position. Image generation is performed based on the trajectory point sequence after coordinate normalization. The handwriting path is drawn in a pixel-level manner to generate a binary image. The image is then centered and filled with boundaries to obtain the standardized binary image.

3. The lightweight hand-drawn graphic recognition and parameter regression method according to claim 1, characterized in that, In step 2, a lightweight convolutional neural network is used as the feature extraction network; specifically, it includes: The top classifier of the lightweight convolutional neural network is removed and replaced with an identity mapping, so that the feature extraction network is used only to extract image features, which are high-level semantic features, to provide input for the classification branch and the parametric regression branch.

4. The lightweight hand-drawn graphic recognition and parameter regression method according to claim 3, characterized in that, The lightweight convolutional neural network uses the MobileNetV2 model.

5. The lightweight hand-drawn graphic recognition and parameter regression method according to claim 1, characterized in that, In step 3, at least two parameter regression branches include: The first parameter regression branch is used to predict the geometric parameters of the polygon family shape composed of triangles and rectangles; The second parameter regression branch is used to predict the geometric parameters of the family of curves whose shape is being predicted. The third parameter regression branch is used to predict the geometric parameters of the family of circular shapes consisting of circles, ellipses, and clouds.

6. The lightweight hand-drawn graphic recognition and parameter regression method according to claim 1, characterized in that, In step 4, the preset complex shapes include pentagons and pentagrams; the dedicated regression sub-network includes: The first dedicated regression subnetwork is used to refine the geometric parameters of the pentagon when the classification branch determination shape is pentagonal; The second dedicated regression subnetwork is used to refine the geometric parameters of the pentagram when the classification branch determines the shape as a pentagram.

7. The lightweight hand-drawn graphic recognition and parameter regression method according to claim 6, characterized in that, The first dedicated regression subnetwork outputs the two-dimensional coordinates of the five vertices, and the second dedicated regression subnetwork outputs the two-dimensional coordinates of the five outer vertices and the five inner concave points.

8. The lightweight hand-drawn graphic recognition and parameter regression method according to claim 1, characterized in that, The feature extraction network, classification branch, and all parameter regression branches are trained through a joint training phase. During training, a masking mechanism is introduced for training samples of different categories with different parameter dimensions to shield invalid parameter dimensions, and the loss is calculated only for the parameter regression results of the valid dimensions. At the same time, based on the true shape category of the training sample, the loss is calculated and the parameters are updated only for the parameter regression branch corresponding to that category, and the other parameter regression branches do not participate in the parameter update of this iteration. Among them, line-type samples only participate in the training of the classification branch during the joint training phase and do not participate in the gradient update of any parameter regression branch. All the dedicated regression subnetworks are trained independently through a dedicated training phase, which is implemented separately from the joint training phase.

9. The lightweight hand-drawn graphic recognition and parameter regression method according to claim 8, characterized in that, In the joint training phase, for training samples of different categories with different parameter dimensions, the geometric parameters of all samples are first filled with zeros to a uniform maximum dimension, and then invalid dimensions are masked through a masking mechanism.

10. A lightweight hand-drawn image recognition and parameter regression system based on multi-task learning, applied to the lightweight hand-drawn image recognition and parameter regression method according to any one of claims 1-9, characterized in that, include: The data preprocessing module is used to acquire the trajectory point sequence input by the user's hand drawing, perform preprocessing operations, and generate a standardized binary image; The feature extraction module, equipped with a pre-trained feature extraction network, is used to extract image features from the standardized binary image; The multi-task recognition and regression module includes a classification branch, at least two parameter regression branches, and a dedicated shape regression sub-network. It is used to output shape category prediction results and corresponding geometric parameters in parallel based on the image features, and to call the dedicated regression sub-network to output refined geometric parameters according to the classification results. The result output module is used to select the corresponding geometric parameters output by the parameter regression branch or the dedicated regression subnetwork according to the shape category output by the classification branch, encapsulate them in a structured protocol in JSON format, and output the final recognition result.

Images