Graph type identification method and device, electronic equipment and readable storage medium

Abstract

The present disclosure relates to a pattern type recognition method and device, electronic equipment and readable storage medium. The method comprises: obtaining an initial image to be recognized, the initial image comprising a pattern to be recognized; obtaining a feature vector of the pattern to be recognized in the initial image based on a preset pattern vector extraction model; obtaining a similarity between the feature vector and each candidate feature vector in a preset database to obtain at least one target candidate feature vector; and determining a type of the pattern to be recognized according to a pattern type of the at least one target candidate feature vector. In this embodiment, the problem that an existing pattern recognition model needs to be retrained for recognition when the pattern to be recognized is a new type can be solved without retraining the model when the pattern to be recognized is a new type, which is beneficial to improving the use efficiency of pattern type recognition.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and in particular to a method, apparatus, electronic device, and readable storage medium for graphic type recognition. Background Technology

[0002] In the field of image type recognition, image recognition models, after being trained, can identify images of a target type. For example, in smart screen applications, users can draw images on the smart screen, and the image recognition model can then identify the drawn images for easy use.

[0003] However, users may create new types of graphics, such as mathematical figures, which may cause the graphic recognition model to fail to recognize them, thus reducing the user experience. Summary of the Invention

[0004] This disclosure provides a method, apparatus, electronic device, and readable storage medium for identifying graphic types to address the shortcomings of related technologies.

[0005] According to a first aspect of the present disclosure, a method for identifying graphic types is provided, the method comprising:

[0006] Obtain an initial image to be identified, the initial image including the graphic to be identified;

[0007] The feature vector of the graphic to be identified in the initial image is obtained based on a preset graphic vector extraction model;

[0008] The similarity between the feature vector and each candidate feature vector in the preset database is obtained to obtain at least one target candidate feature vector;

[0009] The type of the graphic to be identified is determined based on the graphic type of the at least one target candidate feature vector.

[0010] Optionally, the feature vector of the graphic to be identified within the initial image is obtained based on a preset graphic vector extraction model, including:

[0011] Obtain a preset graphic vector extraction model; the input data of the graphic vector extraction model is the image to be identified and the output data is the feature vector of the graphic to be identified;

[0012] The initial image is input into the graphic vector extraction model to obtain the feature vector of the graphic to be identified output by the graphic vector extraction model.

[0013] Optionally, the image vector extraction model is a network model after the fully connected layers are removed after the image recognition model has been trained.

[0014] Optionally, the training process of the image recognition model includes:

[0015] Obtain a graphics training sample set, wherein the graphics training samples in the graphics training sample set include at least one target graphic and the graphic type of the target graphic;

[0016] When the loss value is greater than the preset loss threshold, each graphic training sample is input into the graphic recognition model to obtain the recognition result;

[0017] The recognition result and the target graphic type in the graphic training sample are input into a preset loss function to obtain the loss value of the graphic recognition model;

[0018] When the loss value is less than or equal to a preset loss threshold, training of the image recognition model is stopped.

[0019] Optionally, the preset loss function includes a classification loss function, a metric loss function, and a distribution loss function.

[0020] Optionally, the recognition result and the graphic type of the target graphic in the graphic training samples are input into a preset loss function to obtain the loss value of the graphic recognition model, including:

[0021] The recognition result and the graphic type of the target graphic in the graphic training sample are input into the classification loss function to obtain the first loss value;

[0022] The recognition result and the graphic type of the target graphic in the graphic training sample are input into the loss function to obtain the second loss value;

[0023] The recognition result and the graphic type of the target graphic in the graphic training sample are input into the distribution loss function to obtain the third loss value;

[0024] The weighted sum of the first loss value, the second loss value, and the third loss value is obtained as the loss value of the image recognition model.

[0025] Optionally, the at least one target candidate feature vector is obtained by selecting at least one candidate feature vector starting from the maximum similarity according to the size relationship.

[0026] Optionally, determining the type of the graphic to be identified based on the graphic type of the at least one target candidate feature vector includes:

[0027] Obtain the frequency of occurrence of each graphic type among the graphic types of the at least one target candidate feature vector;

[0028] In response to obtaining the maximum frequency of each graphic type, the graphic type corresponding to the maximum frequency is determined as the type of the graphic to be identified.

[0029] Optionally, the method further includes:

[0030] In response to the fact that the frequency of at least two graphic types is the maximum value, the graphic type of the target candidate feature vector with the highest similarity is determined as the type of the graphic to be identified.

[0031] Optionally, the method further includes:

[0032] In response to the fact that the similarity of each candidate feature vector is less than the preset similarity threshold, a prompt message for the registration graphic is generated and displayed in the operation interface;

[0033] In response to input operations within the user interface, the associated image and graphic type of the initial image are obtained;

[0034] Obtain the feature vector of the graphic within the associated image;

[0035] Establish a matching relationship between the feature vectors of the associated images and the graphic type;

[0036] The feature vectors, graphic types, and matching relationships of the associated images are stored in the preset database.

[0037] Optionally, the method further includes:

[0038] In response to the similarity of each candidate feature vector being less than the preset similarity threshold or in response to the detection of a registration operation, a prompt message for a registration graphic is generated and displayed in the operation interface;

[0039] In response to input operations within the user interface, the associated image and graphic type of the initial image are obtained;

[0040] Obtain the feature vectors of the graphics within the associated image and the average vector of the feature vectors;

[0041] Establish a matching relationship between the average vector and the graph type;

[0042] The average vector, graph type, and matching relationship are stored in the preset database.

[0043] Optionally, the associated image is obtained in the following way:

[0044] The initial image is randomly scaled to obtain multiple scaled images;

[0045] Each scaled image is blurred and used as the associated image;

[0046] And / or,

[0047] The initial image is randomly rotated to obtain multiple rotated images;

[0048] Each rotated image is pixelated to form the associated image.

[0049] According to a second aspect of the present disclosure, a graphic type recognition device is provided, the device comprising:

[0050] An initial image acquisition module is used to acquire an initial image to be identified, wherein the initial image includes a graphic to be identified;

[0051] The feature vector acquisition module is used to acquire the feature vector of the graphic to be identified in the initial image based on a preset graphic vector extraction model.

[0052] The target vector acquisition module is used to obtain the similarity between the feature vector and each candidate feature vector in the preset database, and to obtain at least one target candidate feature vector.

[0053] The image type determination module is used to determine the type of the image to be identified based on the image type of the at least one target candidate feature vector.

[0054] Optionally, the feature vector of the graphic to be identified within the initial image is obtained based on a preset graphic vector extraction model, including:

[0055] Obtain a preset graphic vector extraction model; the input data of the graphic vector extraction model is the image to be identified and the output data is the feature vector of the graphic to be identified;

[0056] The initial image is input into the graphic vector extraction model to obtain the feature vector of the graphic to be identified output by the graphic vector extraction model.

[0057] Optionally, the image vector extraction model is a network model after the fully connected layers are removed after the image recognition model has been trained.

[0058] Optionally, it also includes a model training module for training the image recognition model, including:

[0059] The sample set acquisition submodule is used to acquire a graphics training sample set, wherein the graphics training samples in the graphics training sample set include at least one target graphic and the graphic type of the target graphic;

[0060] The recognition result acquisition submodule is used to input each graphic training sample into the graphic recognition model to obtain the recognition result when the loss value is greater than the preset loss threshold;

[0061] The loss value acquisition submodule is used to input the recognition result and the target graphic type in the graphic training sample into a preset loss function to obtain the loss value of the graphic recognition model;

[0062] The loss value judgment submodule is used to stop training the image recognition model when the loss value is less than or equal to a preset loss threshold.

[0063] Optionally, the preset loss function includes a classification loss function, a metric loss function, and a distribution loss function.

[0064] Optionally, the loss value acquisition submodule includes:

[0065] The first loss value acquisition unit is used to input the recognition result and the graphic type of the target graphic in the graphic training sample into the classification loss function to obtain the first loss value;

[0066] The second loss value acquisition unit is used to input the recognition result and the graphic type of the target graphic in the graphic training sample into the metric loss function to obtain the second loss value;

[0067] The third loss value acquisition unit is used to input the recognition result and the graphic type of the target graphic in the graphic training sample into the distribution loss function to obtain the third loss value;

[0068] The loss value acquisition unit is used to acquire the weighted sum of the first loss value, the second loss value and the third loss value as the loss value of the image recognition model.

[0069] Optionally, the at least one target candidate feature vector is obtained by selecting at least one candidate feature vector starting from the maximum similarity according to the size relationship.

[0070] Optionally, the graphics type determination module includes:

[0071] The type frequency acquisition submodule is used to acquire the frequency of occurrence of each graphic type in the graphic types of the at least one target candidate feature vector;

[0072] The graphic type recognition submodule is used to determine the graphic type corresponding to the maximum frequency of each graphic type as the type of the graphic to be recognized in response to obtaining the maximum frequency of each graphic type.

[0073] Optionally, the graphic type determination module is further configured to, in response to the acquisition that the frequencies of at least two graphic types are both at their maximum values, determine the graphic type of the target candidate feature vector with the highest similarity as the type of the graphic to be identified.

[0074] Optionally, the device further includes:

[0075] The prompt message generation module is used to generate a prompt message for the registration graphic and display it in the operation interface in response to the similarity of each candidate feature vector being less than the preset similarity threshold.

[0076] The associated image acquisition module is used to acquire the associated image and graphic type of the initial image in response to input operations within the operation interface;

[0077] A feature vector acquisition module is used to acquire feature vectors of graphics within the associated image;

[0078] A matching relationship establishment module is used to establish a matching relationship between the feature vectors of the associated images and the graphic type;

[0079] The storage module is used to store the feature vectors, graphic types, and matching relationships of the associated images into the preset database.

[0080] Optionally, the method further includes:

[0081] The prompt message generation module is used to generate a prompt message of registration graphics and display it in the operation interface in response to the similarity of each candidate feature vector being less than the preset similarity threshold or in response to the detection of a registration operation.

[0082] The associated image acquisition module is used to acquire the associated image and graphic type of the initial image in response to input operations within the operation interface;

[0083] The average vector acquisition module is used to acquire the feature vectors of the graphics within the associated image and the average vector of the feature vectors.

[0084] The matching relationship establishment module is used to establish the matching relationship between the average vector and the graphic type;

[0085] The storage module is used to store the average vector, graph type, and matching relationship into the preset database.

[0086] Optionally, the associated image acquisition module includes:

[0087] The image scaling acquisition submodule is used to acquire the size of the randomly scaled initial image to obtain multiple scaled images;

[0088] The associated image acquisition submodule is used to blur each scaled image and use it as the associated image;

[0089] And / or,

[0090] The image rotation acquisition submodule is used to randomly rotate the orientation of the initial image to obtain multiple rotated images;

[0091] The associated image acquisition submodule is used to perform mosaic processing on each rotated image to obtain the associated image.

[0092] According to a third aspect of the present disclosure, an electronic device is provided, comprising:

[0093] processor;

[0094] Memory for storing computer programs executable by the processor;

[0095] The processor is configured to execute a computer program in the memory to implement the method described in the first aspect.

[0096] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided that, when an executable computer program in the storage medium is executed by a processor, enables the implementation of the method described in the first aspect.

[0097] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0098] As can be seen from the above embodiments, the solution provided in this disclosure obtains an initial image to be identified, which includes a graphic to be identified; then, based on a preset graphic vector extraction model, the feature vector of the graphic to be identified in the initial image is obtained; subsequently, the similarity between the feature vector and each candidate feature vector in a preset database is obtained to obtain at least one target candidate feature vector; finally, the type of the graphic to be identified is determined according to the graphic type of the at least one target candidate feature vector. Thus, this embodiment determines the graphic type of the graphic to be identified by obtaining the feature vector of the graphic to be identified and combining it with similarity. This solves the problem of existing graphic recognition models needing to be retrained and re-identified when the graphic to be identified is a new type, without requiring retraining of the model. This facilitates expansion, reduces costs, and improves the efficiency of graphic type recognition.

[0099] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0100] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0101] Figure 1 This is a flowchart illustrating a graphic type recognition method according to an exemplary embodiment.

[0102] Figure 2 This is a flowchart illustrating an example of obtaining a feature vector.

[0103] Figure 3 This is a schematic diagram illustrating a feature classification according to an exemplary embodiment.

[0104] Figure 4 This is a schematic diagram illustrating another feature classification according to an exemplary embodiment.

[0105] Figure 5 This is a schematic diagram illustrating a distribution direction according to an exemplary embodiment.

[0106] Figure 6 This is a block diagram illustrating a novel graphic registration process according to an exemplary embodiment.

[0107] Figure 7 This is a block diagram illustrating a graphic type recognition device according to an exemplary embodiment. Detailed Implementation

[0108] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described below by way of example do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatus consistent with some aspects of this disclosure as detailed in the appended claims. It should be noted that, without conflict, the following embodiments and features in the implementation methods can be combined with each other.

[0109] To address the aforementioned technical problems, this disclosure provides a graphic type recognition method applicable to electronic devices, including but not limited to smartphones, tablets, smart displays, or electronic whiteboards. Figure 1 This is a flowchart illustrating a graphic type recognition method according to an exemplary embodiment.

[0110] See Figure 1 A method for identifying image types, comprising steps 11 to 14:

[0111] In step 11, an initial image to be identified is obtained, the initial image including the graphic to be identified.

[0112] In this embodiment, the processor of the electronic device can acquire an initial image to be recognized. For example, the display screen of the electronic device can display a drawing interface. The user can draw a graphic on this drawing interface, which will later be referred to as the graphic to be recognized. The display screen can acquire an image that matches the display screen and transmit it to the processor. At this time, the processor can acquire an initial image, which includes the graphic to be recognized.

[0113] In this embodiment, the graphics may include, but are not limited to, planar graphics such as straight lines, straight arrows, wavy lines, squares, rectangles, rhombuses, trapezoids, circles, ellipses, parallelograms, pentagons, and pentagrams, as well as three-dimensional graphics such as spheres, cylinders, and cubes. The graphics can be set according to the specific scenario and are not limited here.

[0114] In step 12, the feature vector of the graphic to be identified in the initial image is obtained based on a preset graphic vector extraction model.

[0115] In this embodiment, the processor of the electronic device can obtain the feature vector of the graphic to be identified in the initial image based on a preset graphic vector extraction model. See [link to relevant documentation]. Figure 2 This includes steps 21 and 22.

[0116] In step 21, the processor can acquire a preset graphic vector extraction model; the input data of the graphic vector extraction model is the image to be identified and the output data is the feature vector of the graphic to be identified.

[0117] In this step, the electronic device can store an image vector extraction model. The input data of this model is the image to be recognized, and the output data is the feature vector of the image to be recognized. In one example, this image vector extraction model is a network model after the fully connected layers have been removed from the trained image recognition model, and it includes at least one convolutional layer and at least one fully connected layer. The image recognition model can be implemented using a neural network model, such as the DenseNet model, and the choice is made according to the specific scenario; no limitation is made here.

[0118] In one possible embodiment, the process of acquiring the above-described image vector extraction model, which is also the training process of the above-described image recognition model, includes:

[0119] The processor can acquire a set of image training samples, which includes at least one target image and the image type of the target image. In one example, each type of training sample in the image training sample set may include an actually acquired image, such as a hand-drawn or printed image, and a composite image synthesized based on the actually acquired image. For example, the size of the actually acquired image may be randomly scaled to obtain multiple scaled images; each scaled image is then blurred to obtain a composite image; or the orientation of the actually acquired image may be randomly rotated to obtain multiple rotated images; each rotated image is then mosaicked to obtain a composite image. In one example, the ratio of actual image to composite image is 1:4, thereby ensuring the number of candidate feature vectors in the preset database to improve the accuracy of image recognition.

[0120] When the loss value is greater than a preset loss threshold, the processor can input each image training sample into the image recognition model to obtain a recognition result. The processor can then input the recognition result and the image type of the target image in the image training samples into a preset loss function to obtain the loss value of the image recognition model. When the loss value is less than or equal to the preset loss threshold, the processor can stop training the image recognition model.

[0121] In one example, the aforementioned preset loss function may include a classification loss function, a metric loss function, and a distribution loss function. The classification loss function uses cross-entropy loss (nn.CrossEntropyLoss) to ensure that the probability of the image type in the training samples matches the probability of that type in the image recognition model's output. Cross-entropy loss is a loss function almost universally used in classification tasks. Compared to absolute error (which directly reflects whether the network's prediction is correct—either a correct prediction results in a loss of 0, or a wrong prediction results in a loss of 1), its loss variation is smoother, reducing network fluctuations during training.

[0122] For example, in the MNIST handwritten digit recognition dataset, each image contains a digit between 0 and 9, and each image can only have one fixed label (i.e., the digit itself). The cross-entropy loss function is calculated for a single sample as follows:

[0123]

[0124] In equation (1), y i Represents the true distribution. This represents the predicted distribution, where n represents the number of categories.

[0125] In a handwritten digit recognition task, if an image is the digit "5", then the true distribution should be: [0,0,0,0,0,1,0,0,0,0], with only the label corresponding to the digit "5" being 1, and the other positions from 0 to 9 being 0.

[0126] If the network output distribution is [0.1,0.1,0,0,0,0,0.7,0,0.1,0,0], and position 5 has a higher score while other positions have lower scores, then the loss function calculated using the cross-entropy function is:

[0127] Loss=-0*log0.1-0*log0.1-0*log0-0*log0-0*log0

[0128] -1*log0.7-0*log0-0*log0.1-0*log0-0*log0≈0.3567.

[0129] If the network output distribution is [0.2,0.3,0.1,0,0,0.3,0.1,0,0,0], then the loss function is calculated as follows:

[0130] Loss=-0*log0.2-0*log0.3-0*log0.1-0*log0-0*log0

[0131] -1*log0.3-0*log0.1-0*log0-0*log0-0*log0≈1.2040.

[0132] Comparing the two scenarios, the calculated loss of 0.3567 for the first distribution is significantly lower than the loss of 1.2040 for the second distribution. This indicates that the first distribution is closer to the true distribution, which is indeed the case. For an image that is actually the number "5", the confidence score for the position of the 5 in the first network's output is 0.7, while the second output is only 0.3. This demonstrates that cross-entropy loss can effectively measure whether the distribution of the network's output closely resembles the true situation. A larger loss indicates a poorer prediction result at that moment, thus incurring a larger penalty for the network. Training the network in this way allows it to gradually improve its prediction accuracy, thereby ensuring that the network possesses basic classification capabilities.

[0133] The loss function used is triplet loss, which is employed to measure the distance between two graphics. This means that the distance between two similar graphics is smaller, while the distance between dissimilar graphics is larger. Metric learning can classify graphics by calculating the feature distances between different graphics. This requires the network to extract features that are as discriminative as possible; that is, the features should not only be able to classify correctly, but also ensure that the intra-class distances of graphics within the same class are as close as possible, and the inter-class distances of graphics from different classes are as large as possible.

[0134] For example, see Figure 3 , Figure 3 Solid and hollow dots are equivalent to features of this image, and can be separated from hollow dots by the dashed line in the middle. See also Figure 4 The dotted line in the middle can separate the solid and hollow dots; a comparison shows that... Figure 4 The distance between the solid point and the hollow point is greater than Figure 3 The distance between the solid point and the hollow point.

[0135] In this example, we hope the model will learn the following features: Figure 4 This means that the intra-class distance should be small, while the inter-class distance should be large. Therefore, in one example, the triplet loss function is used as the metric for loss.

[0136] In this embodiment, considering that each graphic can be projected in six directions (horizontal, vertical, etc.), the differences in the distribution in these six directions are statistically analyzed to determine the degree of difference in the skeleton structure between the actual and generated graphics. Since different graphics have different skeleton structures and can be projected in different directions, resulting in different distributions, these distributions can also be used to determine whether the model's predicted graphics match the real graphics. When using distribution loss, the graphic to be predicted and the standard graphic need to be normalized. If the network predicts the current graphic is a square, it will calculate the distribution loss between the input graphic and the square, i.e., calculate the structural similarity to determine the correctness of the prediction. If the prediction is correct, the cumulative distribution loss in the six directions is relatively small; if the prediction is incorrect, the cumulative loss will be relatively large.

[0137] The distribution loss function is implemented using equation (1) to obtain the distribution loss of the graphic at the edges in various directions.

[0138]

[0139] Equation (1), where P represents the number of distributions, see [reference]. Figure 5 P represents the number of 6 directions; Y0 and Y1 represent the two figures to be compared, and L represents the KL dispersion.

[0140] The preset loss function in this embodiment can include a classification loss function, a metric loss function, and a distribution loss function, which can make the model have strong classification ability and good inter-class discrimination, thus making it more robust and accurate when directly using feature vectors to calculate distances in the future.

[0141] During each training iteration, the processor can input the recognition result and the target graphic type in the graphic training samples into a classification loss function to obtain a first loss value; the processor can input the recognition result and the target graphic type in the graphic training samples into a metric loss function to obtain a second loss value; the processor can input the recognition result and the target graphic type in the graphic training samples into a distribution loss function to obtain a third loss value; the processor can obtain a weighted sum of the first loss value, the second loss value, and the third loss value as the loss value of the graphic recognition model.

[0142] The weights of each loss function can be obtained through the following steps:

[0143] First, a subset of training samples is extracted from the image training sample set as a validation set. Weights are then assigned to the classification loss function, metric loss function, and distribution loss function, resulting in multiple sets of weights. Based on these weights, the image recognition model is trained in parallel. After the image recognition model corresponding to each set of weights has been trained, the set of weights corresponding to the image recognition model with the highest recognition accuracy is determined as the weight values ​​for the classification loss function, metric loss function, and distribution loss function. Then, the above training process is repeated.

[0144] Taking the image recognition model DenseNet-264 as an example, the label format of each image training sample is as follows:

[0145] { 0:

[0147] {'000000000139.jpg':['. / data / aaa / train / 1 / 000000000139.jpg',0],

[0148] '000000000285.jpg':['. / data / aaa / train / 1 / 000000000285.jpg',0],

[0149] '000000000785.jpg':['. / data / aaa / train / 1 / 000000000785.jpg',0],

[0150] '000000000802.jpg':['. / data / aaa / train / 1 / 000000000802.jpg',0]...}, 1:

[0152] {'000000001490.jpg':['. / data / aaa / train / 2 / 000000001490.jpg',1],

[0153] '000000001503.jpg':['. / data / aaa / train / 2 / 000000001503.jpg',1]...} ...

[0155] }

[0156] Furthermore, each batch of triplet samples includes three data points (two positive samples and one negative sample), as shown below:

[0157] [['. / data / aaa / train / 2 / 000000001503.jpg',1],

[0158] ['. / data / aaa / train / 2 / 000000000285.jpg',1]

[0159] ['. / data / aaa / train / 1 / 000000000139.jpg',0]].

[0160] During training, the processor can increase the learning rate of the image recognition model from 0 to the initial learning rate, and then gradually decrease it in a cosine manner, thereby improving training efficiency.

[0161] The above method can be used to train the image recognition model. Then, by removing the fully connected layers of the image recognition model, the image vector extraction model can be obtained.

[0162] In this step, the processor can call the above-mentioned graphics vector extraction model.

[0163] In step 22, the processor can input the initial image into the graphic vector extraction model to obtain the feature vector of the graphic to be identified output by the graphic vector extraction model.

[0164] In step 13, the similarity between the feature vector and each candidate feature vector in the preset database is obtained to obtain at least one target candidate feature vector; the at least one target candidate feature vector is obtained by selecting at least one candidate feature vector starting from the maximum similarity according to the size relationship.

[0165] In one embodiment, the electronic device stores a preset database, which includes feature vectors of several graphics, graphic types, and matching relationships; in other words, each feature vector corresponds to a graphic type. Thus, the processor can obtain the similarity between the feature vector of the graphic to be identified and each candidate feature vector in the preset database. Then, the processor can sort the similarity vectors according to their magnitude to obtain at least one target candidate feature vector. That is, at least one target candidate feature vector is obtained by selecting at least one candidate feature vector starting from the highest similarity, ensuring that a target candidate feature vector is obtained in each comparison process, facilitating subsequent type identification to obtain recognition results.

[0166] In another embodiment, the processor can also sort the similarity according to the size relationship to obtain at least one target candidate feature vector with the similarity greater than or equal to the similarity threshold. That is, at least one target candidate feature vector is obtained by selecting at least one candidate feature vector from the largest similarity according to the size relationship, which is beneficial to improving the accuracy of type recognition.

[0167] In step 14, the type of the graphic to be identified is determined based on the graphic type of the at least one target candidate feature vector.

[0168] In this step, considering that the graphic types of the aforementioned target candidate feature vectors may be the same, the processor also counts the frequency of each graphic type among at least one target candidate feature vector, and takes the graphic type corresponding to the maximum frequency as the type of the graphic to be identified. For example, if there are 5 target candidate feature vectors, and the frequency of the rectangle type is 3 and the frequency of the pentagon type is 2, then the graphic type of the graphic to be identified is determined to be the rectangle type. In another example, in response to obtaining at least two graphic types with maximum frequencies, the processor also determines the image type of the target candidate feature vector with the highest similarity as the type of the graphic to be identified. For example, if there are 5 target candidate feature vectors, and the frequency of the rectangle type is 2, the frequency of the pentagon type is 2, and the frequency of the triangle type is 1, then the image type of the target candidate feature vector with the highest similarity (e.g., 0.85) is found to be the type of the graphic to be identified.

[0169] In one embodiment, when the similarity is less than a preset similarity threshold, it indicates that the recognition accuracy of the graphic to be recognized is low or that the graphic to be recognized cannot be recognized, or that the graphic to be recognized is a new graphic. The processor can register the new graphic, at which time it can generate a registration prompt message and display it in the operation interface, such as "The graphic to be recognized is a new graphic, please follow the guidance information to register the graphic." The user can input the new graphic and the graphic type of the new graphic in the operation interface, i.e., the initial image in step 11.

[0170] The processor can acquire associated images of the initial image. For example, the processor can randomly scale the size of the initial image to obtain multiple scaled images; blur each scaled image to obtain the associated images. Alternatively, the processor can randomly rotate the orientation of the initial image to obtain multiple rotated images; mosaic each rotated image to obtain the associated images. In this way, the processor can acquire several associated images of the initial image.

[0171] The processor can obtain the feature vectors of graphics within the associated image; see [link / reference]. Figure 6 The processor then establishes a matching relationship between the feature vectors of the associated images and the graphic types. Finally, the processor stores the feature vectors, graphic types, and matching relationships of the associated images into the preset database, thereby completing the registration of the new graphic. In this way, the processor can obtain the graphic type of the new graphic during subsequent recognition processes.

[0172] In one embodiment, a user may need to register a new graphic to increase the number of graphic types in the preset database. In this case, when the electronic device detects a registration operation for a new graphic, it can display a prompt message on the display interface indicating the generation of the new graphic, such as "Please register a new graphic according to the prompt," "Please enter a new graphic," or "Please enter a graphic type." For example, first, a first prompt signal can be displayed: "Please enter a new graphic." The user can hold the new graphic and place it in the preview scene of the electronic device's camera. At this time, the electronic device can call the camera to capture the preview scene; or the user can draw the new graphic on the display interface, and the electronic device can capture an image of the display interface and detect the new graphic in the screenshot; or the user can move an image containing the new graphic from a specified location (local storage, external mobile storage, or the cloud) to the display interface, and the electronic device can obtain the image and then capture the new graphic. In this way, the electronic device can obtain the initial image in multiple ways. Then, "Please enter a graphic type" is displayed, and the electronic device can obtain the graphic type of the new graphic in the initial image, i.e., the initial image in step 11.

[0173] Then, the processor can acquire associated images of the initial image. For example, the processor can randomly scale the size of the initial image to obtain multiple scaled images; blur each scaled image to obtain the associated images. Alternatively, the processor can randomly rotate the orientation of the initial image to obtain multiple rotated images; mosaic each rotated image to obtain the associated images. In this way, the processor can acquire several associated images of the initial image.

[0174] Finally, the processor can obtain the feature vectors of the graphics within the associated image, see [link to related image]. Figure 6 The processor then establishes a matching relationship between the feature vectors of the associated images and the graphic types. Finally, the processor stores the feature vectors, graphic types, and matching relationships of the associated images into the preset database, thereby completing the registration of the new graphic. In this way, the processor can obtain the graphic type of the new graphic during subsequent recognition processes.

[0175] In the above embodiments, when registering a new graphic, associated images of the initial image are obtained, and feature vectors of each associated image are acquired. This enriches the number of candidate feature vectors in the preset database and is applicable to various graphics of the same type to be recognized. In one embodiment, the electronic device can obtain the average vector of feature vectors of the same graphic type and use this average vector as the feature vector of that graphic type. The average vector refers to the vector obtained by averaging the data at the same position in the feature vector and replacing the data at that position with the average value. In this example, each graphic type corresponds to one feature vector, thereby reducing the number of candidate feature vectors in the preset database and improving the efficiency and accuracy of graphic recognition.

[0176] Thus, in this embodiment, the graphic type of the graphic to be identified is determined by obtaining the feature vector of the graphic to be identified and combining it with the similarity. This solves the problem that the existing graphic recognition model needs to be retrained and re-identified when the graphic to be identified is a new type, without the need to retrain the model. This is beneficial to improving the efficiency of graphic type recognition.

[0177] Based on the graphic type recognition method provided in this disclosure, this embodiment also provides a graphic type recognition device, see [link to relevant documentation]. Figure 7 The device includes:

[0178] The initial image acquisition module 71 is used to acquire an initial image to be identified, wherein the initial image includes a graphic to be identified;

[0179] The feature vector acquisition module 72 is used to acquire the feature vector of the graphic to be identified in the initial image based on a preset graphic vector extraction model.

[0180] The target vector acquisition module 73 is used to acquire the similarity between the feature vector and each candidate feature vector in the preset database, and to obtain at least one target candidate feature vector.

[0181] The image type determination module is used to determine the type of the image to be identified based on the image type of the at least one target candidate feature vector.

[0182] In one embodiment, obtaining the feature vector of the graphic to be identified within the initial image based on a preset graphic vector extraction model includes:

[0183] Obtain a preset graphic vector extraction model; the input data of the graphic vector extraction model is the image to be identified and the output data is the feature vector of the graphic to be identified;

[0184] The initial image is input into the graphic vector extraction model to obtain the feature vector of the graphic to be identified output by the graphic vector extraction model.

[0185] In one embodiment, the image vector extraction model is a network model after the fully connected layers are removed after the image recognition model has been trained.

[0186] In one embodiment, the system further includes a model training module for training the image recognition model, comprising:

[0187] The sample set acquisition submodule is used to acquire a graphics training sample set, wherein the graphics training samples in the graphics training sample set include at least one target graphic and the graphic type of the target graphic;

[0188] The recognition result acquisition submodule is used to input each graphic training sample into the graphic recognition model to obtain the recognition result when the loss value is greater than the preset loss threshold;

[0189] The loss value acquisition submodule is used to input the recognition result and the target graphic type in the graphic training sample into a preset loss function to obtain the loss value of the graphic recognition model;

[0190] The loss value judgment submodule is used to stop training the image recognition model when the loss value is less than or equal to a preset loss threshold.

[0191] In one embodiment, the preset loss function includes a classification loss function, a metric loss function, and a distribution loss function.

[0192] In one embodiment, the loss value acquisition submodule includes:

[0193] The first loss value acquisition unit is used to input the recognition result and the graphic type of the target graphic in the graphic training sample into the classification loss function to obtain the first loss value;

[0194] The second loss value acquisition unit is used to input the recognition result and the graphic type of the target graphic in the graphic training sample into the metric loss function to obtain the second loss value;

[0195] The third loss value acquisition unit is used to input the recognition result and the graphic type of the target graphic in the graphic training sample into the distribution loss function to obtain the third loss value;

[0196] The loss value acquisition unit is used to acquire the weighted sum of the first loss value, the second loss value and the third loss value as the loss value of the image recognition model.

[0197] In one embodiment, the at least one target candidate feature vector is obtained by selecting at least one candidate feature vector starting from the maximum similarity according to the size relationship.

[0198] In one embodiment, the graphics type determination module includes:

[0199] The type frequency acquisition submodule is used to acquire the frequency of occurrence of each graphic type in the graphic types of the at least one target candidate feature vector;

[0200] The graphic type recognition submodule is used to determine the graphic type corresponding to the maximum frequency of each graphic type as the type of the graphic to be recognized in response to obtaining the maximum frequency of each graphic type.

[0201] In one embodiment, the graphic type determination module is further configured to determine the graphic type of the target candidate feature vector with the highest similarity as the type of the graphic to be identified in response to the acquisition that the frequencies of at least two graphic types are both at the maximum value.

[0202] In one embodiment, the graphics type determination module includes:

[0203] The graphic type determination unit is used to determine the graphic type of the target candidate feature vector with the highest similarity as the type of the graphic to be identified.

[0204] In one embodiment, the device further includes:

[0205] The prompt message generation module is used to generate a prompt message of registration graphics and display it in the operation interface in response to the similarity of each candidate feature vector being less than the preset similarity threshold or in response to the detection of a registration operation.

[0206] The associated image acquisition module is used to acquire the associated image and graphic type of the initial image in response to input operations within the operation interface;

[0207] A feature vector acquisition module is used to acquire feature vectors of graphics within the associated image;

[0208] A matching relationship establishment module is used to establish a matching relationship between the feature vectors of the associated images and the graphic type;

[0209] The storage module is used to store the feature vectors, graphic types, and matching relationships of the associated images into the preset database.

[0210] In one embodiment, the device further includes:

[0211] The prompt message generation module is used to generate a prompt message of registration graphics and display it in the operation interface in response to the similarity of each candidate feature vector being less than the preset similarity threshold or in response to the detection of a registration operation.

[0212] The associated image acquisition module is used to acquire the associated image and graphic type of the initial image in response to input operations within the operation interface;

[0213] The average vector acquisition module is used to acquire the feature vectors of the graphics within the associated image and the average vector of the feature vectors.

[0214] The matching relationship establishment module is used to establish the matching relationship between the average vector and the graphic type;

[0215] The storage module is used to store the average vector, graph type, and matching relationship into the preset database.

[0216] In one embodiment, the associated image acquisition module includes:

[0217] The image scaling acquisition submodule is used to acquire the size of the randomly scaled initial image to obtain multiple scaled images;

[0218] The associated image acquisition submodule is used to blur each scaled image and use it as the associated image;

[0219] And / or,

[0220] The image rotation acquisition submodule is used to randomly rotate the orientation of the initial image to obtain multiple rotated images;

[0221] The associated image acquisition submodule is used to perform mosaic processing on each rotated image to obtain the associated image.

[0222] It should be noted that the device embodiment shown in this embodiment matches the content of the above method embodiment, and the content of the above method embodiment can be referred to, and will not be repeated here.

[0223] In an exemplary embodiment, an electronic device is also provided, comprising:

[0224] Display screen;

[0225] processor;

[0226] Memory for storing computer programs executable by the processor;

[0227] The processor is configured to execute a computer program in the memory to implement the method described above.

[0228] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including an executable computer program that can be executed by a processor to implement the method as described in the above embodiments. The readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, or optical data storage device, etc.

[0229] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This disclosure is intended to cover any variations, uses, or adaptations that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0230] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A method for recognizing graphic types, characterized in that, The method includes: Obtain an initial image to be identified, the initial image including the graphic to be identified; The feature vector of the graphic to be identified in the initial image is obtained based on a preset graphic vector extraction model; The similarity between the feature vector and each candidate feature vector in the preset database is obtained to obtain at least one target candidate feature vector; The type of the graphic to be identified is determined based on the graphic type of the at least one target candidate feature vector; Determining the type of the graphic to be identified based on the graphic type of the at least one target candidate feature vector includes: The frequency of occurrence of each graphic type in the graphic types of the at least one target candidate feature vector is obtained; the at least one target candidate feature vector is obtained by selecting at least one candidate feature vector from the maximum similarity according to the size relationship; In response to obtaining the maximum frequency of each graphic type, the graphic type corresponding to the maximum frequency is determined as the type of the graphic to be identified.

2. The method according to claim 1, characterized in that, Based on a preset graphic vector extraction model, feature vectors of the graphic to be identified within the initial image are obtained, including: Obtain a preset graphic vector extraction model; the input data of the graphic vector extraction model is the image to be identified and the output data is the feature vector of the graphic to be identified; The initial image is input into the graphic vector extraction model to obtain the feature vector of the graphic to be identified output by the graphic vector extraction model.

3. The method according to claim 2, characterized in that, The graphic vector extraction model is a network model after the fully connected layers are removed after the graphic recognition model has been trained.

4. The method according to claim 3, characterized in that, The training process of the image recognition model includes: Obtain a graphics training sample set, wherein the graphics training samples in the graphics training sample set include at least one target graphic and the graphic type of the target graphic; Each graphic training sample is input into the graphic recognition model to obtain the recognition result; The recognition result and the target graphic type in the graphic training sample are input into a preset loss function to obtain the loss value of the graphic recognition model; When the loss value is less than or equal to a preset loss threshold, training of the image recognition model is stopped.

5. The method according to claim 4, characterized in that, The preset loss functions include classification loss function, metric loss function and distribution loss function.

6. The method according to claim 5, characterized in that, The recognition result and the target graphic type in the graphic training samples are input into a preset loss function to obtain the loss value of the graphic recognition model, including: The recognition result and the graphic type of the target graphic in the graphic training sample are input into the classification loss function to obtain the first loss value; The recognition result and the graphic type of the target graphic in the graphic training sample are input into the loss function to obtain the second loss value; The recognition result and the graphic type of the target graphic in the graphic training sample are input into the distribution loss function to obtain the third loss value; The weighted sum of the first loss value, the second loss value, and the third loss value is obtained as the loss value of the image recognition model.

7. The method according to claim 1, characterized in that, The method further includes: In response to the fact that the frequency of at least two graphic types is the maximum value, the graphic type of the target candidate feature vector with the highest similarity is determined as the type of the graphic to be identified.

8. The method according to claim 1, characterized in that, Determining the type of the graphic to be identified based on the graphic type of the at least one target candidate feature vector includes: The graphic type of the target candidate feature vector with the highest similarity is determined as the type of the graphic to be identified.

9. The method according to claim 1, characterized in that, The method further includes: In response to the similarity of each candidate feature vector being less than a preset similarity threshold or in response to the detection of a registration operation, a prompt message for a registration graphic is generated and displayed in the operation interface; In response to input operations within the user interface, the associated image and graphic type of the initial image are obtained; Obtain the feature vector of the graphic within the associated image; Establish a matching relationship between the feature vectors of the associated images and the graphic type; The feature vectors, graphic types, and matching relationships of the associated images are stored in the preset database.

10. The method according to claim 1, characterized in that, The method further includes: In response to the similarity of each candidate feature vector being less than a preset similarity threshold or in response to the detection of a registration operation, a prompt message for a registration graphic is generated and displayed in the operation interface; In response to input operations within the user interface, the associated image and graphic type of the initial image are obtained; Obtain the feature vectors of the graphics within the associated image and the average vector of the feature vectors; Establish a matching relationship between the average vector and the graph type; The average vector, graph type, and matching relationship are stored in the preset database.

11. The method according to claim 9 or 10, characterized in that, The associated image is obtained through the following method: The initial image is randomly scaled to obtain multiple scaled images; Each scaled image is blurred and used as the associated image; And / or, The initial image is randomly rotated to obtain multiple rotated images; Each rotated image is pixelated to form the associated image.

12. A graphic type recognition device, characterized in that, The device includes: An initial image acquisition module is used to acquire an initial image to be identified, wherein the initial image includes a graphic to be identified; The feature vector acquisition module is used to acquire the feature vector of the graphic to be identified in the initial image based on a preset graphic vector extraction model. The target vector acquisition module is used to obtain the similarity between the feature vector and each candidate feature vector in the preset database, and to obtain at least one target candidate feature vector. The image type determination module is used to determine the type of the image to be identified based on the image type of the at least one target candidate feature vector; The graphics type determination module includes: The type frequency acquisition submodule is used to acquire the frequency of each graphic type in the graphic types of the at least one target candidate feature vector; the at least one target candidate feature vector is obtained by selecting at least one candidate feature vector from the largest similarity according to the size relationship. The graphic type recognition submodule is used to determine the graphic type corresponding to the maximum frequency of each graphic type as the type of the graphic to be recognized in response to obtaining the maximum frequency of each graphic type.

13. An electronic device, characterized in that, include: processor; Memory for storing computer programs executable by the processor; The processor is configured to execute a computer program in the memory to implement the method as described in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, When the executable computer program in the storage medium is executed by a processor, it can implement the method as described in any one of claims 1 to 11.

Images