Image recognition method and device, storage medium and electronic device

Abstract

The embodiments of the present disclosure provide an image recognition method and device, a storage medium and an electronic device, and relate to the field of image recognition technology. The method comprises: obtaining the image segmentation features to be recognized of the image to be recognized; using an image recognition model to recognize the image segmentation features to be recognized, so as to determine the target recognition label of the image to be recognized; wherein the image recognition model is obtained by training the image segmentation features and image classification features extracted from the image training data. The image recognition method provided by the present disclosure can improve the accuracy of language recognition in images and provide a basis for OCR text recognition.

Description

Background Art

[0002] This section is intended to provide a background or context to the embodiments of the disclosure that are recited in the claims. No description herein is admitted to be prior art by inclusion in this section.

[0003] Optical Character Recognition (OCR) is the process of analyzing and identifying image files of text materials to obtain text and layout information.

[0004] In real life, there are multiple language environments, such as Chinese, Tibetan, Uyghur, English, Japanese, German, Korean, etc. These complex multilingual scenarios directly affect the accuracy of OCR text recognition results. Therefore, language recognition is the basis of OCR recognition.

[0005] However, the existing language recognition network has poor recognition accuracy and compatibility. When used in OCR, it cannot meet the OCR's requirements for recognition accuracy. Summary of the invention

[0006] In this context, embodiments of the present disclosure are intended to provide an image recognition method and apparatus, a storage medium, and an electronic device.

[0007] In a first aspect of the embodiments of the present disclosure, there is provided an image recognition method, comprising:

[0008] Obtaining image segmentation features of the image to be identified;

[0009] Using an image recognition model to identify the segmentation features of the image to be identified, so as to determine a target identification label of the image to be identified;

[0010] The image recognition model is obtained by training with image segmentation features and image classification features extracted from image training data.

[0011] In some embodiments of the present disclosure, using the image segmentation features and the image classification features to train an image recognition model includes:

[0012] Using the image classification features to train an image recognition model to obtain a classification recognition label;

[0013] Retraining the image recognition model using the image classification features with the classification recognition labels;

[0014] The retrained image recognition model is trained using the image segmentation features to obtain a trained image recognition model.

[0015] In some embodiments of the present disclosure, using the image recognition model to recognize the segmentation features of the image to be recognized to determine the target recognition label of the image to be recognized includes:

[0016] Using the image recognition model to identify the segmentation features of the image to be identified, and obtaining a segmentation identification label of the image to be identified;

[0017] The target recognition label is determined according to the segmentation recognition label of the image to be recognized.

[0018] In some embodiments of the present disclosure, the classification identification label is an overall image label, and the segmentation identification label is a pixel label of each pixel in the image.

[0019] In some embodiments of the present disclosure, determining the target recognition label according to the segmentation recognition label of the image to be recognized includes:

[0020] According to each pixel label of the image to be identified, the pixel label with the largest number in the image to be identified is determined as the target identification label of the image to be identified.

[0021] In some embodiments of the present disclosure, the pixel tag further includes a background tag; and the method further includes:

[0022] Before determining the pixel label with the largest number in the image to be identified, the background label is removed.

[0023] In some embodiments of the present disclosure, the image training data includes: labeled training data and unlabeled training data; the method further includes:

[0024] Using the image recognition model to recognize the unlabeled training data, and determining the confidence of the recognition result;

[0025] Adding labels to the unlabeled training data whose confidence meets a preset threshold, and adding the labeled unlabeled training data to the labeled training data;

[0026] The image segmentation features and the image classification features are extracted from the labeled training data.

[0027] In some embodiments of the present disclosure, the method further includes:

[0028] A non-rigid transformation is performed on the labeled training data.

[0029] In some embodiments of the present disclosure, the method further includes:

[0030] After determining the target recognition label of the image to be recognized, the image to be recognized that meets a preset condition and is added with the target recognition label is added to the labeled training data.

[0031] In some embodiments of the present disclosure, the target identification tag is a language tag; and the method further includes:

[0032] Generate multilingual image data using font libraries and corpora;

[0033] Pre-training the image recognition model using the multilingual image data;

[0034] The pre-trained image recognition model is trained using the image segmentation features and the image classification features to obtain the trained image recognition model.

[0035] In some embodiments of the present disclosure, the method further includes:

[0036] Grid distortion is performed on labeled training data in the image training data to generate images of different handwritings.

[0037] In some embodiments of the present disclosure, the image segmentation feature is a texture feature of a single character; and the image classification feature is a semantic feature.

[0038] In some embodiments of the present disclosure, the image recognition model includes a Backbone network and a DCN module; wherein,

[0039] The DCN modules are connected in series to the output end of the Backbone network.

[0040] In a second aspect of the embodiments of the present disclosure, there is provided an image recognition device, comprising:

[0041] A feature extraction module, used for obtaining the image segmentation features to be identified of the image to be identified;

[0042] A label determination module, used to identify the segmentation features of the image to be identified using an image recognition model to determine a target identification label of the image to be identified;

[0043] The image recognition model is obtained by training with image segmentation features and image classification features extracted from image training data.

[0044] In a third aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and the computer program implements the above method when executed by a processor.

[0045] In a fourth aspect of the embodiments of the present disclosure, there is provided an electronic device, including:

[0046] Processor; and

[0047] A memory, configured to store executable instructions of the processor;

[0048] The processor is configured to perform the above method by executing the executable instructions.

[0049] According to the image recognition method and device, storage medium and electronic device of the embodiment of the present disclosure, on the one hand, by acquiring the image segmentation features and image classification features in the image training data to train the image recognition model, the image segmentation features can be used to perform fine-grained supervision on the image recognition model while macro-supervising the image recognition model, thereby improving the precision and accuracy of the image recognition model training. On the other hand, since the image training data is trained by image segmentation features and image classification features, and the image segmentation features are fine-grained image features, the image recognition model can be used to only recognize the image segmentation features of the image to be recognized, and the target recognition label of the image to be recognized can be accurately determined, thereby simplifying the recognition process and reducing the amount of calculation. On the other hand, the image recognition method provided by the exemplary embodiment of the present disclosure can improve the accuracy of language recognition in images and provide a basis for OCR text recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, in which:

[0051] Figure 1 A flowchart of an image recognition method according to an exemplary embodiment of the present disclosure is schematically shown;

[0052] Figure 2 A flowchart of an image recognition model training method according to an exemplary embodiment of the present disclosure is schematically shown;

[0053] Figure 3 A flowchart schematically shows the process of training and identifying an image recognition model according to an exemplary embodiment of the present disclosure;

[0054] Figure 4 A flowchart of pre-training an image recognition model according to an exemplary embodiment of the present disclosure is schematically shown;

[0055] Figure 5 A schematic diagram of performing non-rigid transformation on labeled training data according to an exemplary embodiment of the present disclosure is schematically shown;

[0056] Figure 6 Schematically showing images of different handwritings obtained according to different grid distortion ratios according to an exemplary embodiment of the present disclosure;

[0057] Figure 7 A schematic diagram of a model for feature extraction and recognition using an image recognition module according to an exemplary embodiment of the present disclosure is schematically shown;

[0058] Figure 8 A block diagram schematically shows an image recognition device according to an exemplary embodiment of the present disclosure;

[0059] Fig. 9 A block diagram of an electronic device according to an exemplary embodiment of the present disclosure is schematically shown.

[0060] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts. DETAILED DESCRIPTION

[0061] The principles and spirit of the present disclosure will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present disclosure, and are not intended to limit the scope of the present disclosure in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0062] Those skilled in the art will appreciate that the embodiments of the present disclosure may be implemented as a system, device, apparatus, method or computer program product. Therefore, the present disclosure may be specifically implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0063] According to an embodiment of the present disclosure, an image recognition method and device are proposed.

[0064] In this document, any number of elements in the drawings is used for illustration rather than limitation, and any naming is used only for distinction and does not have any limiting meaning.

[0065] The principle and spirit of the present disclosure are explained in detail below with reference to several representative embodiments of the present disclosure.

[0066] Public Overview

[0067] The present inventors have discovered that in real life scenarios, there are multiple language environments, which will greatly affect the accuracy of OCR text recognition.

[0068] Based on the above content, the present disclosure proposes an image recognition method based on image semantic segmentation, thereby improving the accuracy of language recognition in images and providing a basis for OCR text recognition.

[0069] After introducing the basic principles of the present disclosure, various non-limiting embodiments of the present disclosure are described in detail below.

[0070] Application Scenario Overview

[0071] Multilingual OCR technology has always been an important component of OCR technology and is widely used in life. For example, technologies such as minority language text recognition and image text translation all use multilingual OCR technology. The image recognition method provided in the present disclosure provides the basis for accurate language recognition for OCR text recognition.

[0072] In any text recognition scenario, the initial language recognition is indispensable. In particular, in image text recognition technology, the recognition of the language in the image is a prerequisite. Any language recognition device or translation device cannot be separated from the basis of language recognition. Therefore, the image recognition method provided by the present disclosure can be applied to any scenario such as recognizing or translating text in an image, and the exemplary embodiments of the present disclosure do not make special limitations on this.

[0073] Exemplary Methods

[0074] Reference below Figure 1 An image recognition method according to an exemplary embodiment of the present disclosure will be described.

[0075] Figure 1 The flowchart of the image recognition method according to an exemplary embodiment of the present disclosure is schematically shown. Figure 1 , the image recognition method according to an exemplary embodiment of the present disclosure may include the following steps:

[0076] Step S12: Obtaining the image segmentation features of the image to be identified;

[0077] Step S14: using the image recognition model to identify the segmentation features of the image to be identified, so as to determine the identification label of the image to be identified;

[0078] Among them, the image recognition model is obtained by training the image segmentation features and image classification features extracted from the image training data.

[0079] It can be seen from the steps of the above-mentioned image recognition method that the identification label of the image to be identified is obtained by identifying the image recognition model. Before performing image recognition, training the image recognition model is also an indispensable part of the present disclosure. Therefore, before the image recognition method is introduced in detail, the exemplary embodiment of the present disclosure introduces the training process of the image recognition model in detail in advance to provide a basis for the introduction of the image recognition method.

[0080] As mentioned above, the above image recognition model is obtained by training the image segmentation features and image classification features extracted from the image training data. Therefore, using the image segmentation features and image classification features to train the image recognition model is the key to the image recognition model training of the present disclosure. Based on this, Figure 2 The flowchart of the image recognition model training method according to an exemplary embodiment of the present disclosure is schematically shown. Figure 2 According to an exemplary embodiment of the present disclosure, the image recognition model training method may include the following steps:

[0081] Step S22: Extracting image segmentation features and image classification features from the image training data.

[0082] Image training data is data composed of a large amount of image information, which is used to train the image recognition model. In the exemplary embodiment of the present disclosure, the image training data has a variety of different sources. In order to improve the accuracy of model training, labeled training data can be used to train the model at the beginning. Since the labeled training data used at the beginning are all manually annotated, and the cost of manual annotation is high and the efficiency is low, therefore, in addition to using the above-mentioned labeled training data, in the exemplary embodiment of the present disclosure, the image training data also includes unlabeled training data.

[0083] In an exemplary embodiment of the present disclosure, a label refers to a mark for identifying an image. For example, in language identification, the label is the language type of the image, such as Chinese, English, Tibetan, Uyghur, etc.; in species identification, the label is the biological type of the image, such as animals, plants, or humans, etc.

[0084] The labeled training data used at the beginning of image recognition model training is mostly formed by manually adding corresponding labels after image recognition, mainly to improve the accuracy of model training and improve training efficiency.

[0085] In an exemplary embodiment of the present disclosure, in addition to training the image recognition model by using labeled training data, the image recognition model can also be used to recognize unlabeled training data to obtain a recognition result of the unlabeled training data. Then, the confidence of the recognition result can be determined.

[0086] In actual applications, image recognition models will accumulate a large amount of unlabeled real scene data during their formal operation. Therefore, in addition to using the above-mentioned labeled training data to train image recognition models, these unlabeled training data can also be used to further optimize image recognition models.

[0087] Specifically, in an exemplary embodiment of the present disclosure, first, an image recognition model is used to identify unlabeled training data to determine the confidence of the recognition result. The confidence is the reliability of the current recognition result as the real result. For example, after an image is identified by an image recognition model, the probability that the text in the image is Chinese is 95%, and the confidence that the recognition result is Chinese can also be 95%. Secondly, a label is added to the unlabeled training data whose confidence meets the preset threshold, and the unlabeled training data with the label is added to the labeled training data; wherein, the preset threshold can be determined according to the actual situation. For example, the preset threshold can be determined as 90%, and the above-listed recognition result is Chinese. The confidence of 95% meets the preset threshold, and the recognition result can be used as a label and added to the unlabeled training data, so that the unlabeled training data also becomes part of the labeled training data for image recognition model training.

[0088] Reference Figure 3 , schematically shows a flowchart of training and recognition of an image recognition model according to an exemplary embodiment of the present disclosure. Figure 3 In the example, before using labeled training data 310 to train image recognition model 330, non-rigid transformation 320 may be performed on labeled training data. Non-rigid transformation 320 refers to a data enhancement method that performs scaling, affine, transmission, polynomial, and other transformations on image data to improve the generalization ability of image recognition model 330. Different non-rigid transformation methods may be determined based on different training data.

[0089] In addition, as mentioned above, combined Figure 3 , the image recognition model 330 can also be used to identify the unlabeled training data 311 to add labels to the unlabeled training data 311 and increase the number of labeled training data 310.

[0090] Next, the image segmentation features 340 and image classification features 341 of the labeled training data 310 are extracted, wherein the labeled training data 310 may be data after non-rigid transformation 320 or data without non-rigid transformation 320, and the exemplary embodiments of the present disclosure do not impose any special limitation on this.

[0091] It should be noted that, in the exemplary embodiment of the present disclosure, the image segmentation feature 340 is a feature of each pixel in the image, focusing on the recognition of each pixel; while the image classification feature 341 is an overall feature of the image, which is a semantic feature.

[0092] Step S24: Use the image classification features to train the image recognition model to obtain classification recognition labels.

[0093] In the exemplary embodiment of the present disclosure, the image segmentation feature 340 is the main learning feature, and the image classification feature 341 is the auxiliary learning feature. The image recognition module 330 is trained by the image classification feature 341, and the obtained classification recognition label 331 is the overall image label, which can supervise the image as a whole as a reference label, but cannot be used as the final recognition label.

[0094] Step S26: Retrain the image recognition model using the image classification features with the classification recognition labels.

[0095] In the exemplary embodiment of the present disclosure, in addition to the above-mentioned sources of unlabeled training data 311 and labeled training data 310, image training data can also use image classification features 341 to obtain classification identification labels 331 to retrain the image recognition model 330. This plays a role in macro-monitoring the image recognition model 330, and by means of back propagation, the obtained classification identification labels 331 are fed back to the image recognition model 330, which can improve the precision and accuracy of the image recognition model 330 training.

[0096] Step S28. Use the image segmentation features to train the retrained image recognition model to obtain a trained image recognition model.

[0097] In the exemplary embodiment of the present disclosure, the image recognition model 330 is ultimately determined by training the image recognition model 330 with a more refined image segmentation feature 340. Since the image segmentation feature 340 contains the features of each pixel, the description of the image features is more accurate, which helps to improve the accuracy of the image recognition model 330 and is also conducive to the determination of the parameters in the model. The parameters of the model are determined by the specific model and are not specifically limited here.

[0098] In an exemplary embodiment of the present disclosure, after the image recognition model 330 is obtained through training, the image recognition model 330 can be used to recognize the image 350 to be recognized.

[0099] In step S12, the image segmentation features of the image to be recognized are obtained.

[0100] In an exemplary embodiment of the present disclosure, during the recognition process of the image to be recognized 350 , the image segmentation features of the image to be recognized 350 are mainly used, and the image segmentation features of the image to be recognized are input into the image recognition model 330 for recognition by the image recognition model 330 .

[0101] In step S14, the image recognition model is used to recognize the segmentation features of the image to be recognized, so as to determine the recognition label of the image to be recognized.

[0102] In an exemplary embodiment of the present disclosure, after the segmentation features of the image to be identified are input into the image recognition model 330, the image recognition model 330 can identify the segmentation features of the image to be identified, obtain the segmentation identification label 351 of the image to be identified, and then determine the target identification label 352 of the image to be identified 350 based on the segmentation identification label 351 of the image to be identified.

[0103] In an exemplary embodiment of the present disclosure, the segmentation identification label 351 is generally a pixel label of each pixel in the image. In the process of determining the target identification label 352 of the image to be identified 350 according to the segmentation identification label 351, the pixel labels can be classified and counted according to each pixel label in the image to be identified 350, and the pixel label with the largest number is determined as the target identification label 352 of the image to be identified 350. For example, a total of 1000 pixel labels are identified in a certain image to be identified 350, wherein these 1000 pixel labels are divided into three categories: Chinese, Japanese and Korean. After classifying and counting the above three types of pixel labels, it is found that there are 850 pixel labels in Chinese, 100 pixel labels in Japanese, and 50 pixel labels in Korean. It can be seen that the pixel label with the largest number is Chinese, so Chinese can be determined as the target identification label 352 of the image to be identified 350.

[0104] It should be noted that in the process of determining the pixel labels of the image to be identified 350, background labels containing only the background may be involved. In order to avoid interference of the background labels on the recognition results, in an exemplary embodiment of the present disclosure, before determining the pixel labels with the largest number in the image to be identified, the background labels need to be removed first.

[0105] In practical applications, there are many ways to remove background labels. For example, in the process of determining the pixel label of each pixel, when a pixel label of a certain pixel is a background label, the background label can be deleted without subsequent classification and statistics. Alternatively, after classifying and counting all pixel labels, all background labels can be determined and deleted. The exemplary embodiments of the present disclosure do not specifically limit the specific method of removing background labels.

[0106] In an exemplary embodiment of the present disclosure, after determining the target recognition label 352 of the image to be recognized 350, the image recognition model 330 may be trained for multiple rounds of iterations. That is, the image recognition model 330 is retrained using the image to be recognized 350 with the target recognition label 352, so as to perform incremental learning on the image recognition model 330 and improve the generalization ability of the image recognition model 330.

[0107] In an exemplary embodiment of the present disclosure, in order to improve the recognition accuracy of the image recognition model 330, after determining the target recognition label 352 of the image to be recognized 350, the image to be recognized 350 that meets preset conditions and has the target recognition label 352 added thereto can be added to the labeled training data 310, so as to train the image recognition model 330 using the image to be recognized 350 with the target recognition label as training data.

[0108] In practical applications, the preset conditions can be set according to actual conditions. For example, the preset condition can be that the proportion of the largest number of pixel labels to the total pixel labels is more than 90%, that is, the proportion of the target identification label 352 to the total pixel labels is more than 90%. At this time, the target identification label 352 can be added to the image to be identified 350, and the image to be identified 350 can be used as labeled training data 310 for subsequent model training. The exemplary embodiments of the present disclosure do not specifically limit the specific preset conditions.

[0109] Before training the image recognition model 330, the image recognition model 330 can also be pre-trained using large-scale generated data, which can be, for example, relevant image data collected on the Internet. Due to the convenience of the network, millions, tens of millions, or even more data can be generated for pre-training the image recognition model 330. By pre-training the image recognition model 330 with these large-scale generated data, since the coverage of these data is very wide, the generalization ability and robustness of the image recognition model 330 can be improved, and each parameter in the image recognition model 330 can be fully utilized to maximize the performance of the image recognition model 330.

[0110] In addition, in the process of pre-training the image recognition model 330 using the above-mentioned large-scale generated data, there is no need to manually label these data, as long as the purpose of model convergence is achieved. After pre-training the image recognition model 330 using the large-scale generated data, the pre-trained image recognition model 330 is adjusted using the above-mentioned image training data to improve the generalization ability of the image recognition model 330 while improving the recognition ability of the image recognition model 330.

[0111] Next, taking language recognition in image recognition as an example, the recognition method in the exemplary embodiment of the present disclosure is described in more detail:

[0112] In language recognition, especially in minority language recognition, the required training data is scarce. Therefore, using the above-mentioned large-scale generated data to pre-train the image recognition model 330 is a prerequisite for improving the model recognition accuracy. Figure 4 The process of pre-training the image recognition model 330 using the above-mentioned large-scale generated data may include the following steps:

[0113] Step S42: Generate multilingual image data using the font library and corpus.

[0114] In practical applications, font libraries of dozens of different languages ​​and a corpus of millions of words can be collected to generate tens of millions of multilingual image data for recognition for pre-training. The specific number of font libraries and corpora can be determined according to actual conditions, and the exemplary embodiments of the present disclosure do not specifically limit this.

[0115] For language recognition, the above-mentioned target recognition label is the language label.

[0116] Step S44: Pre-train the image recognition model using multilingual image data.

[0117] Due to the large amount of multilingual image data and its wide coverage, the pre-trained image recognition model has stronger generalization ability and robustness. In addition, the multilingual image data does not require manual data annotation, which greatly saves labor costs.

[0118] After pre-training the image recognition model using multilingual image data, labeled training data in real scenes can be added to train the pre-trained image recognition model to fine-tune the image recognition model and improve the image recognition model's ability to recognize real scenes. The recognition ability is the ability to recognize languages. The labeled training data in real scenes can be artificially annotated data, or can be converted from unlabeled training data, or can be images to be recognized with target recognition labels, and the exemplary embodiments of the present disclosure do not specifically limit this.

[0119] The process of training the pre-trained image recognition model may include: determining the image segmentation features and image classification features of the real labeled training data, and using the image segmentation features and the image classification features to train the pre-trained image recognition model to obtain a trained image recognition model, which is a language recognition model. The specific process of determining the image segmentation features and the image classification features, as well as the training process can refer to the above-mentioned implementation method section, which will not be repeated here. Among them, for language recognition, the image classification features are semantic features, and the image segmentation features are the texture features of a single character. The texture features can make the image recognition model more focused on the character area and learn the texture features and differences between each character.

[0120] For language recognition, non-rigid transformation of labeled training data in image training data can be grid distortion of labeled training data. Grid distortion is to divide the image into multiple grids, move the positions of grid points within a preset range, compare the initial grid point positions with the moved grid point positions, and solve the parameters in the transformation matrix. Finally, the image can be transformed according to the transformation matrix to generate images of different handwritings.

[0121] For example, grid distortion can be to divide the image into n*m square grids, randomly move the positions of the grid points in a small range, and then make a one-to-one correspondence between the initial grid point positions and the moved grid point positions, and then use the least squares method to solve the parameters of the transformation matrix, and finally transform the image according to the transformation matrix to generate images of different handwritings, such as Figure 5 As shown. Among them, Figure 5 The left side of the figure shows a schematic diagram of an image distorted toward the center, and the right side shows a schematic diagram of an image distorted toward the periphery. Figure 6 , it shows images of different handwritings obtained according to different grid distortion ratios. Figure 6 In the figure, two types of handwriting are grid-distorted. The four images on the right side of the first row are images of different handwritings obtained by grid-distorting the leftmost image in the first row. There are slight differences between these four images. The four images on the right side of the second row are images of different handwritings obtained by grid-distorting the leftmost image in the second row. There are also slight differences between these four images. These slightly different images can be used as different image training data to provide rich data support for model training.

[0122] In the exemplary embodiments of the present disclosure, grid distortion can be used to simulate the changes in different handwritings, so as to achieve the purpose of font recognition of handwritten handwriting, thereby improving the recognition ability of the image recognition model and expanding the recognition range. In particular, in the case of a lack of training data such as recognition of small languages, a large amount of training data can be obtained through grid distortion to meet the accuracy requirements of the image recognition model and provide resources for language recognition.

[0123] For language recognition, the above-mentioned image recognition module may include a Backbone network and a DCN module, wherein the Backbone network is a neural network model that is mainly responsible for extracting image features, such as the above-mentioned image segmentation features and image classification features. In practical applications, feature extraction may be included in the image recognition module, or feature extraction may be performed separately from image recognition, and the exemplary embodiments of the present disclosure do not specifically limit this.

[0124] In the exemplary implementation of the present disclosure, the shape and texture features of each word can be modeled through the Backbone network, wherein the Backbone network can be an HRnet network. The main feature of the HRnet network is that it starts from the first stage, that is, gradually increases the subnets from high resolution to low resolution, forms more stages, and connects the multi-resolution subnets in parallel, and performs multiple multi-scale fusions, so that each high-resolution to low-resolution representation repeatedly receives information from other parallel representations, thereby obtaining rich high-resolution representations. In addition, the above-mentioned Backbone network can also be a ResNet network, a Mobilenet network, etc., which is not specifically limited here.

[0125] In an exemplary embodiment of the present disclosure, the DCN module is connected in series to the output end of the Backbone network, which can enhance the feature modeling capability and make the network learn the texture features of characters more accurately and effectively.

[0126] Reference Figure 7 , showing a schematic diagram of a model for extracting and recognizing features of a font using an image recognition module, Figure 7 In the example, after the character 710 passes through the Backbone network 721 and the DCN module 722, a recognition result 730 related to the image segmentation feature is obtained, and a recognition result 740 related to the image classification feature is also obtained. The final target recognition label is determined by the recognition result 730 related to the image segmentation feature, which will not be repeated here.

[0127] The technical solution of the above-mentioned embodiments of the present disclosure, on the one hand, by acquiring image segmentation features and image classification features in image training data to train the image recognition model, the image segmentation features can be used to perform fine-grained supervision on the image recognition model while performing macro-supervision on the image recognition model, thereby improving the precision and accuracy of the image recognition model training. On the other hand, since the image training data is trained by image segmentation features and image classification features, and the image segmentation features are fine-grained image features, the image recognition model can be used to only recognize the image segmentation features of the image to be recognized, and the target recognition label of the image to be recognized can be accurately determined, thereby simplifying the recognition process and reducing the amount of calculation. On the other hand, the image recognition method provided by the exemplary embodiment of the present disclosure can improve the accuracy of language recognition in images and provide a basis for OCR text recognition.

[0128] Exemplary Devices

[0129] After introducing the image recognition method according to the exemplary embodiment of the present disclosure, Figure 8 The image recognition device according to the exemplary embodiment of the present disclosure is described, wherein the device embodiment part may inherit the relevant description in the method embodiment, so that the device embodiment can obtain the support of the relevant specific description in the method embodiment.

[0130] refer to Figure 8 According to an exemplary embodiment of the present disclosure, the image recognition device 800 may include: a feature extraction module 802 and a label determination module 804 .

[0131] Specifically, the feature extraction module 802 can be used to obtain the image segmentation features to be identified of the image to be identified; the label determination module 804 can be used to use the image recognition model to identify the image segmentation features to be identified to determine the target identification label of the image to be identified; wherein the image recognition model is obtained by training the image segmentation features and image classification features extracted from the image training data.

[0132] In some embodiments of the present disclosure, it also includes: a model training module 806; wherein the model training module 806 can use image classification features to train the image recognition model to obtain classification recognition labels; use image classification features with classification recognition labels to retrain the image recognition model; use image segmentation features to train the retrained image recognition model to obtain a trained image recognition model.

[0133] In some embodiments of the present disclosure, the label determination module 804 can be used to use the image recognition model to identify the segmentation features of the image to be identified, obtain the segmentation identification label of the image to be identified, and determine the target identification label based on the segmentation identification label of the image to be identified.

[0134] In some embodiments of the present disclosure, the classification identification label is the overall label of the image, and the segmentation identification label is the pixel label of each pixel in the image.

[0135] In some embodiments of the present disclosure, the label determination module 804 may be configured to determine, based on each pixel label of the image to be identified, the pixel label with the largest number in the image to be identified as the target identification label of the image to be identified.

[0136] In some embodiments of the present disclosure, the pixel labels also include background labels; the label determination module 804 may be used to remove the background labels before determining the pixel labels with the largest number in the image to be identified.

[0137] In some embodiments of the present disclosure, image training data includes: labeled training data and unlabeled training data; a model training module 806 can be used to use an image recognition model to identify the unlabeled training data and determine the confidence of the recognition result; add labels to the unlabeled training data whose confidence meets a preset threshold, and add the labeled unlabeled training data to the labeled training data; and extract image segmentation features and image classification features from the labeled training data.

[0138] In some embodiments of the present disclosure, the model training module 806 may be used to perform non-rigid transformation on labeled training data.

[0139] In some embodiments of the present disclosure, the model training module 806 may be used to add the image to be identified that meets preset conditions and has the target identification label added thereto to the labeled training data after determining the target identification label of the image to be identified.

[0140] In some embodiments of the present disclosure, the target recognition label is a language label; the model training module 806 can be used to generate multilingual image data using a font library and a corpus; pre-train an image recognition model using the multilingual image data; and train the pre-trained image recognition model using image segmentation features and image classification features to obtain a trained image recognition model.

[0141] In some embodiments of the present disclosure, the model training module 806 may be used to perform grid distortion on labeled training data in the image training data to generate images of different handwritings.

[0142] In some embodiments of the present disclosure, the image segmentation feature is a texture feature of a single character; and the image classification feature is a semantic feature.

[0143] In some embodiments of the present disclosure, the image recognition model includes a Backbone network and a DCN module; wherein the DCN module is connected in series to the output end of the Backbone network.

[0144] Since the functional modules of the image recognition device in the embodiment of the present disclosure are the same as those in the above-mentioned method disclosure embodiment, they will not be described in detail here.

[0145] The image recognition device of the above-mentioned embodiment of the present disclosure, on the one hand, trains the image recognition model by acquiring the image segmentation features and the image classification features in the image training data, and can use the image segmentation features to perform fine-grained supervision on the image recognition model while macro-supervising the image recognition model, thereby improving the precision and accuracy of the image recognition model training. On the other hand, since the image training data is trained by the image segmentation features and the image classification features, and the image segmentation features are fine-grained image features, the image recognition model can be used to recognize only the image segmentation features of the image to be recognized, and the target recognition label of the image to be recognized can be accurately determined, thereby simplifying the recognition process and reducing the amount of calculation.

[0146] Exemplary Devices

[0147] After introducing the image recognition method and device according to the exemplary embodiment of the present disclosure, the electronic device according to the exemplary embodiment of the present disclosure will be described below. The electronic device according to the exemplary embodiment of the present disclosure includes the above-mentioned image recognition device.

[0148] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0149] In some possible implementations, the electronic device according to the present disclosure may include at least one processing unit and at least one storage unit. The storage unit stores a program code, and when the program code is executed by the processing unit, the processing unit executes the steps in various exemplary implementations according to the present disclosure described in the above “method” section of this specification.

[0150] Refer to the following Fig. 9The electronic device 2000 according to this embodiment of the present disclosure is described. Fig. 9 The electronic device 2000 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0151] like Fig. 9 As shown, the electronic device 2000 is in the form of a general computing device. The components of the electronic device 2000 may include but are not limited to: the at least one processing unit 2010, the at least one storage unit 2020, a bus 2030 connecting different system components (including the storage unit 2020 and the processing unit 2010), and a display unit 2040.

[0152] The storage unit stores program codes, which can be executed by the processing unit 2010, so that the processing unit 2010 performs the steps according to various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of this specification. For example, the processing unit 2010 can perform the following steps: Figure 1 , Figure 2 and Figure 4 Follow the steps shown in .

[0153] The storage unit 2020 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 20201 and / or a cache storage unit 20202 , and may further include a read-only storage unit (ROM) 20203 .

[0154] The storage unit 2020 may also include a program / utility 20204 having a set (at least one) of program modules 20205, such program modules 20205 including but not limited to: an operating system, one or more application programs, other program modules and program data, each of which or some combination may include the implementation of a network environment.

[0155] Bus 2030 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0156] The electronic device 2000 may also communicate with one or more external devices 2070 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 2000, and / or communicate with any device that enables the electronic device 2000 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 2050. Furthermore, the electronic device 2000 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 2060. As shown, the network adapter 2060 communicates with other modules of the electronic device 2000 via a bus 2030. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 2000, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0157] Exemplary Program Products

[0158] In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product, which includes a program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the image recognition method according to various exemplary implementations of the present disclosure described in the above "Method" section of this specification.

[0159] The program product may adopt any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical disk, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In addition, as technology develops, readable storage media should also be interpreted accordingly.

[0160] Program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user computing device, partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0161] It should be noted that, although several modules or submodules in the above-mentioned device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules described above can be embodied in one module. Conversely, the features and functions of one module described above can be further divided into multiple modules to be embodied.

[0162] In addition, although the operations of the disclosed method are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0163] Although the spirit and principle of the present disclosure have been described with reference to several specific embodiments, it should be understood that the present disclosure is not limited to the disclosed specific embodiments, and the division of various aspects does not mean that the features in these aspects cannot be combined to benefit, and such division is only for the convenience of expression. The present disclosure is intended to cover various modifications and equivalent arrangements included in the spirit and scope of the attached claims.

Claims

1. An image recognition method, characterized in that: include: Obtaining image segmentation features of the image to be identified; Using an image recognition model to identify the segmentation features of the image to be identified, so as to determine a target identification label of the image to be identified; The image recognition model is obtained by training the image segmentation features and image classification features extracted from the image training data; the image segmentation feature is the feature of each pixel in the image, and the image classification feature is the overall feature of the image; Wherein, the target identification label is a language label; and the method further comprises: Generate multilingual image data using font libraries and corpora; Pre-training the image recognition model using the multilingual image data; The pre-trained image recognition model is trained using the image segmentation features and the image classification features to obtain the trained image recognition model.

2. The method according to claim 1, characterized in that The method further comprises: The image recognition model is trained using the image classification features to obtain a classification recognition label; the classification recognition label is the overall label of the image; Retraining the image recognition model using the image classification features with the classification recognition labels; The retrained image recognition model is trained using the image segmentation features to obtain a trained image recognition model.

3. The method according to claim 2, characterized in that Using the image recognition model to recognize the segmentation features of the image to be recognized to determine the target recognition label of the image to be recognized includes: Using the image recognition model to identify the segmentation features of the image to be identified, and obtaining a segmentation recognition label of the image to be identified; the segmentation recognition label is a pixel label of each pixel in the image; The target recognition label is determined according to the segmentation recognition label of the image to be recognized.

4. The method according to claim 3, characterized in that Determining the target identification label according to the segmentation identification label of the image to be identified includes: According to each pixel label of the image to be identified, the pixel label with the largest number in the image to be identified is determined as the target identification label of the image to be identified.

5. The method according to claim 4, characterized in that The pixel tag also includes a background tag; the method also includes: Before determining the pixel label with the largest number in the image to be identified, the background label is removed.

6. The method according to claim 1, characterized in that The image training data includes: labeled training data and unlabeled training data; the method further includes: Using the image recognition model to recognize the unlabeled training data, and determining the confidence of the recognition result; Adding labels to the unlabeled training data whose confidence meets a preset threshold, and adding the labeled unlabeled training data to the labeled training data; The image segmentation features and the image classification features are extracted from the labeled training data.

7. The method according to claim 6, characterized in that The method further comprises: A non-rigid transformation is performed on the labeled training data.

8. The method according to claim 6, characterized in that The method further comprises: After the target recognition label of the image to be recognized is determined, the image to be recognized that meets a preset condition and is added with the target recognition label is added to the labeled training data.

9. The method according to claim 1, characterized in that: The method further comprises: Grid distortion is performed on labeled training data in the image training data to generate images of different handwritings.

10. The method according to claim 1, characterized in that The image segmentation feature is a texture feature of a single character; the image classification feature is a semantic feature.

11. The method according to claim 1, characterized in that: The image recognition model includes a Backbone network and a DCN module; wherein, The DCN modules are connected in series to the output end of the Backbone network.

12. An image recognition device, characterized in that: include: A feature extraction module, used for obtaining the image segmentation features to be identified of the image to be identified; A label determination module, used to identify the segmentation features of the image to be identified using an image recognition model to determine a target identification label of the image to be identified; The image recognition model is obtained by training the image segmentation features and image classification features extracted from the image training data; the image segmentation feature is the feature of each pixel in the image, and the image classification feature is the overall feature of the image; Among them, the target recognition label is a language label; the device also includes a model training module, which is used to: use a font library and a corpus to generate multilingual image data; use the multilingual image data to pre-train the image recognition model; use the image segmentation features and the image classification features to train the pre-trained image recognition model to obtain the trained image recognition model.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

14. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to perform the method of any one of claims 1 to 11 by executing the executable instructions.

Images