Pinyin recognition method, device, storage medium and electronic device

Abstract

The present disclosure relates to a pinyin recognition method, device, storage medium and electronic device to solve the problems existing in the related art. The method comprises: obtaining an image to be recognized; inputting the image to be recognized into a trained pinyin localization model to obtain a pinyin text sub-image output by the pinyin localization model; inputting the pinyin text sub-image into a trained pinyin recognition model to obtain the pinyin text recognized from the pinyin text sub-image.

Description

Technical Field

[0001] The present disclosure relates to the field of image recognition, and in particular, to a pinyin recognition method, device, storage medium and electronic device. Background Art

[0002] With the development of the Internet and education, Pinyin has gradually entered the field of online education through the Internet. People can enter Pinyin text into the Internet to carry out related online teaching, online homework grading, photo search and other applications.

[0003] In related technologies, the education industry usually uses a combination of traditional image processing and statistical machine learning to recognize Chinese pinyin. However, the model trained by traditional methods cannot handle complex and changing scenarios, and the pinyin recognition effect in practical applications is poor. Summary of the invention

[0004] The purpose of the present invention is to provide a pinyin recognition method, device, storage medium and electronic device to solve the problems existing in the related art.

[0005] In order to achieve the above object, according to a first aspect of an embodiment of the present disclosure, a pinyin recognition method is provided, the method comprising:

[0006] Obtain an image to be recognized;

[0007] Inputting the image to be recognized into the trained pinyin localization model to obtain the pinyin text sub-image output by the pinyin localization model;

[0008] Inputting the pinyin text sub-image into a trained pinyin recognition model to obtain the pinyin text recognized from the pinyin text sub-image;

[0009] The training process of the pinyin localization model includes:

[0010] Generate a pinyin probability map and a threshold map according to the image sample, and calculate a binary map according to the pinyin probability map and the threshold map, wherein each pixel in the image sample corresponds to a probability label, a threshold label and a binary label;

[0011] Calculate the first loss information according to the probability label corresponding to each pixel in the image sample and the pinyin probability map, calculate the second loss information according to the threshold label corresponding to each pixel in the image sample and the threshold map, and calculate the third loss information according to the binary label corresponding to each pixel in the image sample and the binary map;

[0012] The training parameters of the pinyin localization model are adjusted according to the first loss information, the second loss information, and the third loss information.

[0013] Optionally, the calculating a binary map according to the pinyin probability map and the threshold map includes:

[0014] For each first pixel in the pinyin probability map, the value of the third pixel at the same position as the first pixel in the binary map is calculated based on the value of the first pixel and the value of the second pixel at the same position as the first pixel in the threshold map.

[0015] Optionally, the training parameter includes a second empirical threshold, and for each first pixel in the pinyin probability map, the value of a third pixel at the same position as the first pixel in the binary map is calculated according to the value of the first pixel and the value of a second pixel at the same position as the first pixel in the threshold map, including:

[0016] The value of the third pixel is calculated by the following formula:

[0017]

[0018] Among them, B i,j represents the value of the third pixel point with coordinates (i, j) in the binary image, P i,j represents the value of the first pixel with coordinates (i, j) in the probability map, T i,j The value of the second pixel with coordinates (i, j) in the threshold map is represented, and k is the second empirical threshold.

[0019] Optionally, the step of inputting the image to be recognized into a trained pinyin localization model to obtain a pinyin text sub-image output by the pinyin localization model comprises:

[0020] Inputting the image to be recognized into the trained pinyin localization model to generate a target pinyin probability map;

[0021] The pinyin text sub-image is determined according to the magnitude relationship between the value of each pixel point in the target pinyin probability map and the first empirical threshold.

[0022] Optionally, the pinyin localization model is trained in the following manner:

[0023] Training a pinyin localization model to be trained according to a first image sample and a first pinyin text sub-image sample corresponding to the first image sample to obtain an initial pinyin localization model, wherein the first image sample is obtained by adding noise information to a synthetic image sample;

[0024] Training the initial pinyin localization model according to the second image sample and the second pinyin text sub-image sample corresponding to the second image sample to obtain the trained pinyin localization model;

[0025] Among them, the second pinyin text sub-image sample is obtained by inputting the second image sample into the initial pinyin positioning model, obtaining the pinyin text sub-image sample to be corrected output by the initial pinyin positioning model, and then correcting the pinyin text sub-image sample to be corrected, and the image sample includes the first image sample and the second image sample.

[0026] Optionally, the pinyin recognition model is trained in the following manner:

[0027] Training a pinyin recognition model to be trained according to the first pinyin text sub-image sample and a first pinyin text sample corresponding to the first pinyin text sub-image sample to obtain an initial pinyin recognition model;

[0028] Training the initial pinyin recognition model according to the second pinyin text sub-image sample and the second pinyin text sample corresponding to the second pinyin text sub-image sample to obtain the trained pinyin recognition model;

[0029] Among them, the second pinyin text sample is obtained by inputting the second pinyin text sub-image sample into the initial pinyin recognition model, obtaining the pinyin text to be corrected output by the initial pinyin recognition model, and then correcting the pinyin text to be corrected.

[0030] Optionally, the calculating the first loss information according to the probability label corresponding to each pixel in the image sample and the pinyin probability map, the calculating the second loss information according to the threshold label corresponding to each pixel in the image sample and the threshold map, and the calculating the third loss information according to the binary label corresponding to each pixel in the image sample and the binary map, includes:

[0031] The first loss information is calculated based on the probability label corresponding to each pixel point in the first sampling area in the image sample and the first sampling area of ​​the pinyin probability map, the second loss information is calculated based on the threshold label corresponding to each pixel point in the second sampling area in the image sample and the second sampling area of ​​the threshold map, and the third loss information is calculated based on the binary label corresponding to each pixel point in the first sampling area in the image sample and the first sampling area of ​​the binary map, the second sampling area includes the positive sample area and at least part of the negative sample area of ​​the threshold map, and the at least part of the negative sample area is consistent with the positive sample area in the pinyin probability map.

[0032] Optionally, the training parameters include a loss weight value, and the loss function of the pinyin localization model is:

[0033] L=αL s +βL t +γL b ,

[0034] in,

[0035]

[0036]

[0037] Among them, L represents the loss information, L s represents the first loss information, L t represents the second loss information, L b represents the third loss information, α, β, γ are the loss weight values, S l represents the first sampling area, y n represents the probability label of the nth pixel in the first sampling area in the image sample, x n represents the value of the nth pixel in the first sampling area in the pinyin probability map, y′ n represents the binary label of the nth pixel in the first sampling area in the image sample, x′ n represents the value of the nth pixel in the first sampling area in the binary image, R d represents the second sampling area, represents the threshold label of the nth pixel in the second sampling area in the image sample, Represents the value of the nth pixel in the second sampling area in the threshold map.

[0038] Optionally, the first sampling area is determined by:

[0039] Determine a first total number of all positive sample pixels in the image sample;

[0040] Obtaining a second total number of target negative sample pixels according to the first total number and a preset multiple;

[0041] Based on the second total number, selecting a corresponding number of target negative sample pixel points from the negative sample pixel points of the image sample, wherein the loss value corresponding to any of the target negative sample pixel points is greater than the loss values ​​of other negative sample pixel points;

[0042] The first sampling area is determined according to all the positive sample pixels and all the target negative sample pixels.

[0043] According to a second aspect of an embodiment of the present disclosure, a pinyin recognition device is provided, the device comprising:

[0044] An acquisition module, used for acquiring an image to be recognized;

[0045] A first input module, used for inputting the image to be recognized into a trained pinyin localization model to obtain a pinyin text sub-image output by the pinyin localization model;

[0046] A second input module, used for inputting the pinyin text sub-image into a trained pinyin recognition model to obtain the pinyin text recognized from the pinyin text sub-image;

[0047] The first training module is used to train the pinyin localization model in the following manner:

[0048] Generate a pinyin probability map and a threshold map according to the image sample, and calculate a binary map according to the pinyin probability map and the threshold map, wherein each pixel in the image sample corresponds to a probability label, a threshold label and a binary label;

[0049] Calculate the first loss information according to the probability label corresponding to each pixel in the image sample and the pinyin probability map, calculate the second loss information according to the threshold label corresponding to each pixel in the image sample and the threshold map, and calculate the third loss information according to the binary label corresponding to each pixel in the image sample and the binary map;

[0050] The training parameters of the pinyin localization model are adjusted according to the first loss information, the second loss information, and the third loss information.

[0051] According to a third aspect of an embodiment of the present disclosure, there is provided a non-temporary computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps of any one of the methods described in the first aspect are implemented.

[0052] According to a fourth aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0053] a memory having a computer program stored thereon;

[0054] A processor is used to execute the computer program in the memory to implement the steps of any one of the methods described in the first aspect above.

[0055] By adopting the above technical solution, at least the following technical effects can be achieved:

[0056] By obtaining an image to be identified, the image to be identified is input into a trained pinyin localization model to obtain a pinyin text sub-image output by the pinyin localization model. Then the pinyin text sub-image is input into the trained pinyin recognition model to obtain the pinyin text recognized from the pinyin text sub-image. Wherein, the training process of the pinyin localization model includes: generating a pinyin probability map and a threshold map according to the image sample, and calculating a binary map according to the pinyin probability map and the threshold map, so that the pinyin probability map and the threshold map can be linked through the binary map. Since the training process of the pinyin localization model in the present disclosure can adopt a supervised learning method, each pixel in the image sample can correspond to a probability label, a threshold label and a binary label. On this basis, the first loss information can be calculated according to the probability label and the pinyin probability map corresponding to each pixel in the image sample, the second loss information can be calculated according to the threshold label and the threshold map corresponding to each pixel in the image sample, and the third loss information can be calculated according to the binary label and the binary map corresponding to each pixel in the image sample. Thus, the training parameters of the pinyin localization model can be adjusted according to the first loss information, the second loss information and the third loss information.

[0057] After adjusting the pinyin localization model according to the first loss information, the second loss information and the third loss information in a training, the pinyin probability map and the threshold map generated by the model in the next training according to the image sample can be affected, and the binary map calculated according to the pinyin probability map and the threshold map can be affected. Through the binary map, the foreground image (i.e., the pinyin text sub-image) and the background image (i.e., the non-pinyin text sub-image) can be distinguished. On this basis, multiple trainings can be performed for the image sample. In the process of multiple trainings, since the pinyin probability map and the threshold map are linked together through the binary map, adaptive probability values ​​and / or threshold values ​​can be trained for each pixel. In this way, the trained pinyin localization model can accurately output the pinyin text sub-image (i.e., the foreground image) for the image composed of pixels in different situations, and then the effect of the pinyin recognition model recognizing the pinyin text can be improved. Adopting this method disclosed by the present invention, the pinyin recognition effect of the model in practical application can be improved.

[0058] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The accompanying drawings are used to provide a further understanding of the present disclosure and constitute a part of the specification. Together with the following specific embodiments, they are used to explain the present disclosure but do not constitute a limitation of the present disclosure. In the accompanying drawings:

[0060] Figure 1 The figure is a flow chart of a pinyin recognition method according to an exemplary embodiment of the present disclosure.

[0061] Figure 2 It is a block diagram of a pinyin recognition device according to an exemplary embodiment of the present disclosure.

[0062] Figure 3 It is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0063] The specific implementation of the present disclosure is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described herein is only used to illustrate and explain the present disclosure, and is not used to limit the present disclosure.

[0064] In the related art, the education industry usually adopts a combination of traditional image processing and statistical machine learning to recognize Chinese pinyin. By performing preprocessing operations on the input image, then performing single-character area positioning and character feature extraction, the area where the pinyin text is located is located, and using statistical machine learning methods such as SVM (Support Vector Machine), logistic regression, and decision trees to build a Chinese pinyin recognition classifier to perform predictive recognition of a single character. Although the preprocessing operation can improve the image quality, the process is cumbersome and there is error transmission, and when extracting single-character features, it is very dependent on single-character area positioning and segmentation. When there are font changes, background interference, character distortion adhesion or noise interference, it is easy to affect the recognition effect. In addition, in the existing model training process for locating the area where the pinyin text is located, for each pixel in the image, the same empirical threshold is usually used to determine the classification of each pixel in the probability map output by the model. The model trained in this way is not conducive to processing complex and changeable scenes, and the pinyin recognition effect in practical applications is poor.

[0065] In view of this, the present disclosure provides a pinyin recognition method, device, storage medium and electronic device to solve the problems existing in the related art.

[0066] The technical solution of the present disclosure is described in detail below with reference to embodiments.

[0067] Figure 1 is a flowchart of a pinyin recognition method according to an exemplary embodiment of the present disclosure. Figure 1 As shown, the method includes:

[0068] S101: Obtain an image to be recognized.

[0069] S102, inputting the image to be recognized into the trained pinyin localization model to obtain the pinyin text sub-image output by the pinyin localization model.

[0070] Among them, the image to be recognized may contain Pinyin text, and the Pinyin positioning model can be used to locate the area where the Pinyin text is located in the image to be recognized. In the present disclosure, the image to be recognized is input into the trained Pinyin positioning model, and the Pinyin text sub-image output by the Pinyin positioning model can be obtained. It can be understood that the Pinyin text sub-image can represent the area where the Pinyin text is located in the image to be recognized. In addition, if the image to be recognized does not contain Pinyin text, after the image to be recognized is input into the trained Pinyin positioning model, the model can output a default value, and the default value represents that the image to be recognized does not contain Pinyin text.

[0071] S103, inputting the pinyin text sub-image into the trained pinyin recognition model to obtain the pinyin text recognized from the pinyin text sub-image.

[0072] The pinyin recognition model can be used to recognize the pinyin text in the pinyin text sub-image. In the present disclosure, the pinyin text sub-image is input into the trained pinyin recognition model to obtain the pinyin text recognized from the pinyin text sub-image.

[0073] It should be noted that the training process of the pinyin localization model may include:

[0074] Generate a pinyin probability map and a threshold map according to the image sample, and calculate a binary map according to the pinyin probability map and the threshold map, wherein each pixel in the image sample corresponds to a probability label, a threshold label and a binary label;

[0075] Calculate the first loss information according to the probability label and the pinyin probability map corresponding to each pixel in the image sample, calculate the second loss information according to the threshold label and the threshold map corresponding to each pixel in the image sample, and calculate the third loss information according to the binary label and the binary map corresponding to each pixel in the image sample;

[0076] The training parameters of the pinyin localization model are adjusted according to the first loss information, the second loss information, and the third loss information.

[0077] In the present disclosure, a pinyin probability map can be generated according to the probability of whether each pixel in the image sample represents a pinyin text, and a corresponding threshold map can be generated according to the critical value of the probability of whether each pixel in the image sample represents a pinyin text. On this basis, a binary map can be calculated according to the pinyin probability map and the threshold map. Wherein, the probability and critical value of the pinyin text represented by each pixel are obtained after the pixel value of the pixel passes through the convolution, pooling and other operations in the pinyin positioning model, and then through the Sigmoid function mapping, and it can be understood that the range of values ​​after the Sigmoid function mapping is between 0 and 1. In addition, the pixels in the pinyin probability map, the threshold map and the binary map can correspond one to one with the pixels in the image sample.

[0078] Since the training process of the pinyin localization model can adopt a supervised learning method, each pixel in the image sample can correspond to a probability label, a threshold label and a binary label. Among them, the probability label, the threshold label and the binary label are all related to whether the pixel represents the pinyin text. The probability label can represent the probability that a certain pixel represents the pinyin text. The threshold label can represent the critical value of the probability of whether a certain pixel represents the pinyin text. The binary label can take two values, which respectively represent the case where the probability value is less than the critical value and the case where the probability value is greater than or equal to the critical value, and it can be stipulated that when the probability value of the pixel is less than the critical value, the value of the binary label is determined and it is determined that the pixel does not represent the pinyin text; when the probability value of the pixel is greater than the critical value, the value of the binary label is determined and it is determined that the pixel represents the pinyin text. For example, if the probability label of a pixel point is 0.6 and the threshold label is 0.5, then when the binary label takes values ​​0 and 1, and it is stipulated that when the probability value of the pixel point is less than the critical value, the value is 0, and when the probability value of the pixel point is greater than the critical value, the value is 1, then it can be determined that the binary label of the pixel point is 1, and it can be determined that the pixel point represents pinyin text.

[0079] It should be noted that the first loss information can be calculated based on the probability label corresponding to each pixel in the pinyin probability map and the image sample, the second loss information can be calculated based on the threshold map and the threshold label corresponding to each pixel in the image sample, and the third loss information can be calculated based on the binary map and the binary label corresponding to each pixel in the image sample. Thus, the training parameters of the pinyin localization model can be adjusted based on the first loss information, the second loss information, and the third loss information.

[0080] The above method is adopted, by obtaining an image to be identified, inputting the image to be identified into a trained pinyin localization model, and obtaining a pinyin text sub-image output by the pinyin localization model. Then the pinyin text sub-image is input into the trained pinyin recognition model to obtain the pinyin text recognized from the pinyin text sub-image. Among them, the training process of the pinyin localization model includes: generating a pinyin probability map and a threshold map according to the image sample, and calculating a binary map according to the pinyin probability map and the threshold map, so that the pinyin probability map and the threshold map can be linked through the binary map. On this basis, the training parameters of the pinyin localization model can be adjusted according to the first loss information of the pinyin probability map, the second loss information of the threshold map, and the third loss information of the binary map.

[0081] Since the model can be trained multiple times for image samples, and the pinyin probability map and the threshold map can be linked through a binary map, and the foreground image (i.e., the pinyin text sub-image) and the background image (i.e., the non-pinyin text sub-image) can be distinguished according to the binary map, therefore, after adjusting the pinyin positioning model according to the first loss information, the second loss information, and the third loss information in a training, the pinyin probability map and the threshold map generated by the model in the next training can be affected, and the binary map calculated according to the pinyin probability map and the threshold map can be affected. On this basis, an accurately recognized binary map can be obtained after multiple trainings, so that the trained pinyin positioning model can accurately output the pinyin text sub-image (i.e., the foreground image) for the image composed of pixels in different situations, and then the effect of the pinyin recognition model recognizing the pinyin text can be improved. Adopting this method disclosed in the present invention, the pinyin recognition effect of the model in practical applications can be improved.

[0082] In order to make it easier for ordinary technicians in the field to understand the technical solution provided by the present disclosure, the above steps are explained in detail with examples below.

[0083] Optionally, the process of calculating the binary map according to the pinyin probability map and the threshold map may include:

[0084] For each first pixel in the pinyin probability map, the value of the third pixel at the same position as the first pixel in the binary map is calculated based on the value of the first pixel and the value of the second pixel at the same position as the first pixel in the threshold map.

[0085] Among them, since the first pixel and the second pixel have the same position coordinates in the image sample, they have the same initial pixel value. The value of the first pixel and the value of the second pixel can be based on the initial pixel value after convolution, pooling and other operations in the pinyin positioning model, and then the value obtained by the Sigmoid function mapping. It can be understood that the value range after the Sigmoid function mapping is between 0 and 1. On this basis, the value of the third pixel can be calculated according to the value of the first pixel and the value of the second pixel.

[0086] For example, the value of the third pixel can be calculated by the following formula:

[0087]

[0088] Among them, B i,j represents the value of the third pixel point with coordinates (i, j) in the binary image, P i,j represents the value of the first pixel with coordinates (i, j) in the probability map, T i,jThe value of the second pixel with coordinates (i, j) in the threshold map is represented, and k is the second empirical threshold.

[0089] Among them, the second empirical threshold can be set according to actual conditions, for example, to 50, and the present disclosure does not make specific limitations on this. It can be understood that according to the above formula, if the second empirical threshold k is a positive number, then when the value of the first pixel is greater than or equal to the value of the second pixel, the value of the third pixel is greater than 0.5; when the value of the first pixel is less than the value of the second pixel, the value of the third pixel is less than 0.5. In this case, it can be stipulated that when the value of the third pixel approaches 1, it is determined that the pixel represents the pinyin text in the image sample; when the value of the third pixel approaches 0, it is determined that the pixel does not represent the pinyin text in the image sample.

[0090] It should be understood that after all third pixel points representing the pinyin text are determined, the pinyin text sub-image sample in the image sample can be determined.

[0091] Optionally, the binary image calculated during the training of the pinyin localization model can be used to determine the pinyin text sub-image sample in the image sample. On this basis, the above step S102 may include:

[0092] Input the image to be recognized into the trained pinyin localization model to generate a target pinyin probability map;

[0093] The pinyin text sub-image is determined according to the magnitude relationship between the value of each pixel point in the target pinyin probability map and the first empirical threshold.

[0094] It should be noted that after the pinyin localization model training is completed, in order to reduce the computational overhead, only the target pinyin probability map can be generated for the input image to be recognized. And the pinyin text sub-image can be determined according to the value of each pixel point in the target pinyin probability map and the first empirical threshold. Among them, the first empirical threshold can be set according to the actual situation, and the present disclosure does not make specific restrictions on this.

[0095] For example, when the value of a target pixel in the target pinyin probability map is greater than or equal to a first empirical threshold, it can be determined that the target pixel represents a pinyin text, and when the value of the target pixel is less than the first empirical threshold, it can be determined that the target pixel does not represent a pinyin text. After all target pixels representing pinyin text are determined, a pinyin text sub-image in the image to be identified can be determined.

[0096] For example, if the first empirical threshold is 0.6, then when the value of the target pixel in the target pinyin probability map is greater than or equal to 0.6, it can be determined that the target pixel represents the pinyin text, and when the value of the target pixel is less than 0.6, it can be determined that the target pixel does not represent the pinyin text.

[0097] It is worth noting that, in the process of training the pinyin localization model, the pinyin probability map and the threshold map are linked through a binary map, so that an adaptive probability value and / or threshold value can be trained for each pixel. In the case of obtaining an adaptive probability value for each pixel point through training, the trained pinyin localization model can more accurately output the probability of whether each pixel point in the image represents a pinyin text for an image composed of pixels in different situations, thereby obtaining an accurate pinyin text sub-image according to the first empirical threshold.

[0098] In the case of training each pixel to obtain an adaptive threshold, the trained Pinyin localization model can generate a corresponding Pinyin probability map and threshold map for an image composed of pixels in different situations. The threshold of each pixel in the threshold map can have different values ​​corresponding to the pixel, so that the binary map obtained based on the Pinyin probability map and the threshold map is more accurate. In this case, the corresponding Pinyin probability map and threshold map can be generated for the input image to be identified, and the binary map can be calculated based on the Pinyin probability map and the threshold map, so as to determine the target pixel points representing the Pinyin text based on the binary map. After determining all the target pixel points representing the Pinyin text, the Pinyin text sub-image in the image to be identified can be determined.

[0099] Under the situation that the adaptive probability value and threshold value are obtained for each pixel point training, the trained pinyin positioning model can more accurately determine the probability of whether each pixel point in the image represents the pinyin text and the critical value of the probability of whether each pixel point represents the pinyin text for the image composed of pixels points in different situations, and generate corresponding pinyin probability map and threshold map. In this case, a binary map can be calculated according to the pinyin probability map and the threshold map, thereby determining the target pixel point representing the pinyin text based on the binary map. After determining all the target pixel points representing the pinyin text, the pinyin text sub-image in the image to be identified can be determined.

[0100] Optionally, the pinyin localization model can be trained in the following way:

[0101] Training the pinyin localization model to be trained according to the first image sample and the first pinyin text sub-image sample corresponding to the first image sample to obtain an initial pinyin localization model, wherein the first image sample is obtained after adding noise information to the synthetic image sample;

[0102] Training an initial pinyin localization model according to the second image sample and the second pinyin text sub-image sample corresponding to the second image sample to obtain a trained pinyin localization model;

[0103] Among them, the second pinyin text sub-image sample is obtained by inputting the second image sample into the initial pinyin positioning model, obtaining the pinyin text sub-image sample to be corrected output by the initial pinyin positioning model, and then correcting the pinyin text sub-image sample to be corrected. The image sample includes the first image sample and the second image sample.

[0104] It should be noted that the image samples required in the training process of the pinyin positioning model can use synthetic data. This is because, in supervised learning, labeled data is usually used for model training, and the cost of manually labeled data is high and the efficiency is low. Therefore, the embodiment of the present disclosure can use synthetic image samples to reduce costs. It can be understood that the process of synthesizing image samples involves synthesizing pinyin text, and while synthesizing the pinyin text, annotation information can be added to it, and the annotation information can be used to characterize the area where the pinyin text is located and the pinyin text in the area.

[0105] For example, data amplification can be performed by analyzing the font, structural features, and sequence features of Chinese Pinyin in real scenes, combined with image enhancement methods, to obtain synthetic image samples. The image enhancement method may refer to adding noise information to the synthetic image samples, and the noise information includes random salt and pepper noise, random watermarks, and random Gaussian noise. By adding noise information to the synthetic image samples, the diversity of the synthetic image samples can be guaranteed, and the generalization of the model can be improved, so that the model can accurately locate the Pinyin text sub-image in actual applications when facing watermarked and / or noisy images to be identified.

[0106] It is understandable that the first image sample can be obtained by adding noise information to the synthesized image sample, and the first pinyin text sub-image sample corresponding to the first image sample can be the area where the pinyin text is located in the first image sample. The area where the pinyin text is located in the first text image sample can be determined by the annotation information added in the process of synthesizing the image sample. After training the pinyin localization model to be trained based on the first image sample and the first pinyin text sub-image sample corresponding to the first image sample, an initial pinyin localization model can be obtained.

[0107] In some embodiments, the initial pinyin localization model can be used as a trained pinyin localization model.

[0108] In other embodiments, the initial pinyin positioning model can be trained according to the second image sample and the second pinyin text sub-image sample corresponding to the second image sample to obtain a trained pinyin positioning model. Wherein, the second image sample can be an image sample collected from a real scene. This is because, on the basis of obtaining the initial pinyin positioning model through the first image sample training to reduce costs, the initial pinyin positioning model can be used to pre-mark the real image sample (i.e., locate the area where the pinyin text in the second image sample is located), and manually correct the pre-marking result (including but not limited to correcting the pre-marking result, the pre-marking result can be the frame coordinates of the area where the pinyin text is located). Afterwards, the artificial correction result (i.e., the second pinyin text sub-image sample) and the second image sample are input into the initial pinyin positioning model for supervised training, so that the effect of the model in practical applications can be improved. It should be understood that the cost of the pre-marking result of the artificial correction model is lower than the cost of pure artificial annotation data, and the re-training of the model after the pre-marking result of the artificial correction model can greatly improve the effect of the model in practical applications.

[0109] For example, after the second image sample is input into the initial pinyin localization model to obtain the pinyin text sub-image to be corrected output by the initial pinyin localization model, the pinyin text sub-image sample to be corrected can be manually corrected to obtain the second pinyin text sub-image sample. On this basis, the initial pinyin localization model can be trained according to the second image sample and the second pinyin text sub-image sample to obtain a trained pinyin localization model.

[0110] Optionally, the pinyin recognition model can be trained in the following way:

[0111] Training the pinyin recognition model to be trained according to the first pinyin text sub-image sample and the first pinyin text sample corresponding to the first pinyin text sub-image sample to obtain an initial pinyin recognition model;

[0112] Training an initial pinyin recognition model according to the second pinyin text sub-image sample and the second pinyin text sample corresponding to the second pinyin text sub-image sample to obtain a trained pinyin recognition model;

[0113] The second pinyin text sample is obtained by inputting the second pinyin text sub-image sample into an initial pinyin recognition model to obtain the pinyin text to be corrected output by the initial pinyin recognition model, and then correcting the pinyin text to be corrected.

[0114] It should be noted that the first pinyin text sub-image sample is output by the pinyin positioning model, and the first pinyin text sample corresponding to the first pinyin text sub-image sample can be the pinyin text in the first pinyin text sub-image sample. The first pinyin text sample can be determined by the annotation information added in the process of synthesizing the image sample. Similar to the training process of the pinyin positioning model, after training the pinyin recognition model to be trained according to the first pinyin text sub-image sample and the first pinyin text sample corresponding to the first pinyin text sub-image sample, an initial pinyin recognition model can be obtained. It can be understood that this method of training through synthesized image samples can reduce costs.

[0115] In some embodiments, the initial pinyin recognition model can be used as a trained pinyin localization model.

[0116] In other embodiments, the initial pinyin positioning model can be trained according to the second pinyin text sub-image sample and the second pinyin text sample corresponding to the second pinyin text sub-image sample to obtain a trained pinyin positioning model. After the pinyin text to be corrected is obtained by inputting the second pinyin text sub-image sample into the initial pinyin recognition model, the pinyin text to be corrected can be manually corrected to obtain the second pinyin text sample. On this basis, the initial pinyin recognition model can be trained according to the second pinyin text sub-image sample and the second pinyin text sample to obtain a trained pinyin recognition model.

[0117] After pre-annotating the second pinyin text sub-image sample in the real image sample through the initial pinyin recognition model (i.e. identifying the pinyin text in the second pinyin text sub-image sample), combining the pre-annotation results with manual correction (including but not limited to correcting the pre-annotation results, the pre-annotation results may be the pinyin text recognized by the model), and inputting the manual correction results and the second pinyin text sub-image sample into the initial pinyin recognition model for supervised training, the effect of the model in practical applications can be improved.

[0118] Optionally, the process of calculating the first loss information according to the probability label corresponding to each pixel in the image sample and the pinyin probability map, calculating the second loss information according to the threshold label corresponding to each pixel in the image sample and the threshold map, and calculating the third loss information according to the binary label corresponding to each pixel in the image sample and the binary map may include:

[0119] The first loss information is calculated based on the probability labels corresponding to each pixel point in the first sampling area in the image sample and the first sampling area of ​​the pinyin probability map, the second loss information is calculated based on the threshold labels corresponding to each pixel point in the second sampling area in the image sample and the second sampling area of ​​the threshold map, and the third loss information is calculated based on the binary labels corresponding to each pixel point in the first sampling area in the image sample and the first sampling area of ​​the binary map, the second sampling area includes the positive sample area of ​​the threshold map and at least part of the negative sample area, and at least part of the negative sample area is consistent with the positive sample area in the pinyin probability map.

[0120] Among them, the sampling area of ​​the image sample may be less than or equal to all areas of the image sample. The positive sample area of ​​the pinyin probability map and the binary map can represent the area where the pinyin text is located, and the negative sample area of ​​the pinyin probability map and the binary map can represent the area where the non-pinyin text is located. Because, when the training effect of the pinyin positioning model is very good, for the pixel point representing the pinyin text, the value of the pixel point in the pinyin probability map is close to 1, and the value in the threshold map is close to 0, therefore, at least part of the negative sample area in the threshold map can be consistent with the positive sample area in the pinyin probability map.

[0121] Optionally, the training parameters include a loss weight value, and the loss function of the pinyin localization model may be:

[0122] L=αL s +βL t +γL b ,

[0123] in,

[0124]

[0125]

[0126] Among them, L represents the loss information, L s Indicates the first loss information, L t Represents the second loss information, L b represents the third loss information, α, β, γ are the loss weight values, S l represents the first sampling area, y n Represents the probability label of the nth pixel in the first sampling area of ​​the image sample, x n Represents the value of the nth pixel in the first sampling area of ​​the pinyin probability map, y′ n Represents the binary label of the nth pixel in the first sampling area of ​​the image sample, x′ n Represents the value of the nth pixel in the first sampling area of ​​the binary image, R d represents the second sampling area, Represents the threshold label of the nth pixel in the second sampling area of ​​the image sample, Represents the value of the nth pixel in the second sampling area of ​​the threshold map.

[0127] It should be noted that the pixel value of the nth pixel in the image sample can be obtained as x′ after convolution and pooling in the model training process. n and The value of x n According to x′ n and Calculated.

[0128] Optionally, the first sampling area may be determined by:

[0129] Determine a first total number of all positive sample pixels in the image sample;

[0130] Obtaining a second total number of target negative sample pixels according to the first total number and a preset multiple;

[0131] Based on the second total number, selecting a corresponding number of target negative sample pixel points from the negative sample pixel points of the image sample, wherein the loss value corresponding to any target negative sample pixel point is greater than the loss values ​​of other negative sample pixel points;

[0132] A first sampling area is determined according to all positive sample pixels and all target negative sample pixels.

[0133] Among them, the positive sample pixel points can represent the pixel points in the area where the pinyin text is located, and the negative sample pixel points can represent the pixel points in the area where the non-pinyin text is located. The preset multiples can be determined according to actual conditions. For example, the second total number can be 3 times the first total number. The present disclosure does not make any specific limitations on this.

[0134] It should be noted that in the image samples, the area where the phonetic text is located is usually much smaller than the area where the non-phonetic text is located, which will lead to the problem of imbalance between positive and negative samples, and thus affect the training effect of the model. To overcome this problem, the number of negative samples can be reduced, and a part of the negative samples with the highest loss value can be selected from all negative samples to calculate the loss function, which is conducive to improving the recall rate of the model for negative samples (that is, the probability of correctly classifying negative samples as negative samples).

[0135] For example, the number of negative sample pixels to be sampled can be determined according to the number of all positive sample pixels, and based on the determined number, target negative sample pixels corresponding to the number are selected from all negative sample pixels. It can be understood that the loss values ​​corresponding to the target negative sample pixels are greater than the loss values ​​of other negative sample pixels. On this basis, the first sampling area can be determined according to all positive sample pixels and all target negative sample pixels. By training the pixels in the first sampling area, a model with better effect can be obtained.

[0136] The above method is adopted, by obtaining an image to be identified, inputting the image to be identified into a trained pinyin localization model, and obtaining a pinyin text sub-image output by the pinyin localization model. Then the pinyin text sub-image is input into the trained pinyin recognition model to obtain the pinyin text recognized from the pinyin text sub-image. Among them, the training process of the pinyin localization model includes: generating a pinyin probability map and a threshold map according to the image sample, and calculating a binary map according to the pinyin probability map and the threshold map, so that the pinyin probability map and the threshold map can be linked through the binary map. Since the training process of the pinyin localization model in the present disclosure can adopt a supervised learning method, each pixel in the image sample can correspond to a probability label, a threshold label and a binary label. On this basis, the first loss information can be calculated according to the probability label and the pinyin probability map corresponding to each pixel in the image sample, the second loss information can be calculated according to the threshold label and the threshold map corresponding to each pixel in the image sample, and the third loss information can be calculated according to the binary label and the binary map corresponding to each pixel in the image sample. Therefore, the training parameters of the pinyin localization model can be adjusted according to the first loss information, the second loss information and the third loss information.

[0137] After adjusting the pinyin localization model according to the first loss information, the second loss information and the third loss information in a training, the pinyin probability map and the threshold map generated by the model in the next training according to the image sample can be affected, and the binary map calculated according to the pinyin probability map and the threshold map can be affected. Through the binary map, the foreground image (i.e., the pinyin text sub-image) and the background image (i.e., the non-pinyin text sub-image) can be distinguished. On this basis, multiple trainings can be performed for the image sample. In the process of multiple trainings, since the pinyin probability map and the threshold map are linked together through the binary map, adaptive probability values ​​and / or threshold values ​​can be trained for each pixel. In this way, the trained pinyin localization model can accurately output the pinyin text sub-image (i.e., the foreground image) for the image composed of pixels in different situations, and then the effect of the pinyin recognition model recognizing the pinyin text can be improved. Adopting this method disclosed by the present invention, the pinyin recognition effect of the model in practical application can be improved.

[0138] Based on the same inventive concept, the present disclosure also provides a pinyin recognition device, see Figure 2 , Figure 2 is a block diagram of a pinyin recognition device according to an exemplary embodiment of the present disclosure. Figure 2 As shown, the pinyin recognition device 100 includes:

[0139] An acquisition module 101 is used to acquire an image to be recognized;

[0140] The first input module 102 is used to input the image to be recognized into the trained pinyin localization model to obtain the pinyin text sub-image output by the pinyin localization model;

[0141] The second input module 103 is used to input the pinyin text sub-image into the trained pinyin recognition model to obtain the pinyin text recognized from the pinyin text sub-image;

[0142] The first training module 104 is used to train the pinyin localization model in the following manner:

[0143] Generate a pinyin probability map and a threshold map according to the image sample, and calculate a binary map according to the pinyin probability map and the threshold map, wherein each pixel in the image sample corresponds to a probability label, a threshold label and a binary label;

[0144] Calculate the first loss information according to the pinyin probability map and the probability label corresponding to each pixel in the image sample, calculate the second loss information according to the threshold map and the threshold label corresponding to each pixel in the image sample, and calculate the third loss information according to the binary map and the binary label corresponding to each pixel in the image sample;

[0145] The training parameters of the pinyin localization model are adjusted according to the first loss information, the second loss information, and the third loss information.

[0146] The above-mentioned device is adopted, by obtaining an image to be identified, inputting the image to be identified into a trained pinyin localization model, and obtaining a pinyin text sub-image output by the pinyin localization model. Then the pinyin text sub-image is input into the trained pinyin recognition model to obtain the pinyin text recognized from the pinyin text sub-image. Among them, the training process of the pinyin localization model includes: generating a pinyin probability map and a threshold map according to the image sample, and calculating a binary map according to the pinyin probability map and the threshold map, so that the pinyin probability map and the threshold map can be linked through the binary map. Since the training process of the pinyin localization model in the present disclosure can adopt a supervised learning method, each pixel in the image sample can correspond to a probability label, a threshold label and a binary label. On this basis, the first loss information can be calculated according to the probability label and the pinyin probability map corresponding to each pixel in the image sample, the second loss information can be calculated according to the threshold label and the threshold map corresponding to each pixel in the image sample, and the third loss information can be calculated according to the binary label and the binary map corresponding to each pixel in the image sample. Therefore, the training parameters of the pinyin localization model can be adjusted according to the first loss information, the second loss information and the third loss information.

[0147] After adjusting the pinyin localization model according to the first loss information, the second loss information and the third loss information in a training, the pinyin probability map and the threshold map generated by the model according to the image sample in the next training can be affected, and the binary map calculated according to the pinyin probability map and the threshold map can be affected. Through the binary map, the foreground image (i.e., the pinyin text sub-image) and the background image (i.e., the non-pinyin text sub-image) can be distinguished. On this basis, multiple trainings can be performed for the image sample. In the process of multiple trainings, since the pinyin probability map and the threshold map are linked together through the binary map, adaptive probability values ​​and / or threshold values ​​can be trained for each pixel. In this way, the trained pinyin localization model can accurately output the pinyin text sub-image (i.e., the foreground image) for the image composed of pixels in different situations, and then the effect of the pinyin recognition model recognizing the pinyin text can be improved. Adopting this device disclosed in the present invention can improve the pinyin recognition effect of the model in practical applications.

[0148] Optionally, the first training module 104 is further used for:

[0149] For each first pixel in the pinyin probability map, the value of the third pixel at the same position as the first pixel in the binary map is calculated based on the value of the first pixel and the value of the second pixel at the same position as the first pixel in the threshold map.

[0150] Optionally, the training parameter includes a second experience threshold, and the first training module 104 is further configured to:

[0151] The value of the third pixel is calculated by the following formula:

[0152]

[0153] Among them, B i,j represents the value of the third pixel point with coordinates (i, j) in the binary image, P i,j represents the value of the first pixel with coordinates (i, j) in the probability map, T i,j The value of the second pixel with coordinates (i, j) in the threshold map is represented, and k is the second empirical threshold.

[0154] Optionally, the first input module 102 is further used for:

[0155] Inputting the image to be recognized into the trained pinyin localization model to generate a target pinyin probability map;

[0156] The pinyin text sub-image is determined according to the magnitude relationship between the value of each pixel point in the target pinyin probability map and the first empirical threshold.

[0157] Optionally, the first training module 104 is further used to train the pinyin localization model in the following manner:

[0158] Training a pinyin localization model to be trained according to a first image sample and a first pinyin text sub-image sample corresponding to the first image sample to obtain an initial pinyin localization model, wherein the first image sample is obtained by adding noise information to a synthetic image sample;

[0159] Training the initial pinyin localization model according to the second image sample and the second pinyin text sub-image sample corresponding to the second image sample to obtain the trained pinyin localization model;

[0160] Among them, the second pinyin text sub-image sample is obtained by inputting the second image sample into the initial pinyin positioning model, obtaining the pinyin text sub-image sample to be corrected output by the initial pinyin positioning model, and then correcting the pinyin text sub-image sample to be corrected, and the image sample includes the first image sample and the second image sample.

[0161] Optionally, the device 100 further includes a second training module, which is used to train the pinyin recognition model in the following manner:

[0162] Training a pinyin recognition model to be trained according to the first pinyin text sub-image sample and a first pinyin text sample corresponding to the first pinyin text sub-image sample to obtain an initial pinyin recognition model;

[0163] Training the initial pinyin recognition model according to the second pinyin text sub-image sample and the second pinyin text sample corresponding to the second pinyin text sub-image sample to obtain the trained pinyin recognition model;

[0164] Among them, the second pinyin text sample is obtained by inputting the second pinyin text sub-image sample into the initial pinyin recognition model, obtaining the pinyin text to be corrected output by the initial pinyin recognition model, and then correcting the pinyin text to be corrected.

[0165] Optionally, the first training module 104 is further used for:

[0166] The first loss information is calculated based on the probability label corresponding to each pixel point in the first sampling area in the image sample and the first sampling area of ​​the pinyin probability map, the second loss information is calculated based on the threshold label corresponding to each pixel point in the second sampling area in the image sample and the second sampling area of ​​the threshold map, and the third loss information is calculated based on the binary label corresponding to each pixel point in the first sampling area in the image sample and the first sampling area of ​​the binary map, the second sampling area includes the positive sample area and at least part of the negative sample area of ​​the threshold map, and the at least part of the negative sample area is consistent with the positive sample area in the pinyin probability map.

[0167] Optionally, the training parameters include a loss weight value, and the loss function of the pinyin localization model is:

[0168] L=αL s +βL t +γL b ,

[0169] in,

[0170]

[0171]

[0172] Among them, L represents the loss information, L s represents the first loss information, L t represents the second loss information, L b represents the third loss information, α, β, γ are the loss weight values, S l represents the first sampling area, y n represents the probability label of the nth pixel in the first sampling area in the image sample, x n represents the value of the nth pixel in the first sampling area in the pinyin probability map, y′ nrepresents the binary label of the nth pixel in the first sampling area in the image sample, x′ n represents the value of the nth pixel in the first sampling area in the binary image, R d represents the second sampling area, represents the threshold label of the nth pixel in the second sampling area in the image sample, Represents the value of the nth pixel in the second sampling area in the threshold map.

[0173] Optionally, the apparatus 100 further includes a determining module, the determining module being configured to:

[0174] Determine a first total number of all positive sample pixels in the image sample;

[0175] Obtaining a second total number of target negative sample pixels according to the first total number and a preset multiple;

[0176] Based on the second total number, selecting a corresponding number of target negative sample pixel points from the negative sample pixel points of the image sample, wherein the loss value corresponding to any of the target negative sample pixel points is greater than the loss values ​​of other negative sample pixel points;

[0177] The first sampling area is determined according to all the positive sample pixels and all the target negative sample pixels.

[0178] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0179] Based on the same inventive concept, the embodiment of the present disclosure also provides a non-temporary computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps of the above-mentioned pinyin recognition method are implemented.

[0180] Specifically, the computer-readable storage medium can be a flash memory, a hard disk, a multimedia card, a card-type memory (for example, SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a disk, an optical disk, a server, a public cloud server, etc.

[0181] Regarding the computer-readable storage medium in the above embodiment, the steps of the pinyin recognition method implemented when the computer program stored thereon is executed have been described in detail in the embodiment of the method and will not be elaborated here.

[0182] Based on the same inventive concept, an embodiment of the present disclosure further provides an electronic device, the electronic device comprising:

[0183] a memory having a computer program stored thereon;

[0184] The processor is used to execute the computer program in the memory to implement the steps of the above-mentioned pinyin recognition method.

[0185] Figure 3 FIG. 2 is a block diagram of an electronic device 200 according to an exemplary embodiment. Figure 3 As shown, the electronic device 200 may include: a processor 201 and a memory 202. The electronic device 200 may also include one or more of a multimedia component 203, an input / output (I / O) interface 204, and a communication component 205.

[0186] The processor 201 is used to control the overall operation of the electronic device 200 to complete all or part of the steps in the above-mentioned pinyin recognition method. The memory 202 is used to store various types of data to support the operation of the electronic device 200, and these data may include instructions for any application or method used to operate on the electronic device 200, and application-related data, such as contact data, sent and received messages, pictures, audio, video, etc. The memory 202 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (Static Random Access Memory, referred to as SRAM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, referred to as EEPROM), erasable programmable read-only memory (Erasable Programmable Read-Only Memory, referred to as EPROM), programmable read-only memory (Programmable Read-Only Memory, referred to as PROM), read-only memory (Read-Only Memory, referred to as ROM), magnetic memory, flash memory, disk or optical disk. The multimedia component 203 may include a screen and an audio component. The screen may be, for example, a touch screen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone, which is used to receive external audio signals. The received audio signal may be further stored in the memory 202 or sent through the communication component 205. The audio component also includes at least one speaker for outputting audio signals. The I / O interface 204 provides an interface between the processor 201 and other interface modules, and the above-mentioned other interface modules may be keyboards, mice, buttons, etc. These buttons may be virtual buttons or physical buttons. The communication component 205 is used for wired or wireless communication between the electronic device 200 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, 4G or 5G, NB-IOT (Narrow Band Internet of Things), or a combination of one or more of them, so the corresponding communication component 205 may include: Wi-Fi module, Bluetooth module, NFC module.

[0187] In an exemplary embodiment, the electronic device 200 can be implemented by one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), controllers, microcontrollers, microprocessors or other electronic components to execute the above-mentioned pinyin recognition method.

[0188] The preferred embodiments of the present disclosure are described in detail above in conjunction with the accompanying drawings; however, the present disclosure is not limited to the specific details in the above embodiments. Within the technical concept of the present disclosure, a variety of simple modifications can be made to the technical solution of the present disclosure, and these simple modifications all fall within the protection scope of the present disclosure.

[0189] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, the present disclosure will not further describe various possible combinations.

[0190] In addition, various embodiments of the present disclosure may be arbitrarily combined, and as long as they do not violate the concept of the present disclosure, they should also be regarded as the contents disclosed by the present disclosure.

Claims

1. A pinyin recognition method, characterized in that: The method comprises: Obtain an image to be recognized; Inputting the image to be recognized into the trained pinyin localization model to obtain the pinyin text sub-image output by the pinyin localization model; Inputting the pinyin text sub-image into a trained pinyin recognition model to obtain the pinyin text recognized from the pinyin text sub-image; The training process of the pinyin localization model includes: Generate a pinyin probability map and a threshold map according to the image sample, and calculate a binary map according to the pinyin probability map and the threshold map, wherein each pixel in the image sample corresponds to a probability label, a threshold label and a binary label; Calculate the first loss information according to the probability label corresponding to each pixel in the image sample and the pinyin probability map, calculate the second loss information according to the threshold label corresponding to each pixel in the image sample and the threshold map, and calculate the third loss information according to the binary label corresponding to each pixel in the image sample and the binary map; The training parameters of the pinyin localization model are adjusted according to the first loss information, the second loss information, and the third loss information.

2. The method according to claim 1, characterized in that The step of calculating a binary map according to the pinyin probability map and the threshold map includes: For each first pixel in the pinyin probability map, the value of the third pixel at the same position as the first pixel in the binary map is calculated based on the value of the first pixel and the value of the second pixel at the same position as the first pixel in the threshold map.

3. The method according to claim 2, characterized in that The training parameters include a second empirical threshold, and for each first pixel in the pinyin probability map, the value of a third pixel at the same position as the first pixel in the binary map is calculated according to the value of the first pixel and the value of a second pixel at the same position as the first pixel in the threshold map, including: The value of the third pixel is calculated by the following formula: Among them, B i,j represents the value of the third pixel point with coordinates (i, j) in the binary image, P i,j represents the value of the first pixel with coordinates (i, j) in the probability map, T i,j The value of the second pixel with coordinates (i, j) in the threshold map is represented, and k is the second empirical threshold.

4. The method according to claim 1, characterized in that: The step of inputting the image to be recognized into the trained pinyin localization model to obtain the pinyin text sub-image output by the pinyin localization model comprises: Inputting the image to be recognized into the trained pinyin localization model to generate a target pinyin probability map; The pinyin text sub-image is determined according to the magnitude relationship between the value of each pixel point in the target pinyin probability map and the first empirical threshold.

5. The method according to claim 1, characterized in that The pinyin localization model is trained in the following way: Training a pinyin localization model to be trained according to a first image sample and a first pinyin text sub-image sample corresponding to the first image sample to obtain an initial pinyin localization model, wherein the first image sample is obtained by adding noise information to a synthetic image sample; Training the initial pinyin localization model according to the second image sample and the second pinyin text sub-image sample corresponding to the second image sample to obtain the trained pinyin localization model; Among them, the second pinyin text sub-image sample is obtained by inputting the second image sample into the initial pinyin positioning model, obtaining the pinyin text sub-image sample to be corrected output by the initial pinyin positioning model, and then correcting the pinyin text sub-image sample to be corrected, and the image sample includes the first image sample and the second image sample.

6. The method according to claim 5, characterized in that The pinyin recognition model is trained in the following way: Training a pinyin recognition model to be trained according to the first pinyin text sub-image sample and a first pinyin text sample corresponding to the first pinyin text sub-image sample to obtain an initial pinyin recognition model; Training the initial pinyin recognition model according to the second pinyin text sub-image sample and the second pinyin text sample corresponding to the second pinyin text sub-image sample to obtain the trained pinyin recognition model; Among them, the second pinyin text sample is obtained by inputting the second pinyin text sub-image sample into the initial pinyin recognition model, obtaining the pinyin text to be corrected output by the initial pinyin recognition model, and then correcting the pinyin text to be corrected.

7. The method according to any one of claims 1 to 6, characterized in that The first loss information is calculated according to the probability label corresponding to each pixel in the image sample and the pinyin probability map, the second loss information is calculated according to the threshold label corresponding to each pixel in the image sample and the threshold map, and the third loss information is calculated according to the binary label corresponding to each pixel in the image sample and the binary map, including: The first loss information is calculated based on the probability label corresponding to each pixel point in the first sampling area in the image sample and the first sampling area of ​​the pinyin probability map, the second loss information is calculated based on the threshold label corresponding to each pixel point in the second sampling area in the image sample and the second sampling area of ​​the threshold map, and the third loss information is calculated based on the binary label corresponding to each pixel point in the first sampling area in the image sample and the first sampling area of ​​the binary map, the second sampling area includes the positive sample area and at least part of the negative sample area of ​​the threshold map, and the at least part of the negative sample area is consistent with the positive sample area in the pinyin probability map.

8. The method according to claim 7, characterized in that The training parameters include loss weight values, and the loss function of the pinyin localization model is: L=αL s +βL t +γL b , in, Among them, L represents the loss information, L s represents the first loss information, L t represents the second loss information, L b represents the third loss information, α, β, γ are the loss weight values, S l represents the first sampling area, y n represents the probability label of the nth pixel in the first sampling area in the image sample, x n represents the value of the nth pixel in the first sampling area in the pinyin probability map, y′ n represents the binary label of the nth pixel in the first sampling area in the image sample, x′ n represents the value of the nth pixel in the first sampling area in the binary image, R d represents the second sampling area, represents the threshold label of the nth pixel in the second sampling area in the image sample, Represents the value of the nth pixel in the second sampling area in the threshold map.

9. The method according to claim 8, characterized in that The first sampling area is determined by: Determine a first total number of all positive sample pixels in the image sample; Obtaining a second total number of target negative sample pixels according to the first total number and a preset multiple; Based on the second total number, selecting a corresponding number of target negative sample pixel points from the negative sample pixel points of the image sample, wherein the loss value corresponding to any of the target negative sample pixel points is greater than the loss values ​​of other negative sample pixel points; The first sampling area is determined according to all the positive sample pixels and all the target negative sample pixels.

10. A pinyin recognition device, characterized in that: The device comprises: An acquisition module, used for acquiring an image to be recognized; A first input module, used for inputting the image to be recognized into a trained pinyin localization model to obtain a pinyin text sub-image output by the pinyin localization model; A second input module, used for inputting the pinyin text sub-image into a trained pinyin recognition model to obtain the pinyin text recognized from the pinyin text sub-image; The first training module is used to train the pinyin localization model in the following manner: Generate a pinyin probability map and a threshold map according to the image sample, and calculate a binary map according to the pinyin probability map and the threshold map, wherein each pixel in the image sample corresponds to a probability label, a threshold label and a binary label; Calculate the first loss information according to the pinyin probability map and the probability label corresponding to each pixel in the image sample, calculate the second loss information according to the threshold map and the threshold label corresponding to each pixel in the image sample, and calculate the third loss information according to the binary map and the binary label corresponding to each pixel in the image sample; The training parameters of the pinyin localization model are adjusted according to the first loss information, the second loss information, and the third loss information.

11. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 9 are implemented.

12. An electronic device, characterized in that: include: a memory having a computer program stored thereon; A processor, configured to execute the computer program in the memory to implement the steps of the method according to any one of claims 1 to 9.

Images