Nameplate information identification method

Abstract

The invention relates to a nameplate information identification method, and belongs to the technical field of nameplate information identification. The method comprises the following steps: inputting a to-be-recognized nameplate picture into a trained overall model comprising a detection sub-model and a recognition sub-model to obtain a text corresponding to target information; the method comprises the following steps: firstly, inputting nameplate pictures in a training set into a detection sub-model to obtain a target information prediction position; inputting the picture part of the prediction position of the target information into the recognition sub-model to obtain a text corresponding to the target information; respectively calculating loss functions of the identification sub-model and the detection sub-model; and finally, respectively updating the identification sub-model and the detection sub-model according to the calculated loss functions of the identification sub-model and the detection sub-model, and iterating the steps to train the whole model until an iteration stop condition is met, thereby finishing the training. According to the method, the target information prediction position of a certain item target on the nameplate picture is accurately positioned through the model, and then the text corresponding to the target information of the position is accurately extracted and recognized.

Description

Technical Field

[0001] The invention relates to a nameplate information recognition method, belonging to the technical field of nameplate information recognition. Background Art

[0002] The nameplate of power electronic equipment usually contains various power parameter information such as product name, product model, product rated power, etc. General power electronic equipment manufacturers will enter certain information on the nameplate of the equipment. Nameplate information entry plays a huge role in product identification and traceability, inventory and asset management, etc. At present, the factory's equipment nameplate information entry method is usually manual on-site entry. Since the nameplate model specifications of power electronic equipment produced by various manufacturers are not uniform, the manual entry method is prone to errors and inefficiency.

[0003] In addition, some manufacturers use general OCR technology to input nameplate information, convert the text information on the nameplate into digital format for detection and recognition, and return the recognition results in the form of text. This input method is more efficient than manual input; however, general OCR technology will recognize all the text on the nameplate and cannot effectively eliminate interference information, so it is difficult to accurately extract information on specific items on the nameplate. Summary of the invention

[0004] The purpose of the present invention is to provide a nameplate information recognition method to solve the problem that the current nameplate information recognition method is difficult to accurately extract information of specific items on the nameplate.

[0005] In order to solve the above technical problems, the present invention provides a nameplate information recognition method, which inputs the nameplate image to be recognized into a trained overall model including a detection sub-model and a recognition sub-model to obtain a text corresponding to the target information;

[0006] The training steps of the overall model are as follows:

[0007] S1: Input the nameplate images in the training set into the detection sub-model to obtain the predicted position of the target information;

[0008] Inputting the image portion of the target information predicted position into the recognition sub-model to obtain the text corresponding to the target information as the recognition result;

[0009] The loss function of the recognition sub-model is calculated based on the recognition result of the recognition sub-model and the actual content of the target information in the nameplate image; the loss function of the detection sub-model is calculated based on the predicted position of the target information obtained by the detection sub-model and the actual position of the target information in the nameplate image;

[0010] S2: Update the detection sub-model and the recognition sub-model according to their loss functions respectively, and iterate S1-S2 until the iteration stop condition is met to complete the training.

[0011] Furthermore, the detection sub-model obtains the predicted position of the target information in the following manner:

[0012] First, feature extraction is performed on the nameplate image input to the detection sub-model, and then the extracted features are respectively input into the probability prediction branch and the threshold prediction branch of the detection sub-model to obtain a probability map and a threshold map respectively, and then an approximate binary map is obtained based on the probability map and the threshold map to indicate whether the position of each predicted pixel point belongs to the target information position, and the predicted position of the target information is obtained based on the approximate binary map.

[0013] Furthermore, the loss function of the detection sub-model is the sum of the probability map loss, the threshold map loss and the approximate binary map loss.

[0014] Furthermore, the training set includes nameplate images and labels corresponding to the nameplate images; the labels corresponding to the nameplate images include the actual content of the target information in the nameplate images and the actual position of the target information in the nameplate images.

[0015] Furthermore, the nameplate image and the label corresponding to the nameplate image correspond to each other through the image name of the nameplate image.

[0016] Furthermore, the method of extracting features from the nameplate image input to the detection sub-model includes: using ResNet50 as a basic feature extraction network to extract features from the nameplate image input to the detection sub-model.

[0017] Furthermore, the method of obtaining the predicted position of the target information according to the approximate binary image includes: expanding the pixel point area predicted to belong to the target information position in the approximate binary image to obtain the coordinate position of the prediction box as the predicted position of the target information.

[0018] Furthermore, the method in which the recognition sub-model obtains the text corresponding to the target information includes: firstly performing feature extraction on the image portion of the target information prediction position input into the recognition sub-model, inputting the extracted features into the global feature mixing sub-module, and extracting the global mixed features; passing the global mixed features through the global pooling layer, the fully connected layer and the nonlinear activation function to obtain a digital character feature sequence, and then performing linear prediction based on the obtained digital character feature sequence to output the text corresponding to the target information.

[0019] Furthermore, the method of extracting features from the image portion of the target information prediction position of the input recognition sub-model includes: using the first three stages of PP-LCNet as a basic feature extraction network to extract features from the image portion of the target information prediction position of the input recognition sub-model.

[0020] Furthermore, the method of inputting the extracted features into the probability prediction branch and the threshold prediction branch of the detection sub-model respectively includes: adjusting the size of the extracted features to be consistent with the size of the original nameplate image, and then inputting them into the probability prediction branch and the threshold prediction branch respectively.

[0021] Beneficial effects: As a pioneering invention, this method first uses the detection sub-model to locate the area where the target information is located in the nameplate image, and then uses the recognition sub-model to only recognize the specific text information in the area, so as to ensure that the text finally obtained by the overall model only contains the text corresponding to the target information without being mixed with other interfering information; through the series iterative training of the two sub-models, an overall model that is consistent with the actual position and actual content of the target information in the nameplate image can be trained, so that the detection sub-model in the model can accurately locate the predicted position of the target information of a certain target on the nameplate image, and the recognition sub-model can accurately extract and recognize the text corresponding to the target information at that position. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 is a flow chart of a nameplate information identification method in an embodiment of a nameplate information identification method of the present invention;

[0023] Figure 2 It is a diagram of the organization form of a training set label file in an embodiment of a nameplate information recognition method of the present invention;

[0024] Figure 3 It is a flowchart of obtaining a training set in an embodiment of a nameplate information recognition method of the present invention;

[0025] Figure 4 It is a structural principle diagram of a detection sub-model in an overall model in an embodiment of a nameplate information recognition method of the present invention;

[0026] Figure 5 It is an example diagram of an embodiment of a nameplate information recognition method of the present invention, in which the coordinate position of a prediction box is obtained by expanding the pixel point area predicted to belong to the target information position in the approximate binary image as the predicted position of the target information;

[0027] Figure 6 It is a structural principle diagram of an identification sub-model in an overall model in an embodiment of a nameplate information identification method of the present invention;

[0028] Figure 7 The present invention is a diagram of a nameplate information recognition process of a photovoltaic inverter in an embodiment of a nameplate information recognition method of the present invention. DETAILED DESCRIPTION

[0029] The specific implementation of the present invention will be further described below in conjunction with the accompanying drawings.

[0030] Embodiment of the nameplate information identification method

[0031] This embodiment provides a technical solution for a nameplate information recognition method, which inputs a nameplate image to be recognized into a trained overall model including a detection sub-model and a recognition sub-model to obtain text corresponding to the target information;

[0032] Among them, the training steps of the overall model are as follows:

[0033] S1: Input the nameplate images in the training set into the detection sub-model to obtain the predicted position of the target information;

[0034] The image portion of the target information predicted position is input into the recognition sub-model to obtain the text corresponding to the target information as the recognition result; the loss function of the recognition sub-model is calculated according to the recognition result of the recognition sub-model and the actual content of the target information in the nameplate image; the loss function of the detection sub-model is calculated according to the target information predicted position obtained by the detection sub-model and the actual position of the target information in the nameplate image; wherein the target information is the information of a specific item that needs to be accurately extracted, the target information predicted position is the area in the nameplate image where the target information predicted by the detection sub-model is located, and the recognition sub-model only recognizes the information in this area, thereby ensuring that the obtained text only contains the text corresponding to the target information without being mixed with other interfering information.

[0035] S2: Update the detection sub-model and the recognition sub-model according to their loss functions respectively, and iterate S1-S2 until the iteration stop condition is met to complete the training.

[0036] This method is based on deep learning and neural network training. The model obtained through training can first accurately locate the predicted position of the target information of a certain target on the nameplate image, and then accurately extract and identify the text corresponding to the target information at that position. The implementation flow chart of this method is as follows: Figure 1 As shown, the following takes the nameplate information recognition of the photovoltaic inverter as an example, and takes the model and rated power in the nameplate information of the photovoltaic inverter as the recognition target (i.e., target information) of this embodiment, that is, the two recognition targets of the model and rated power in the nameplate information of the photovoltaic inverter are extracted and recognized for description, and the specific recognition steps are as follows:

[0037] Input the nameplate image to be identified into the trained overall model including the detection sub-model and the recognition sub-model to obtain the text corresponding to the target information;

[0038] The overall model consists of two parts: a detection sub-model and a recognition sub-model. The input nameplate image to be recognized is used to obtain the predicted position of the target information through the detection sub-model of the overall model, and then the predicted position of the target information is input into the recognition sub-model of the overall model to obtain the text corresponding to the target information.

[0039] The training steps of the overall model are as follows:

[0040] S1: Input the nameplate images in the training set into the detection sub-model to obtain the predicted position of the target information;

[0041] The training set includes nameplate images and labels corresponding to the nameplate images; the labels corresponding to the nameplate images include the real content of the target information in the nameplate images and the real location of the target information in the nameplate images. In this embodiment, the nameplate images in the training set are nameplate images of photovoltaic inverters, and the labels corresponding to the nameplate images in the training set are stored in the format of a txt file. The organization form of the txt file (also called the training set label file) is as follows: Figure 2 As shown in the figure, the training set label file describes the actual content of the two target information, model and rated power, corresponding to the nameplate image of the photovoltaic inverter (such as Figure 2 The content in "#target text content") and the real location (such as Figure 2 content "#target location information" in the nameplate image); in this embodiment, the training set label file may also include a picture name, and the nameplate picture and the label corresponding to the nameplate picture correspond to each other through the picture name of the nameplate picture. The picture name is the unique ID of this picture, which is used to distinguish which labels (the actual content of the target information in the nameplate picture, the actual location of the target information in the nameplate picture) belong to which picture.

[0042] The process of obtaining the training set in this embodiment is as follows: Figure 3 As shown in the figure, firstly, the nameplate information pictures of photovoltaic inverters taken on site are selected, and the two target information items of model and rated power are selected as the target data to be labeled. In order to improve the robustness of the data set, the shooting angles of the data samples include front shooting and side shooting. At the same time, in order to consider the influence of illumination, the illumination intensity of the data samples includes strong illumination, weak illumination, and shadow. The process of obtaining the training set includes the data preprocessing stage and the manual annotation stage. First, in the data preprocessing stage, for the photos taken on site, the pictures containing the nameplate information of the photovoltaic inverter are manually screened out as the nameplate pictures of the photovoltaic inverter. Then, the pictures with blurred nameplate information and the content that cannot be seen by the naked eye are deleted to obtain a clear nameplate picture of the photovoltaic inverter.

[0043] In the manual labeling stage, the PPOCRLabel tool is used to label the positions of the two targets, model and rated power, to obtain the real position and content of the target information, which are used as the labels corresponding to the nameplate images in the training set and stored in the format of a txt file. For the front-view images, a rectangular frame is used for labeling; for the side-view images, a multi-point rectangular frame is used for labeling.

[0044] The structural principle of the detection sub-model in the overall model in this embodiment is as follows: Figure 4 As shown, the detection sub-model obtains the predicted position of the target information in a manner including: firstly extracting features of the nameplate image input to the detection sub-model, and extracting features of the nameplate image input to the detection sub-model includes: using ResNet50 as a basic feature extraction network to extract features of the nameplate image input to the detection sub-model, and the basic feature extraction network includes 14 residual blocks, 1 global average pooling layer, and 1 fully connected layer, wherein each residual block consists of 2 convolutional layers and 1 skip connection. Such a design avoids the problem of gradient disappearance, allowing the network to maintain good training results while increasing depth; the extracted features are then input into the probability prediction branch and threshold prediction branch of the detection sub-model to obtain a probability map and a threshold map respectively, and the manner of inputting the extracted features into the probability prediction branch and the threshold prediction branch of the detection sub-model respectively includes: adjusting the size of the extracted features to be consistent with the size of the original nameplate image, and then inputting them into the probability prediction branch and the threshold prediction branch respectively, and adjusting the size of the extracted features to be consistent with the size of the original nameplate image is to ensure that the identified target position corresponds to the nameplate image input to the detection sub-model. For example, the size of the nameplate image input to the detection sub-model is MxN, and the size of the extracted features is QxP. If the feature size is not adjusted, the position predicted on the feature cannot correspond to the nameplate image. In this embodiment, the size is adjusted to the same as the original through convolution and deconvolution operations. Figure 1The probability prediction branch includes: 3 convolutional layers, 2 normalization layers, 2 ReLU activation functions, and 1 Sigmoid activation function; the threshold prediction branch includes: 1 convolutional layer, 2 normalization layers, 2 ReLU activation functions, 2 upsampling layers, and 1 Sigmoid activation function; the probability map represents the probability that each pixel belongs to the text, and its value is between 0 and 1; the threshold map represents the threshold of the probability that each pixel belongs to the text, and its value is between 0 and 1. When the probability that the pixel belongs to the text is greater than the threshold, the pixel is considered to belong to the text; when the probability that the pixel belongs to the text is less than the threshold, the pixel is considered not to belong to the text. Then, an approximate binary map is obtained based on the probability map and the threshold map to indicate whether the predicted position of each pixel point belongs to the target information position, and the probability map and the threshold map are differentiably binarized to obtain an approximate binary map, and the target information predicted position is obtained based on the approximate binary map; because the predicted probability map predicts each pixel point, the pixel point area predicted to belong to the target information position in the approximate binary map may be an irregular image; therefore, the method of obtaining the target information predicted position based on the approximate binary map includes: expanding the pixel point area predicted to belong to the target information position in the approximate binary map to obtain the coordinate position of the prediction box as the target information predicted position, such as Figure 5 shown.

[0045] The image portion of the target information predicted position is input into the recognition sub-model to obtain the text corresponding to the target information as the recognition result;

[0046] The structural principle of the identification sub-model of the overall model in this embodiment is as follows: Figure 6As shown, the method for the recognition sub-model to obtain the text corresponding to the target information includes: first extracting features from the image part of the target information prediction position of the input recognition sub-model, and extracting features from the image part of the target information prediction position of the input recognition sub-model includes: using the first three stages of PP-LCNet as the basic feature extraction network to extract features from the image part of the target information prediction position of the input recognition sub-model; the basic feature extraction network includes 1 convolution layer and 13 depth-separable convolution layers. The extracted features are input to the global feature mixing sub-module to extract mixed features. In the global feature mixing module, the extracted features are projected to three spaces through linear transformation to obtain q matrix, k matrix and v matrix, and the transpose of q matrix and k matrix is ​​matrix multiplied, and then the attention matrix is ​​obtained through the softmax activation function, and then the attention matrix and v matrix are matrix multiplied to obtain the output global mixed features. The global mixed feature is a more refined global feature. The global mixed feature is passed through a global pooling layer, a fully connected layer, and a nonlinear activation function to obtain a digital character feature sequence. The digital character feature is a feature with a height of 1 and a dimension of n (n represents the number of categories), which includes the result of predicting text classification. Linear prediction is then performed based on the obtained digital character feature sequence to output the text corresponding to the target information.

[0047] In addition, the loss function of the recognition sub-model is calculated based on the recognition result of the recognition sub-model and the actual content of the target information in the nameplate image; the loss function of the detection sub-model is calculated based on the predicted position of the target information obtained by the detection sub-model and the actual position of the target information in the nameplate image, wherein the loss function of the detection sub-model is the sum of the probability map loss, the threshold map loss and the approximate binary map loss.

[0048] S2: Update the detection sub-model and the recognition sub-model according to their loss functions respectively, and iterate S1-S2 until the iteration stop condition is met to complete the training.

[0049] The loss function of the detection sub-model is:

[0050] Overall loss (loss function of the detection sub-model) = probability map loss + threshold map loss + approximate binary map loss; Specifically, the probability map loss function used in this embodiment is:

[0051] BCE(x,y)=-[ylog(x)+(1-y)log(1-x)]

[0052] Among them, x represents the probability that the model predicts that a certain pixel belongs to the target position (the value is between 0 and 1), y represents the label of the pixel (0 means it does not belong to the target position, 1 means it belongs to the target position), and the log(x) method returns the natural logarithm of x. When y=1, we hope that the probability predicted by the model is as close to 1 as possible; when y=0, we hope that the probability predicted by the model is as close to 0 as possible; in general, the optimization goal is to make BCE(x,y) as close to 0 as possible. The optimization method is to adjust the parameters of each layer of the network in the model. Here, each network of the model can be regarded as a function, for example, f(x)=ax+b. In the process of iterative optimization, f(x) can be made close to 0 by adjusting the values ​​of coefficients a and b.

[0053] The function of the threshold map loss used in this embodiment is:

[0054] L1(y*,x*)=|y*-x*|

[0055] Where x* represents the predicted threshold of a certain pixel, and y* represents the true threshold of the pixel. It has an initial value and is then adjusted as the network is continuously optimized; the |y*-x*| method returns the absolute values ​​of the two parameters. In the process of iterative optimization, it is usually hoped that the predicted threshold of the model is as close to the true threshold as possible, that is, the loss function L1(y*,x*) is as close to 0 as possible.

[0056] The approximate binary image loss function used in this embodiment is:

[0057] L(X,Y)=1-2|X∩Y| / |X|+|Y|

[0058] Where X represents the range of the prediction box generated according to the prediction probability map (see the prediction box generation Figure 5 ), the number of pixels that are predicted to belong to the target position, and Y represents the number of pixels that belong to the target position within the prediction box. In the iterative optimization process, the optimization goal is also to make L(X,Y) as close to 0 as possible.

[0059] The loss function of the recognition sub-model used in this embodiment is:

[0060] L2(x, y) = -∑ n ylog(x)

[0061] Where y represents the true label (0 or 1), and x represents the probability of being predicted as that category. For example, when the prediction result is [0.5, 0.3, 0.2], the loss is L2 = -1*log(0.5)+(-0*log(0.3))+(-0*log(0.2)); during the iterative optimization process, the optimization goal is also to make L2(X,Y) as close to 0 as possible.

[0062] The above-mentioned probability map loss, threshold map loss, approximate binary map loss, and loss functions of the recognition sub-model are all existing loss functions. The iteration stops when the iteration stop condition is met by adjusting the parameters in the loss function (usually when the iteration reaches a set number of times or the iteration result makes the value of the loss function decrease below the set threshold). Specifically, since the loss functions corresponding to the probability map loss, threshold map loss, and approximate binary map loss are all existing technologies, and the corresponding methods of optimizing and adjusting the detection sub-model or the recognition sub-model through the calculation results of each loss function are also existing technologies, no further elaboration is required here.

[0063] Taking the nameplate image of a photovoltaic inverter as an example, after the image is input into the overall model, the recognition process is as follows: Figure 7 shown.

Claims

1. A method for identifying nameplate information, characterized in that: Input the nameplate image to be identified into the trained overall model including the detection sub-model and the recognition sub-model to obtain the text corresponding to the target information; The training steps of the overall model are as follows: S1: Input the nameplate images in the training set into the detection sub-model to obtain the predicted position of the target information; Inputting the image portion of the target information predicted position into the recognition sub-model to obtain the text corresponding to the target information as the recognition result; The loss function of the recognition sub-model is calculated based on the recognition result of the recognition sub-model and the real content of the target information in the nameplate image; The loss function of the detection sub-model is calculated based on the predicted position of the target information obtained by the detection sub-model and the actual position of the target information in the nameplate image; S2: Update the detection sub-model and the recognition sub-model according to their loss functions respectively, and iterate S1-S2 until the iteration stop condition is met to complete the training.

2. The nameplate information recognition method according to claim 1, characterized in that: The detection sub-model obtains the predicted position of the target information in the following manner: First, feature extraction is performed on the nameplate image input to the detection sub-model, and then the extracted features are respectively input into the probability prediction branch and the threshold prediction branch of the detection sub-model to obtain a probability map and a threshold map respectively, and then an approximate binary map is obtained based on the probability map and the threshold map to indicate whether the position of each predicted pixel point belongs to the target information position, and the predicted position of the target information is obtained based on the approximate binary map.

3. The nameplate information recognition method according to claim 1, characterized in that: The loss function of the detection sub-model is the sum of the probability map loss, the threshold map loss and the approximate binary map loss.

4. The nameplate information recognition method according to claim 1, characterized in that: The training set includes nameplate images and labels corresponding to the nameplate images; the labels corresponding to the nameplate images include the actual content of the target information in the nameplate images and the actual position of the target information in the nameplate images.

5. The nameplate information recognition method according to claim 4, characterized in that: The nameplate image and the label corresponding to the nameplate image correspond to each other through the image name of the nameplate image.

6. The nameplate information recognition method according to claim 2, characterized in that: The method of extracting features of the nameplate image input to the detection sub-model includes: using ResNet50 as a basic feature extraction network to extract features of the nameplate image input to the detection sub-model.

7. The nameplate information recognition method according to claim 2, characterized in that: The method of obtaining the predicted position of the target information according to the approximate binary image includes: expanding the pixel point area predicted to belong to the target information position in the approximate binary image to obtain the coordinate position of the prediction box as the predicted position of the target information.

8. The nameplate information recognition method according to claim 1, characterized in that: The method for the recognition sub-model to obtain the text corresponding to the target information includes: firstly performing feature extraction on the image portion of the target information prediction position input into the recognition sub-model, inputting the extracted features into the global feature mixing sub-module, and extracting the global mixed features; passing the global mixed features through the global pooling layer, the fully connected layer and the nonlinear activation function to obtain a digital character feature sequence, and then performing linear prediction based on the obtained digital character feature sequence to output the text corresponding to the target information.

9. The nameplate information recognition method according to claim 8, characterized in that: The method of extracting features from the image portion of the target information prediction position of the input recognition sub-model includes: using the first three stages of PP-LCNet as the basic feature extraction network to extract features from the image portion of the target information prediction position of the input recognition sub-model.

10. The nameplate information recognition method according to claim 2, characterized in that: The method of inputting the extracted features into the probability prediction branch and the threshold prediction branch of the detection sub-model respectively includes: adjusting the size of the extracted features to be consistent with the size of the original nameplate image, and then inputting them into the probability prediction branch and the threshold prediction branch respectively.

Images