Method and system for automatic recognition of asset nameplate characters

Abstract

The present invention relates to the technical field of intelligent character recognition, and specifically discloses a method and system for automatic recognition of asset nameplate characters, the method comprising the steps of: S1, acquiring a nameplate image of an asset device; S2, preprocessing the acquired nameplate image and inputting it into a nameplate character automatic recognition network; S3, the nameplate character automatic recognition network outputs the recognized characters. The present invention provides a method and system for automatic recognition of asset nameplate characters, which overcomes the inaccuracy of the reading process caused by factors such as the position of a handheld device and ambient lighting, and improves the recognition accuracy and robustness of indoor and outdoor asset nameplate characters. The method and system can be transplanted to a mobile terminal, integrated into applications related to physical inventory of assets, and improve the efficiency of indoor and outdoor asset inventory and the technical capabilities of automatic auditing.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent character recognition, and in particular to a method and system for automatically recognizing characters on asset nameplates. Background Art

[0002] At present, the informatization construction of asset inventory has just started, and there are many cases of discrepancies between accounts and actual assets. In addition, the inventory of large-scale equipment assets needs to be carried out on-site one by one, which has the key problems of complex varieties and easy errors in manual identification. For example, multiple devices belong to an integrated device, and one device contains other devices. If there are omissions or errors, it is necessary to go to the site for manual investigation again. Large-scale equipment is extensive and large in quantity, which not only consumes a lot of manpower, material and financial resources, but also has a long operation and maintenance cycle and low efficiency. Human factors can easily affect the accuracy and objectivity of the results, which seriously restricts the activation of existing assets and reduces the supervision of these large equipment and the energy-saving and efficiency-enhancing effects. In order to efficiently conduct the account-to-actual verification of asset inventory, the automatic reading and recognition of equipment nameplate characters to assist on-site verification and entry has become an important means of enterprise asset inventory.

[0003] In actual engineering applications, since the nameplates of outdoor equipment are exposed to the natural environment for a long time, the nameplate characters often have blurred backgrounds such as scratches and rust; secondly, there are large individual differences in the viewing angles of handheld terminals to read the nameplates. At the same time, the lighting backgrounds of indoor and outdoor nameplates are very different. In view of the above engineering factors, existing methods usually use geometric affine deformation methods for regional correction. Since the pixels are changed through geometric positions, the pixels on the integer grid may appear on non-integer grids after being transformed by this method. Therefore, the concave image pixels transformed by this method are difficult to index. First, the discontinuous insertion of floating-point coordinate pixels makes it inappropriate to perceive; on the other hand, the increase in pixels increases the cost of computing resources. Secondly, after compressing the image, the boundary shape of this method becomes singular, which makes it difficult to decode during the transmission process; finally, this method is highly dependent on the selection of concave points in the image. The current methods all set the concave points in the center of the image, and once other positions are selected, this method will have a very poor effect. Therefore, the idea based on geometric shape transformation is difficult to apply to the handheld nameplate character recognition system in natural scenes, resulting in the deterioration of the accuracy and robustness of the existing nameplate character automatic recognition method. Summary of the invention

[0004] The present invention provides an asset nameplate character automatic recognition method and system, which solve the technical problem that the existing nameplate character automatic recognition method has poor accuracy and robustness.

[0005] In order to solve the above technical problems, the present invention provides a method for automatically recognizing characters on asset nameplates, comprising the steps of:

[0006] S1. Obtain the nameplate image of the asset equipment;

[0007] S2, pre-processing the acquired nameplate image and inputting it into the nameplate character automatic recognition network;

[0008] The nameplate character automatic recognition network includes a sequentially connected convolution layer, a popular learning layer, a fully connected layer and an output layer; a Gabor filter is introduced in the convolution layer, and the convolution kernel function of the convolution layer is:

[0009]

[0010]

[0011] Wherein, x, y represent the horizontal and vertical coordinates of the input feature map, x', y' represent the x, y coordinates after filtering by the Gabor filter, θ is the direction of the Gabor filter, γ represents the aspect ratio of the Gaussian function, λ represents the wavelength of the modulation function, σ represents the covariance of the Gaussian function, and exp represents an exponential function with the natural constant e as the base;

[0012] S3. The nameplate character automatic recognition network outputs the recognized characters.

[0013] Furthermore, the nameplate character automatic recognition network is provided with 13 convolutional layers, the spatial scale of the 13 convolutional layers starts from 3×3, with a step size of 2 pixels, and a maximum scale of 27×27; the λ of the 13 convolutional layers gradually increases from 3.0 to a maximum of 15.4; the σ of the 13 convolutional layers gradually increases from 2.4 to a maximum of 12.3.

[0014] Furthermore, the spatial scales, λ, and σ of the 13 convolutional layers are:

[0015] 3×3, λ=3.0, σ=2.4;

[0016] 5×5, λ=3.3, σ=2.6;

[0017] 7×7, λ=3.5, σ=2.8;

[0018] 9×9, λ=4.6, σ=3.6;

[0019] 11×11, λ=5.6, σ=4.5;

[0020] 13×13, λ=6.8, σ=5.4;

[0021] 15×15, λ=7.9, σ=6.3;

[0022] 17×17, λ=9.1, σ=7.3;

[0023] 19×19, λ=10.3, σ=8.2;

[0024] 21×21, λ=11.5, σ=9.2;

[0025] 23×23, λ=12.7, σ=10.2;

[0026] 25×25, λ=14.1, σ=11.3;

[0027] 27×27, λ=15.4, σ=12.3.

[0028] Furthermore, the processing steps of the popular learning layer include:

[0029] S21, for each point in the input feature map having N points, use the Euclidean distance to search for its nearest K neighboring points to form a K×N adjacency matrix A, where the column vector represents the distance from the corresponding point to the K neighboring points;

[0030] S22. For each point X i and its adjacency matrix A, calculate the reconstruction coefficient W ij , so that the cost function J(W) of the following formula is minimized:

[0031]

[0032] The constraints of the above formula are:

[0033]

[0034] S23, calculate each point X i The low-dimensional vector Y in the embedding space i , so that the cost function of the following formula is minimized:

[0035]

[0036] The constraints of the above formula are: trace() means to find the matrix trace, Y is Y from 1 to N i The matrix composed of W is from 1 to N. ij The matrix is ​​composed of, I is the unit matrix; T represents the matrix transpose; || || represents the matrix norm.

[0037] Furthermore, the fully connected layer is an LSTM network, and the number of hidden layers is set to 8; the output layer calculates the character feature matching probability through a softmax function, and takes the character with the maximum probability of matching as the recognition result.

[0038] Furthermore, the nameplate character automatic recognition network adopts the Tanh function as the activation function, adopts the Adam optimization algorithm for optimization, and adopts the cross entropy as the loss function.

[0039] Furthermore, the step S1 specifically includes the steps of:

[0040] S11, when shooting an image, generating a rectangular frame, guiding the user to place the nameplate characters in the rectangular frame, and capturing the image in the rectangular frame as the nameplate image;

[0041] S12, calculate the maximum value, minimum value, variance and gradient of the pixels in the nameplate image, and determine whether the nameplate image is overexposed or underexposed based on the maximum value and minimum value, and determine whether the area is accurately focused based on the variance and gradient. If there is overexposure or underexposure, or the focus is not accurate, it is determined that the nameplate image is not compliant, and the process returns to step S11 to reshoot. Otherwise, the nameplate image is saved and the process proceeds to step S2.

[0042] Furthermore, in the step S2, the nameplate image is preprocessed, specifically, the nameplate image is processed by Gaussian low-pass filtering and adaptive local histogram equalization.

[0043] The present invention also provides an asset nameplate character automatic recognition system, the key of which is: comprising a nameplate image generation unit, an image preprocessing unit and a nameplate automatic recognition unit;

[0044] The nameplate image generating unit, the image preprocessing unit and the nameplate automatic identifying unit are used to respectively execute steps S1, S2 and S3 described in the above method.

[0045] The present invention provides an asset nameplate character automatic recognition method and system, which can automatically recognize the nameplate image by acquiring the nameplate image and preprocessing it, and designing a nameplate character automatic recognition network to automatically recognize the nameplate image, thereby overcoming the inaccuracy of the reading process caused by factors such as the position of the handheld device and the ambient light, and improving the recognition accuracy and robustness of the indoor and outdoor asset nameplate characters. The method and system can also be transplanted to mobile terminals and integrated into applications related to physical inventory of assets, thereby improving the efficiency of indoor and outdoor asset inventory and the automatic auditing technology capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a flow chart of a method for automatically recognizing characters on an asset nameplate provided by an embodiment of the present invention;

[0047] Figure 2 is an architecture diagram of a nameplate character automatic recognition network provided by an embodiment of the present invention;

[0048] Figure 3is a parameter design diagram of a convolutional layer provided by an embodiment of the present invention;

[0049] Figure 4 It is a structural diagram of an asset nameplate character automatic recognition system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The following specifically illustrates the implementation mode of the present invention in conjunction with the accompanying drawings. The embodiments are provided for illustrative purposes only and cannot be understood as limiting the present invention. The accompanying drawings are provided for reference and illustration only and do not constitute a limitation on the scope of patent protection of the present invention, because many changes can be made to the present invention without departing from the spirit and scope of the present invention.

[0051] Example 1

[0052] The embodiment of the present invention provides a method for automatically recognizing characters on asset nameplates. Figure 1 As shown, the steps include:

[0053] S1. Obtain the nameplate image of the asset equipment;

[0054] S2, pre-processing the acquired nameplate image and inputting it into the nameplate character automatic recognition network;

[0055] S3. The nameplate character automatic recognition network outputs the recognized characters.

[0056] (1) Regarding step S1

[0057] Step S1 specifically includes the following steps:

[0058] S11, when shooting an image, generating a rectangular frame, guiding the user to place the nameplate characters in the rectangular frame, and capturing the image in the rectangular frame as the nameplate image;

[0059] S12, calculate the maximum value, minimum value, variance and gradient of the pixels in the nameplate image, and determine whether the nameplate image is overexposed or underexposed based on the maximum value and minimum value, and determine whether the area is accurately focused based on the variance and gradient. If there is overexposure or underexposure, or the focus is not accurate, the nameplate image is determined to be unqualified, and the process returns to step S11 to reshoot. Otherwise, the nameplate image is saved and the process proceeds to step S2.

[0060] In step S12, when the maximum value of the pixels in the nameplate image is greater than 210, it is considered to be overexposed. When the minimum value of the pixels in the nameplate image is less than 30, it is considered to be underexposed. The average variance and gradient numerical distribution of the focus and defocus areas of the test object can be used as the threshold for judging the image focus.

[0061] Steps S11 and S12 can effectively avoid the problem of poor robustness caused by the large difference in indoor and outdoor lighting.

[0062] (2) Regarding step S2

[0063] In step S2, the nameplate image is preprocessed, specifically: the nameplate image is processed by Gaussian low-pass filtering and adaptive local histogram equalization. Such processing can solve the problem of equal emphasis on light and shade for the same outdoor nameplate.

[0064] (3) Regarding step S3

[0065] like Figure 2 As shown, the nameplate character automatic recognition network includes sequentially connected convolutional layers, popular learning layers, fully connected layers and output layers.

[0066] The Gabor filter is introduced in the convolution layer, and the convolution kernel function of the convolution layer with the Gabor filter is:

[0067]

[0068]

[0069] Among them, x, y represent the horizontal and vertical coordinates of the input feature map, x', y' represent the x, y coordinates after filtering by the Gabor filter, θ is the direction of the Gabor filter, γ represents the aspect ratio of the Gaussian function (usually γ = 0.3), λ represents the wavelength of the modulation function, σ represents the covariance of the Gaussian function, and exp represents the exponential function with the natural constant e as the base. θ is the direction of the filter. By rotating the coordinate system, filtering in any direction can be achieved, thereby extracting the significant features of the nameplate at different inclination angles, so that the character features at different scales can be fully visible. Select θ = 0°, 45°, 90°, 135°.

[0070] The convolution layer extracts the salient features of the nameplate image through convolution operations between the convolution kernel and the input image. The number of convolution layers is set to 13, and the convolution layer introduces Gabor filters. The Gabor function can be expressed as a dual function that alternates the feature positioning of the biological receptive field and its spatial frequency selectivity. It has sensitive directional selectivity. By setting kernel operations in different directions, effective perception of nameplates under different spatial tilts can be achieved. The convolution layer solves the problem that the existing regional correction method cannot take into account the correction effects of angle tilt and viewing angle tilt at the same time.

[0071] like Figure 3 As shown in the figure, the spatial scale of the 13 convolutional layers starts from 3×3, with a step size of 2 pixels and a maximum scale of 27×27; the λ of the 13 convolutional layers gradually increases from 3.0 to a maximum of 15.4; the σ of the 13 convolutional layers gradually increases from 2.4 to a maximum of 12.3. More specifically, the spatial scale, λ and σ of the 13 convolutional layers are:

[0072] 3×3, λ=3.0, σ=2.4;

[0073] 5×5, λ=3.3, σ=2.6;

[0074] 7×7, λ=3.5, σ=2.8;

[0075] 9×9, λ=4.6, σ=3.6;

[0076] 11×11, λ=5.6, σ=4.5;

[0077] 13×13, λ=6.8, σ=5.4;

[0078] 15×15, λ=7.9, σ=6.3;

[0079] 17×17, λ=9.1, σ=7.3;

[0080] 19×19, λ=10.3, σ=8.2;

[0081] 21×21, λ=11.5, σ=9.2;

[0082] 23×23, λ=12.7, σ=10.2;

[0083] 25×25, λ=14.1, σ=11.3;

[0084] 27×27, λ=15.4, σ=12.3. The above parameters are the ones with the best effect after verification.

[0085] Furthermore, this example sets a popular learning layer, which reduces the high-dimensional data of the same character to a low-dimensional space with a popular distribution structure, greatly reducing the feature impact caused by nameplate rotation, scratches, occlusion and rust, improving the character feature expression ability, and facilitating the network's character clustering detection effect and text recognition accuracy under complex backgrounds. The processing steps of the popular learning layer include:

[0086] S21, for each point in the input feature map having N points, use the Euclidean distance to search for its nearest K neighboring points to form a K×N adjacency matrix A, where the column vector represents the distance from the corresponding point to the K neighboring points;

[0087] S22. For each point X i and its adjacency matrix A, calculate the reconstruction coefficient W ij , so that the cost function J(W) of the following formula is minimized:

[0088]

[0089] The constraints of the above formula are:

[0090]

[0091] S23, calculate each point X i The low-dimensional vector Y in the embedding space i , so that the cost function of the following formula is minimized:

[0092]

[0093] The constraints of the above formula are: trace() means to find the matrix trace (to find the sum of the diagonal elements of the matrix, which is also the sum of the eigenvalues), Y is Y from 1 to N i The matrix composed of W is from 1 to N. ij The matrix is ​​composed of, I is the unit matrix; T represents the matrix transpose; |||| represents the matrix norm.

[0094] In step S23, by performing M=(IW) T The eigenvalue decomposition of (IW) yields Y i , that is, let the smallest d non-zero eigenvalues ​​of M and the corresponding eigenvectors be <λ i ,v i >,i=1,…,d, then the jth component of the i-th point embedding is Among them, v ji Yes j The eigenvector dimension is set to 25.

[0095] The fully connected layer is an LSTM network (Long Short-Term Memory Artificial Neural Network), and the number of hidden layers is set to 8; the output layer calculates the character feature matching probability through the softmax function, and the character with the maximum probability of matching is used as the recognition result. The nameplate character automatic recognition network uses the Tanh function as the activation function, the Adam optimization algorithm for optimization, and the cross entropy as the loss function. Taking the tower assets as an example, its character target library consists of common characters such as "site, base station, communication equipment, switching equipment, optical cable line, router, transmitter, receiver, model, number, manufacturer, specification..."

[0096] In the automatic recognition of nearly 500 asset information of the tower company, the overall accuracy of various character recognition methods can reach 90.35%.

[0097] This example integrates four core modules, namely feature extraction, feature expression, text detection and character recognition, into a deep network, making the nameplate recognition network model more lightweight and convenient for the architecture and deployment of the nameplate recognition system. Traditional nameplate character recognition requires nameplate area segmentation and correction operations, but this method does not require the above steps, and directly extracts the deep features of the nameplate under viewing angle tilt, position rotation and complex background conditions, thus achieving effective recognition of nameplate characters in complex scenes.

[0098] The embodiment of the present invention provides an automatic identification method for asset nameplate characters, which can automatically identify nameplate images by acquiring nameplate images and preprocessing them, and designing a nameplate character automatic identification network to automatically identify nameplate images, thereby overcoming the inaccuracy of the reading process caused by factors such as the position of the handheld device and ambient lighting, and improving the recognition accuracy and robustness of indoor and outdoor asset nameplate characters. The method and system can also be transplanted to mobile terminals and integrated into applications related to physical inventory of assets, thereby improving the efficiency of indoor and outdoor asset inventory and the technical capabilities of automatic auditing.

[0099] Example 2

[0100] like Figure 4 As shown, an embodiment of the present invention provides an asset nameplate character automatic recognition system, a nameplate image generation unit, an image preprocessing unit and a nameplate automatic recognition unit, wherein the nameplate image generation unit is integrated on a portable handheld device, and the image preprocessing unit and the nameplate automatic recognition unit are integrated on a cloud computing and storage device. The nameplate image generation unit, the image preprocessing unit and the nameplate automatic recognition unit are respectively used to execute steps S1, S2, and S3 described in the method of Example 1, and the specific implementation process will not be repeated in this example. In other embodiments, depending on different application requirements, the nameplate image generation unit, the image preprocessing unit and the nameplate automatic recognition unit can be integrated into the device in other ways.

[0101] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A method for automatically recognizing characters on asset nameplates, characterized in that: include: Step S1, obtaining a nameplate image of an asset device; the step S1 specifically comprises the following steps: Step S11, when shooting an image, a rectangular frame is generated, the user is guided to place the nameplate characters in the rectangular frame, and the image in the rectangular frame is captured as the nameplate image; Step S12, calculating the maximum value, minimum value, variance and gradient of the pixels in the nameplate image, and determining whether the nameplate image is overexposed or underexposed according to the maximum value and minimum value, and determining whether the area is accurately focused according to the variance and gradient. If there is overexposure or underexposure, or the focus is not accurate, it is determined that the nameplate image does not conform, and the process returns to step S11 to reshoot, otherwise the nameplate image is saved and the process proceeds to step S2; Step S2, pre-processing the acquired nameplate image and inputting it into the nameplate character automatic recognition network; in the step S2, the nameplate image is pre-processed, specifically: the nameplate image is processed by Gaussian low-pass filtering and adaptive local histogram equalization; The nameplate character automatic recognition network includes a sequentially connected convolution layer, a popular learning layer, a fully connected layer and an output layer; a Gabor filter is introduced in the convolution layer, and the convolution kernel function of the convolution layer is: ； ； in, x , y Represents the horizontal and vertical coordinates of the input feature map, x' , y' Represents the Gabor filter after filtering x , y coordinate, θ is the direction of the Gabor filter, θ =0°,45°,90°,135°, γ represents the aspect ratio of the Gaussian function, λ represents the wavelength of the modulation function, σ represents the standard deviation of the Gaussian function, and exp represents the exponential function with the natural constant e as the base; The nameplate character automatic recognition network is provided with 13 convolutional layers, the spatial scale of the 13 convolutional layers starts from 3×3, with a step size of 2 pixels, and the maximum scale is 27×27; λ It gradually increases from 3.0 to a maximum of 15.4; 13 convolutional layers σ It gradually increases from 2.4 to a maximum of 12.3; The spatial scale of the 13 convolutional layers, λ and σ They are: 3×3、 λ =3.0、 σ =2.4； 5×5、 λ =3.3、 σ =2.6； 7×7、 λ =3.5、 σ =2.8； 9×9、 λ =4.6、 σ =3.6； 11×11、 λ =5.6、 σ =4.5； 13×13、 λ =6.8、 σ =5.4； 15×15、 λ =7.9、 σ =6.3； 17×17、 λ =9.1、 σ =7.3； 19×19、 λ =10.3、 σ =8.2； 21×21、 λ =11.5、 σ =9.2； 23×23、 λ =12.7、 σ =10.2； 25×25、 λ =14.1、 σ =11.3； 27×27、 λ =15.4、 σ =12.3； The nameplate character automatic recognition network uses the Tanh function as the activation function, uses the Adam optimization algorithm for optimization, and uses the cross entropy as the loss function; S3. The nameplate character automatic recognition network outputs the recognized characters.

2. The method for automatically recognizing characters on asset nameplates according to claim 1, characterized in that: The processing steps of the popular learning layer include: Step S21: N Each point in the input feature map of points uses the Euclidean distance to search for its nearest K neighboring points, forming K × N The adjacency matrix of A , where the column vector represents the corresponding point to K The distance between the nearest neighbors; Step S22: for each point X i and its adjacency matrix A , calculate the reconstruction coefficient W ij , so that the cost function of the following formula J ( W )Minimum: ； The constraints of the above formula are: ； Step S23: Calculate each point X i Low-dimensional vectors in the embedding space Y i , so that the cost function of the following formula is minimized: ； The constraints of the above formula are: ; trace() means to find the matrix trace, Y From 1 to N Y i The matrix composed of W From 1 to N W ij The matrix composed of I is the identity matrix; T represents matrix transpose; || || represents matrix norm.

3. The method for automatically recognizing characters on an asset nameplate according to claim 1, characterized in that: The fully connected layer is an LSTM network, and the number of hidden layers is set to 8; the output layer calculates the character feature matching probability through the softmax function, and takes the character with the maximum probability matching as the recognition result.

4. An automatic identification system for asset nameplate characters, characterized in that: It includes a nameplate image generating unit, an image preprocessing unit and a nameplate automatic recognition unit; The nameplate image generating unit, the image preprocessing unit and the nameplate automatic recognition unit are used to execute steps S1, S2 and S3 described in any one of claims 1 to 3 respectively.

Images