A sign classification method based on deep learning

Abstract

The present invention relates to the technical field of power sign classification, and proposes a sign classification method based on deep learning. The method automatically identifies the types of signboards through a deep learning model, realizes fast and accurate classification, significantly reduces the time required for manual identification, and improves work efficiency. The method uses a trained neural network model to extract multi-level features of signboards, and can maintain a high recognition accuracy rate for signboards of different sizes, texts, and image styles, effectively reducing the error rate in manual classification. The method combines a robotic arm and an electric suction cup to realize automatic grabbing, classification, and fixed placement of signboards, greatly reducing manual operations and reducing labor intensity. The method uses a classification model and a path planning algorithm to work together to ensure that the signboards are accurately placed in the corresponding classification buckets after classification, effectively improving the management standardization of the signboards.

Description

Technical Field

[0001] The present invention relates to the technical field of power sign classification, and in particular to a sign classification method based on deep learning. Background Art

[0002] At present, in the operation and maintenance of substations, signboards are important tools for clarifying safety warnings, operating instructions and equipment status indications, and are widely used in power facilities and equipment to ensure safe operation. They are usually set up near power equipment, substations, distribution boxes, poles and other conspicuous locations to ensure that personnel can clearly see the content of the signboards. The design and use of power signboards must comply with relevant regulations and standards, such as ISO, NSI or national power department regulations, to ensure their effectiveness and consistency.

[0003] In actual use, the existing signboard recycling management has the following problems:

[0004] The existing signboards are of different sizes, and contain different texts and images. They need to be manually identified, classified and placed when used and recycled. Monthly maintenance requires manual counting of the inventory and checking whether they are placed correctly, resulting in low work efficiency, time-consuming and labor-intensive.

[0005] Based on this, the present invention provides a sign classification method based on deep learning to solve the above problems, reduce the labor intensity of staff, and improve the efficiency and accuracy of sign management. Summary of the invention

[0006] The purpose of the present invention is to solve the shortcomings of the prior art and provide a sign classification method based on deep learning, so as to reduce the labor intensity of staff and improve the efficiency and accuracy of sign management.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] A sign classification method based on deep learning includes the following steps:

[0009] Step S1, obtaining a sample set of original sample images of power signboards;

[0010] Step S2, preprocessing the original sample image sample set of the power sign obtained in step S1, including normalization, size adjustment, grayscale processing, feature enhancement and denoising;

[0011] Step S3, establishing a neural network classification model;

[0012] Step S4, training the classification model established in step S3 based on the data preprocessed in step S2;

[0013] Step S5, input the power sign image to be classified into the classification model trained in step S4, extract image features through the trained classification model, match them with the target category, and output the category label and classification confidence of the sign.

[0014] Preferably, in step S1, a wide-angle camera is used to collect original sample images of power signboards under various lighting conditions to increase sample diversity; after collecting the original sample images of power signboards, a category label is manually annotated for each image.

[0015] Preferably, the step S2 specifically includes the following sub-steps:

[0016] Step S2.1, normalization, scaling the image pixel values ​​to the interval [0, 1], the formula is as follows:

[0017]

[0018] Where I(x, y) is the pixel value of the original image at position (x, y); min(I) and max(I) are the minimum and maximum pixel values ​​of the image, respectively; I′(x, y) is the normalized pixel value;

[0019] Step S2.2, resizing, resizing the image to a fixed size of 640×640 while maintaining the aspect ratio of the original image;

[0020] Step S2.3, grayscale processing, using weighted average method to convert RGB image into grayscale image to reduce calculation requirements;

[0021] Step S2.4, feature enhancement, randomly rotate and scale the image to improve data diversity;

[0022] Step S2.5, denoising, uses Gaussian filtering to remove random noise, and the filter weight is calculated by the following formula:

[0023]

[0024] Among them, G(x, y) is the weight value of the filter at position (x, y); σ is the standard deviation, ranging from 1 to 2; x, y are the distances between the filter position and the center point; and the filter size is 5×5.

[0025] Preferably, in step S2.4, the random rotation angle range is from -15° to 15°, and the random scaling ratio range is from 90% to 110%.

[0026] Preferably, the step S3 specifically includes the following sub-steps:

[0027] Step S3.1, convolutional layer extracts features, and multi-level image features are extracted through filters. The formula is as follows:

[0028]

[0029] in, is the value of the feature map position (i, j) at the kth channel; X i+m，j+n is the pixel value at (i+m, j+n) in the input image; is the weight matrix of the convolution kernel; b (k) is the bias value of the kth convolution kernel; M, N are the sizes of the convolution kernel;

[0030] Step S3.2, activation function, using ReLU activation function to introduce nonlinear expression capability;

[0031] Step S3.3, pooling operation, maximum pooling is used to reduce dimension and extract key features. The formula is as follows:

[0032]

[0033] Among them, P i，j is the value of position (i, j) after pooling; S is the pooling window range;

[0034] Step S3.4, output layer, uses the Softmax activation function to convert the network output into a probability distribution, and outputs the probability distribution of each category to the subsequent modules for category determination and result display.

[0035] Preferably, the convolution kernel size is 3×3, and the step size is 1; the pooling window size is 2×2, and the step size is 2.

[0036] Preferably, the step S4 specifically includes the following sub-steps:

[0037] Step S4.1, data segmentation, dividing the original sample image set of the power sign into a training set, a validation set and a test set, with a ratio of 8:1:1;

[0038] Step S4.2, set the loss function, the loss function formula is as follows:

[0039]

[0040] Where N is the number of samples; C is the number of categories; y ij is the true category label (1 or 0); is the probability of the model output;

[0041] Step S4.3, use the Adam optimization algorithm to update the weights. The formula of the Adam optimization algorithm is as follows:

[0042]

[0043] Among them, θ t is the current weight; α is the learning rate; m t and v t are momentum and second-order momentum estimates, respectively; ∈ is a small value to prevent the denominator from being zero;

[0044] Step S4.4, learning rate adjustment, dynamically adjusting the learning rate through an exponential decay strategy;

[0045] In step S4.5, the accuracy is calculated using the validation set. When the validation loss decreases by less than 0.01 for five consecutive rounds or the validation set accuracy is greater than 95, training is stopped.

[0046] Preferably, the step S5 further comprises the following steps:

[0047] Step S6, placing the classified power signboards into corresponding classification barrels through a robotic arm in cooperation with an electric suction cup.

[0048] Preferably, the step S6 specifically includes the following sub-steps:

[0049] Step S6.1: Determine the target location, and query the physical location coordinates (x i ,y i , z i );

[0050] The target position is determined by the following formula:

[0051] P arm =T camerato-arm ·P camera ;

[0052] Among them, P arm is the target position in the robot coordinate system; P camera is the position of the sign in the camera coordinate system; T camerato-arm is the coordinate transformation matrix from camera to robotic arm;

[0053] Step S6.2: Path planning, using a fast random tree algorithm to generate a collision-free optimal path for the robot arm from the current position to the target classification bucket position;

[0054] Step S6.3, the robotic arm moves the electric suction cup according to the optimal path to grab the sign through negative pressure adsorption and places it stably in the corresponding classification bucket.

[0055] The present invention discloses a sign classification method based on deep learning, which has the following beneficial effects.

[0056] Through deep learning models, the types of signboards can be automatically identified, and fast and accurate classification can be achieved, which significantly reduces the time required for manual identification and improves work efficiency. The trained neural network model is used to extract the multi-level features of signboards, and high recognition accuracy can be maintained for signboards of different sizes, texts and image styles, effectively reducing the error rate in manual classification. The robotic arm and electric suction cup are combined to realize the automatic grabbing, classification and fixed placement of signboards, which greatly reduces manual operations and reduces labor intensity. Through the collaborative work of the classification model and the path planning algorithm, it is ensured that the signboards are accurately placed in the corresponding classification buckets after classification, effectively improving the management standardization of signboards. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 Schematic diagram of the process of the sign classification method based on deep learning of the present invention.

[0058] Figure 2 This is a schematic diagram of the process in Example 2 of the present invention. DETAILED DESCRIPTION

[0059] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0060] Reference to "embodiment" herein means that the specific features, structures or characteristics described in conjunction with the embodiment may be included in at least one embodiment of the present application. The term "embodiment" appearing in various places in the specification does not necessarily refer to the same embodiment, nor does it particularly limit its independence or association with other embodiments. In principle, in the present application, as long as there is no technical contradiction or conflict, the various technical features mentioned in the embodiments can be combined in any way to form a corresponding implementable technical solution.

[0061] Unless otherwise defined, the technical terms used in this document have the same meanings as those generally understood by those skilled in the art to which this application belongs; the use of relevant terms in this document is only for describing specific embodiments and is not intended to limit this application.

[0062] In the description of this application, the term "and / or" is an expression used to describe the logical relationship between objects, indicating that three relationships may exist, for example, A and / or B, which means: A exists, B exists, and A and B exist at the same time. In addition, the character " / " in this article generally indicates that the objects before and after are in an "or" logical relationship.

[0063] Unless otherwise expressly specified or limited, in the description of the embodiments of the present application, the terms such as "install", "connect", "connect", "fix", "set", etc. used should be understood in a broad sense. For example, the "connection" can be a fixed connection, a detachable connection, or an integrated setting; it can be a mechanical connection, an electrical connection, or a communication connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium; it can be the internal connection of two elements or the interaction relationship between two elements. For technicians in the technical field to which the present application belongs, the specific meanings of the above terms in the embodiments of the present application can be understood according to the specific circumstances.

[0064] Example 1

[0065] A sign classification method based on deep learning includes the following steps:

[0066] Step S1, obtaining a sample set of original sample images of power signboards;

[0067] Preferably, in step S1 of this embodiment, a wide-angle camera is used to collect original sample images of power signboards under various lighting conditions to increase sample diversity. In this embodiment, as an example, a wide-angle camera with a viewing angle of not less than 120° is used to collect signboard images. The following scenes are covered during collection:

[0068] Different lighting conditions: including strong light, weak light, shadow, night lighting and other conditions to ensure the lighting diversity of the data;

[0069] Different angles and distances: The images are collected at an angle of ±30° within a distance range of 1.5m to 3m, simulating the diverse viewing angles of the sign in actual use.

[0070] Multiple signboard types: Covering 7 types of commonly used signboards, manually annotate each image with a category label after collection; in this embodiment, the 7 types of commonly used signboards are respectively working here, entering and exiting from here, going up and down from here, prohibiting the closing of the circuit breaker when someone is working, prohibiting the closing of the circuit breaker when someone is working, primary equipment of the cross-operation live isolation point, and secondary equipment of the cross-operation live isolation point. It should be understood that the types of signboards can be expanded or reduced by themselves. After the collection is completed, the signboard image is stored in the sample database and classified according to the text content of the signboard.

[0071] Step S2, preprocessing the original sample image sample set of the power sign obtained in step S1, including normalization, size adjustment, grayscale processing, feature enhancement and denoising;

[0072] Preferably, step S2 in this embodiment specifically includes the following sub-steps:

[0073] Step S2.1, normalization, scaling the image pixel values ​​to the interval [0, 1], the formula is as follows:

[0074]

[0075] Among them, I(x, y) is the pixel value of the original image at position (x, y); min(I) and max(I) are the minimum and maximum pixel values ​​of the image respectively; I′(x, y) is the normalized pixel value; the normalization operation can unify the data distribution and reduce the interference of the classification model due to differences in illumination or contrast.

[0076] Step S2.2, resizing, resizing the image to a fixed size of 640×640 while maintaining the aspect ratio of the original image; in this embodiment, the image is resized to a fixed size of 640×640 by zero padding the edges to maintain the aspect ratio and avoid information loss due to deformation;

[0077] Step S2.3, grayscale processing, uses the weighted average method to convert the RGB image into a grayscale image to reduce the calculation requirements. The formula is as follows:

[0078] I gray =0.299R+0.587G+0.114B;

[0079] Among them, R, G, and B are the pixel values ​​of the red, green, and blue channels respectively. Grayscale processing reduces the data dimension while retaining the structural information of the image.

[0080] Step S2.4, feature enhancement, randomly rotates and scales the image to improve data diversity; through data enhancement, the model can adapt to different lighting conditions and signboard styles, enhancing its applicability.

[0081] Preferably, in step S2.4 of this embodiment, the random rotation angle range is -15° to 15°, and the random scaling ratio range is 90% to 110%;

[0082] Step S2.5, denoising, uses Gaussian filtering to remove random noise, and the filter weight is calculated by the following formula:

[0083]

[0084] Among them, G(x, y) is the weight value of the filter at the position (x, y); σ is the standard deviation, ranging from 1 to 2, and σ=1.5 is used in this embodiment; x, y are the distances between the filter position and the center point; the filter size is 5×5, and Gaussian filtering can effectively eliminate noise while retaining the edge information of the image.

[0085] Step S3, establishing a neural network classification model;

[0086] Preferably, step S3 in this embodiment specifically includes the following sub-steps:

[0087] Step S3.1, convolutional layer extracts features, and multi-level image features are extracted through filters. The formula is as follows:

[0088]

[0089] in, is the value of the feature map position (i, j) at the kth channel; X i+m，j+n is the pixel value at (i+m, j+n) in the input image; is the weight matrix of the convolution kernel; b (k) is the bias value of the kth convolution kernel; M, N are the sizes of the convolution kernel;

[0090] Step S3.2, activation function, using ReLU activation function to introduce nonlinear expression capability;

[0091] Step S3.3, pooling operation, maximum pooling is used to reduce dimension and extract key features. The formula is as follows:

[0092]

[0093] Among them, P i，j is the value of position (i, j) after pooling; S is the pooling window range;

[0094] Step S3.4, output layer, uses the Softmax activation function to convert the network output into a probability distribution, and outputs the probability distribution of each category to the subsequent modules for category determination and result display.

[0095] Preferably, in this embodiment, the convolution kernel size is 3×3, and the step size is 1; the pooling window size is 2×2, and the step size is 2.

[0096] Step S4, training the classification model established in step S3 based on the data preprocessed in step S2;

[0097] Preferably, step S4 in this embodiment specifically includes the following sub-steps:

[0098] Step S4.1, data segmentation, dividing the original sample image set of the power sign into a training set, a validation set and a test set, with a ratio of 8:1:1;

[0099] Step S4.2, set the loss function, the loss function formula is as follows:

[0100]

[0101] Where N is the number of samples; C is the number of categories; yij is the true category label (1 or 0); is the probability of the model output;

[0102] Step S4.3, use the Adam optimization algorithm to update the weights. The formula of the Adam optimization algorithm is as follows:

[0103]

[0104] Among them, θ t is the current weight; α is the learning rate; m t and v t are momentum and second-order momentum estimates, respectively; ∈ is a small value to prevent the denominator from being zero;

[0105] Step S4.4, learning rate adjustment, dynamically adjusting the learning rate through an exponential decay strategy;

[0106] Step S4.5, use the validation set to calculate the accuracy. When the validation loss decreases by less than 0.01 for 5 consecutive rounds or the validation set accuracy is greater than 95%, stop training. It should be understood that the conditions for stopping training can be adjusted appropriately. When the validation loss decreases tends to converge or the validation accuracy exceeds 90%, stop training. The accuracy can be selected between 86% and 98%;

[0107] Step S5, input the image of the power sign to be classified into the classification model trained in step S4, extract the image features through the trained classification model, match it with the target category, and output the category label and classification confidence of the sign. The present invention automatically identifies the type of sign through a deep learning model, achieves fast and accurate classification, significantly reduces the time required for manual identification, and improves work efficiency; uses a trained neural network model to extract multi-level features of the sign, and can maintain a high recognition accuracy rate for signs of different sizes, texts and image styles, effectively reducing the error rate in manual classification.

[0108] Example 2

[0109] Based on Example 1, Figure 2 As shown, in the method for classifying signboards based on deep learning in Embodiment 1 of this embodiment, after step S5, further steps are included:

[0110] Step S6, placing the classified power signboards into corresponding classification barrels through a robotic arm in cooperation with an electric suction cup.

[0111] Preferably, step S6 in this embodiment specifically includes the following sub-steps:

[0112] Step S6.1: Determine the target location, and query the physical location coordinates (xi ,y i , z i ); It should be understood that the physical location coordinates of the classification buckets are entered into the database in advance.

[0113] The target position is determined by the following formula:

[0114] P arm =T camerato-arm ·P camera ;

[0115] Among them, P arm is the target position in the robot coordinate system; P camera is the position of the sign in the camera coordinate system; T camerato-arm is the coordinate transformation matrix from camera to robotic arm;

[0116] Step S6.2: Path planning, using a fast random tree algorithm to generate a collision-free optimal path for the robot arm from the current position to the target classification bucket position;

[0117] Step S6.3, the robotic arm moves the electric suction cup according to the optimal path to grab the sign through negative pressure adsorption and places it stably in the corresponding classification bucket.

[0118] The combination of a robotic arm and an electric suction cup enables automatic grabbing, classification and fixed placement of signboards, greatly reducing manual operations and lowering labor intensity. Through the collaborative work of the classification model and the path planning algorithm, it ensures that the signboards are accurately placed in the corresponding classification bins after classification, effectively improving the management standardization of signboards.

[0119] The above are only preferred specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. The substitution may be a substitution of a part of the structure, device, method step, or a complete technical solution. The equivalent replacement or change according to the technical solution and the inventive concept of the present invention shall be covered within the protection scope of the present invention.

Claims

1. A sign classification method based on deep learning, characterized in that: The method comprises the following steps: Step S1, obtaining a sample set of original sample images of power signboards; Step S2, preprocessing the original sample image sample set of the power sign obtained in step S1, including normalization, size adjustment, grayscale processing, feature enhancement and denoising; Step S3, establishing a neural network classification model; Step S4, training the classification model established in step S3 based on the data preprocessed in step S2; Step S5, input the power sign image to be classified into the classification model trained in step S4, extract image features through the trained classification model, match them with the target category, and output the category label and classification confidence of the sign.

2. The deep learning-based sign classification method according to claim 1, characterized in that: In step S1, a wide-angle camera is used to collect original sample images of power signboards under various lighting conditions to increase sample diversity; after collecting the original sample images of power signboards, a category label is manually labeled for each image.

3. The deep learning-based sign classification method according to claim 1, characterized in that: The step S2 specifically includes the following sub-steps: Step S2.1, normalization, scaling the image pixel values ​​to the interval [0, 1], the formula is as follows: Where I(x, y) is the pixel value of the original image at position (x, y); min(I) and max(I) are the minimum and maximum pixel values ​​of the image, respectively; I′(x, y) is the normalized pixel value; Step S2.2, resizing, resizing the image to a fixed size of 640×640 while maintaining the aspect ratio of the original image; Step S2.3, grayscale processing, using weighted average method to convert RGB image into grayscale image to reduce calculation requirements; Step S2.4, feature enhancement, randomly rotate and scale the image to improve data diversity; Step S2.5, denoising, uses Gaussian filtering to remove random noise, and the filter weight is calculated by the following formula: Among them, G(x, y) is the weight value of the filter at position (x, y); σ is the standard deviation, ranging from 1 to 2; x, y are the distances between the filter position and the center point; and the filter size is 5×5.

4. The deep learning-based sign classification method according to claim 3, characterized in that: In step S2.4, the random rotation angle ranges from -15° to 15°, and the random scaling ratio ranges from 90% to 110%.

5. The deep learning-based sign classification method according to claim 1, characterized in that: The step S3 specifically includes the following sub-steps: Step S3.1, convolutional layer extracts features, and multi-level image features are extracted through filters. The formula is as follows: in, is the value of the feature map position (i, j) at the kth channel; X i+m，j+n is the pixel value at (i+m, j+n) in the input image; is the weight matrix of the convolution kernel; b (k) is the bias value of the kth convolution kernel; M, N are the sizes of the convolution kernel; Step S3.2, activation function, using ReLU activation function to introduce nonlinear expression capability; Step S3.3, pooling operation, maximum pooling is used to reduce dimension and extract key features. The formula is as follows: Among them, P i，j is the value of position (i, j) after pooling; S is the pooling window range; Step S3.4, output layer, uses the Softmax activation function to convert the network output into a probability distribution, and outputs the probability distribution of each category to the subsequent modules for category determination and result display.

6. The deep learning-based sign classification method according to claim 5, characterized in that: The convolution kernel size is 3×3, and the step size is 1; the pooling window size is 2×2, and the step size is 2.

7. The deep learning-based sign classification method according to claim 1, characterized in that: The step S4 specifically includes the following sub-steps: Step S4.1, data segmentation, dividing the original sample image set of the power sign into a training set, a validation set and a test set, with a ratio of 8:1:1; Step S4.2, set the loss function, the loss function formula is as follows: Where N is the number of samples; C is the number of categories; y ij is the true category label (1 or 0); is the probability of the model output; Step S4.3, use the Adam optimization algorithm to update the weights. The formula of the Adam optimization algorithm is as follows: Among them, θ t is the current weight; α is the learning rate; m t and v t are momentum and second-order momentum estimates, respectively; ∈ is a small value to prevent the denominator from being zero; Step S4.4, learning rate adjustment, dynamically adjusting the learning rate through an exponential decay strategy; In step S4.5, the accuracy is calculated using the validation set. When the validation loss decreases by less than 0.01 for five consecutive rounds or the validation set accuracy is greater than 95, training is stopped.

8. The deep learning-based sign classification method according to claim 1, characterized in that: The method further comprises the steps of: Step S6, placing the classified power signboards into corresponding classification barrels through a robotic arm in cooperation with an electric suction cup.

9. The deep learning-based sign classification method according to claim 8, characterized in that: The step S6 specifically includes the following sub-steps: Step S6.1: Determine the target location, and query the physical location coordinates (x i ,y i , z i ); The target position is determined by the following formula: P arm =T camerato-arm ·P camera ; Among them, P arm is the target position in the robot coordinate system; P camera is the position of the sign in the camera coordinate system; T camerato-arm is the coordinate transformation matrix from camera to robotic arm; Step S6.2: Path planning, using a fast random tree algorithm to generate a collision-free optimal path for the robot arm from the current position to the target classification bucket position; Step S6.3, the robotic arm moves the electric suction cup according to the optimal path to grab the sign through negative pressure adsorption, and places it stably in the corresponding classification bucket.

Images