Automatic stone grading method and system based on deep learning

Abstract

The application discloses a kind of stone automatic grading method and system based on deep learning, the method is using stone picture quality evaluation module, stone grading feature extraction module and stone grading data fitting module realizes industrial belt high-speed transport scene stone automatic grading task, stone picture quality evaluation module is a residual neural network based on multilevel feature, for the high-quality dynamic frame of picture quality is evaluated and screened;Stone grading feature extraction module is a segmentation module based on digital image processing using watershed algorithm and concave point detection algorithm, for the stone grading feature information extraction of screened picture;Stone grading data fitting module is an improved multilayer perceptron, for the data fitting correction error of initial stone grading feature result.The application has higher precision and better generalization ability in different backgrounds, and can accurately detect different types of stone transport scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of digital image processing, deep learning and computer vision, and particularly refers to a stone automatic grading method and system based on deep learning. BACKGROUND

[0002] Stone grading is to divide stone aggregates into different grades according to the size of the stone aggregate particle size, that is, stone gradation. Stone is widely used in various fields such as construction, road, railway, etc. In architecture, because different particle sizes of stone have different functions, the particle size of stone needs to be sieved before use. Stone grading not only meets the needs of construction engineering, but also helps enterprises to reduce costs and increase efficiency. The existing stone grading methods include screening method, sedimentation method, microscope method, ultrasonic measurement method and direct online measurement method. The commonly used screening method detection process is to use screening tools with different sieve hole sizes to screen the stone particles to be measured into multiple particle size grades. The stone that can pass through the sieve hole is called undersize, and the stone that cannot pass through the sieve hole is called oversize. Then the stone of each different particle size passing through the sieve hole is weighed respectively, and the mass percentage of each particle size in the total stone mass is calculated to represent the distribution of stone particle size. However, this screening method is a very tedious and time-consuming thing, which is not conducive to the production needs of enterprises. With the increasing maturity of digital image processing technology and computer vision technology, automatic stone grading method can better help enterprise engineering production and optimize company member configuration, improve the production efficiency of construction engineering and generate early warning prompts for potential deviations and abnormalities.

[0003] However, it is still a difficult task to accurately and quickly automatically grade stone in the scene of high-speed belt operation and micro-vibration. First, because the industrial camera installed on the belt will have a slight position shift with the belt vibration, the stone video taken will inevitably have some invalid frames and blurred frames. Second, the fast movement of the belt will inevitably cause the shooting picture to be blurred. Third, the transportation of no stone on the empty belt will cause the shooting picture to have no stone to determine as empty state, which increases the difficulty of automatic stone grading.

[0004] The early stone grading method is mostly a simple grading method relying on manual screening, but the common problem of these methods is high cost and labor consumption, and the distribution of extracted stone particle size is not ideal. Subsequently, the emergence of digital image processing technology, deep learning and computer vision technology is more efficient than traditional manual grading methods, and the accuracy is also very competitive. Compared with the traditional manual stone grading method, the stone automatic grading method based on deep learning and image processing technology realizes the automatic grading process by extracting the initial stone grading features in real time and modifying the errors, without manual screening experiments. Since the belt is running at high speed and has a slight vibration, the collection of clear stone images is the premise and basis for realizing automatic prediction of stone grading. The image segmentation algorithm based on concave point detection has been proved to be able to segment the adherent particles and can be successfully applied to the stone grading task to obtain the surface stone particle size distribution. However, the image segmentation algorithm cannot obtain the deep particle size distribution of the stone from the surface stone particle size distribution obtained by segmenting the clear stone image, which affects the accuracy of the automatic prediction of stone grading. Based on the limitations of the stone unable to obtain the deep particle size distribution of the stone and real-time detection, the present application proposes a stone automatic grading method based on deep learning, which can predict the overall stone particle size distribution under the condition of stone stacking on the belt through the surface stone particle size distribution, greatly improving the accuracy of each stone grading.

[0005] Based on the above discussion, it has high practical value to invent a stone automatic grading method with real-time and high precision. SUMMARY

[0006] The first object of the present application is to overcome the shortcomings and deficiencies of the prior art, and to provide a stone automatic grading method based on deep learning, which uses a multi-level feature residual neural network and an improved multilayer perceptron to realize the stone automatic grading task, has higher precision and better generalization ability in different backgrounds, and can accurately detect different types of stone transportation scenes.

[0007] The second object of the present application is to provide a stone automatic grading system based on deep learning.

[0008] The first object of the application is achieved by the technical scheme of the stone automatic grading method based on deep learning, which is based on deep learning and digital image processing technology to realize the automatic grading task of stone in the industrial belt high-speed transportation scene. Specifically, three modules are adopted, namely, a stone picture quality evaluation module, a stone grading feature extraction module and a stone grading data fitting module. The stone picture quality evaluation module is a residual neural network based on multi-level features, which is used to evaluate and screen high-quality dynamic frames. The stone grading feature extraction module is a segmentation module based on digital image processing, which adopts a watershed algorithm and a concave point detection algorithm, and is used to extract stone grading feature information from the pictures screened by the stone picture quality evaluation module. The stone grading data fitting module is an improved multilayer perceptron, which is used to fit the data of the initial stone grading feature results extracted by the stone grading feature extraction module, make up for the stone grading error caused by the lack of stone depth information, and realize error correction.

[0009] The specific implementation of the stone automatic grading method includes the following steps:

[0010] 1) Collect each batch of stone in the belt high-speed transportation video, downsample each video, and retain a number of pictures for each stage of each video. Arrange the pictures in the format of "production channel / stone batch / frame number". Meanwhile, sample each batch of stone using the four-part method and complete the manual screening experiment. According to the true label obtained by the manual screening experiment, arrange the pictures in the format of "production channel / stone batch number" to form a stone original data set.

[0011] 2) Input the pictures of the stone original data set into the stone picture quality evaluation module, evaluate the quality of the pictures and calculate their sharpness scores, obtain the sharpness score ranking of the pictures of the stone original data set, and select the pictures with high sharpness scores as the stone data set according to the sharpness score ranking.

[0012] 3) Take the stone data set obtained in step 2) as the input data of the stone grading feature extraction module, effectively extract the initial stone grading feature information of the pictures, and generate the initial stone grading feature results of all pictures in the stone data set.

[0013] 4) Input the initial stone grading feature results of all pictures generated by the stone grading feature extraction module into the stone grading data fitting module, fit the data of the initial stone grading feature results to correct the error, and obtain the stone grading feature prediction results after error correction.

[0014] Further, in step 1), first, the video of the transportation process of the stone on the belt is collected by an industrial camera, and each batch of stone transportation process is stored as a video, then the images of each video are sliced at a specific time interval, and the images are named and arranged in the format of "production channel / stone batch / frame number", meanwhile, the four-fold method is used to sample each batch of stone and complete the manual screening experiment, the real label of each batch of stone is obtained according to the manual screening experiment, and is named and arranged in the format of "production channel / stone batch number"; wherein, the four-fold method refers to grinding the stone sample according to the measurement requirements, passing through a screen with a specific aperture, then mixing and laying flat into a circle, dividing into four equal parts, taking the opposite two parts, then dividing again until the requirements are met; the manual screening experiment refers to manual screening by workers through a new standard square hole sandstone screen, which is divided into 10 screen sizes, and the screen sizes are: 53-37.5, 37.5-31.5, 31.5-26.5, 26.5-19.0, 19.0-16.0, 16.0-9.5, 9.5-4.75, 4.75-2.36, 2.36-bottom disc, the screen unit is mm, finally, the real label of each batch of stone is mapped to multiple images obtained by slicing the transportation video of each batch of stone in one-to-many, and the stone original data set is made.

[0015] Further, in step 2), the stone picture quality evaluation module is a multi-level feature-based residual neural network, which mainly aims to eliminate abnormal frame data in the stone original data set, including frame data with full picture blur, large-scale ghosting, extreme lighting, and incomplete shooting during the acquisition process; the multi-level feature-based residual neural network includes a base learner and an integrated learner, the base learner is composed of a residual convolutional neural network, which mainly aims to learn different types of hierarchical features in stone pictures and extract initial features of stone pictures; wherein the residual convolutional neural network contains three different stages, each stage includes different numbers of residual blocks; the first stage contains one 7*7 convolutional layer, one batch normalization layer, one max pooling layer and two residual blocks, aiming to perform initial preprocessing on the stone picture and extract shallow features; the second stage is four residual blocks, aiming to further abstract the information and features in the stone picture; the third stage is two residual blocks, aiming to learn and extract high-level semantic information contained in the stone picture; the integrated learner includes three different encoder layers and a Concat operation, wherein the main purpose of the three different encoder layers is to integrate the initial features of the stone picture extracted by the base learner to form the overall representation of the stone picture, the encoder layers all contain a 1*1 convolutional layer, a 3*3 convolutional layer and an average pooling layer, but the difference is that the channel numbers of the 1*1 convolutional layer are 16, 4 and 4 respectively, the strides of the 3*3 convolutional layer are 7, 4 and 2 respectively, and the padding of the 3*3 convolutional layer is 1, 3 and 2 respectively, the encoder layers aim to perform channel transformation on the representation features of the stone picture after the three stages of the residual convolutional neural network; in order to ensure the integrity of the data, the Concat operation is used to fuse the hierarchical features after the channel transformation of the three different encoder layers, and a fully connected layer is used to map them to 256-dimensional overall features, calculate the stone picture quality definition score, and finally delete the pictures with scores less than the set threshold score to form the stone data set, and perform flip and rotation operations on the stone images in the stone data set for data augmentation, wherein the flip includes horizontal flip, vertical flip and horizontal and vertical flip, and the rotation uses the values of-30°, -15°, 15° and 30°.

[0016] Further, in step 3), the stone grading feature extraction module is a segmentation module based on digital image processing using the watershed algorithm and concave point detection algorithm, which mainly aims to effectively extract the initial grading feature information of the picture and generate the initial stone grading feature result, the segmentation module includes three parts of stone morphological image processing, stone image segmentation and stone particle calibration and granularity calculation, the specific circumstances are as follows:

[0017] The first part is a morphological image processing of stone, and the main purpose is to perform a morphological image processing operation on the stone picture to obtain an image that can meet the image segmentation condition, wherein the morphological image processing operation comprises stone image preprocessing and stone morphological optimization, the stone image preprocessing is composed of image gray scale transformation, image filtering processing and image binarization processing, and the main purpose is to pre-process the image to enhance the image quality, the image gray scale transformation is to perform brightness transformation processing on the pictures in the stone data set to enhance or weaken the brightness of the image, and then perform gray scale processing on the brightness transformed image to reduce the data amount in the image, the image filtering processing is to use bilateral filtering to perform denoising processing on the stone image after image gray scale transformation while retaining the edge information of the stone particles, and the image binarization processing is to perform fast adaptive thresholding processing on the stone image after image filtering processing to achieve the effect of image simplification, and the stone morphological optimization is composed of a series of erosion and dilation combined opening and closing operation, and the main purpose is to perform morphological operation on the image after stone image preprocessing to achieve the purpose of denoising, smoothing, detecting and analyzing the stone particle image.

[0018] The second part is stone image segmentation, and the main purpose is to effectively segment the stone particles in the image after the stone morphological image processing, wherein the stone image segmentation adopts a watershed algorithm and a concave point detection algorithm, the watershed algorithm refers to a dam set between each water area in a topographic map, and a gray scale image is regarded as a topographic map, and the gray scale value of the pixel points in the gray scale image is regarded as the altitude, different gray scale values of the pixel points represent different altitudes, the place with low gray scale value has low altitude, and the place with high gray scale value has high altitude, but directly performing the watershed algorithm on the image will lead to over-segmentation phenomenon, and the image edge information is lost, and the problem that the image with large-area adhered stone particles cannot be effectively segmented, therefore, the concave point detection algorithm is introduced to solve the problem, and the combination of the two can effectively segment the adhered particles in the stone image.

[0019] The third part is stone particle calibration and granularity calculation, and the main purpose is to calculate the contour area of each stone particle on the image after the stone image segmentation, divide the stones into 10 different sizes according to the contour area, and count the number of stones of the 10 different sizes, wherein the stone particle calibration and granularity calculation calibrate the stone particles on site and statistically analyze the distribution information of the 10 different granularities in the stone to obtain the proportion of each granular particle, and the proportions of the 10 different granular particles correspond to the stone initial grading feature information of the 10-dimensional vector, so as to effectively extract the initial stone grading feature information of the picture, obtain the initial stone grading feature result, and the stone initial grading feature information of the 10-dimensional vector corresponds to the 10 screen hole sizes of the new standard square hole sand screen.

[0020] Further, in step 4), the stone grading data fitting module is an improved multilayer perception, the main purpose of which is to solve the problem that the depth particle size distribution of the batch of stones cannot be obtained by the stone image segmentation of the segmentation module to the surface layer of the stone, the improved multilayer perception can predict the overall stone particle size distribution under the condition of stone stacking on the belt through the surface layer of the sandstone particle size distribution, and correct the error of the initial stone grading feature results of all pictures generated by the stone grading feature extraction module. The improved multilayer perception is a multilayer neural network with hidden states, which is composed of an input layer, a hidden variable with hidden states, and an output layer, and the specific conditions are as follows:

[0021] The first part is the input layer, the number of neurons of which is d, which corresponds to the d-dimensional vector of the initial stone grading feature information generated by the stone grading feature extraction module, and a small batch of initial stone grading feature samples are input at time step t X t ∈R n×d , wherein R is a real number, n is the batch size, d is the number of neurons, and is also the number of input features;

[0022] The second part is the hidden variable H t with hidden states, which is determined by the input layer of the current time step, i.e. the input X t of the small batch of initial stone grading feature samples at time step t, and the hidden variable H t-1 of the previous time step t-1, for a small batch of n initial stone grading feature sequence samples, each row of X t corresponds to the initial stone grading feature sample at time step t from the sequence, unlike the original multilayer perception, the improved multilayer perception saves the hidden variable H t-1 of the previous time step, and introduces a new weight parameter W hh ∈R h×h to describe how to use the hidden variable of the previous time step in the current time step, the process is as follows formula (1):

[0023] H t =φ(X t W xh +H t-1 W hh +b h ) (1)

[0024] In the formula, X t ∈R n×d is the input of the small batch of initial stone grading feature samples at time step t, n is the batch size, d is the number of input features, h is the number of hidden units, φ is a nonlinear activation function ReLU, and the weight parameter has W xh ∈R d×hand W hh ∈R h×h , W xh ∈R d×h describes how to use the input values of the current time step in the current time step, the bias unit is b h ∈R 1×h , b h ∈R 1×h describes how to use the bias unit in the current time step, the output is the hidden variable H t of time step t t ∈R n ×h ;

[0025] The third part is the output layer, which is a fully connected layer. The neurons in the output layer are also fully connected with each neuron in the hidden layer. The number of neurons in this layer is q, which corresponds to the q sieve sizes of the new standard square-hole sand and gravel sieve. The process is as follows formula (2):

[0026] O t = H t W hq +b q (2)

[0027] In the formula, H t ∈R n×h is the hidden variable of time step t, the batch size is n, the number of features h is the number of hidden units of the hidden variable, the number of output units is q, the weight parameters and bias parameters of the output layer are W hq ∈R h×q and b q ∈R 1×q , and the output of the output layer is O t , O t ∈R n×q ;

[0028] The initial stone grading features of all pictures in the stone data set extracted in step 3) and the true labels obtained from the artificial screening experiment in step 1) are used as the data set of the improved multilayer perceptron, and the training set, validation set and test set are divided. Then construct multiple selectable improved multilayer perceptron candidate models. In each candidate model, use the cross-entropy loss function to calculate the difference between the initial stone grading features and the true labels on the training set, and use back propagation to update the gradient to train the improved multilayer perceptron candidate model. Get the optimal weight parameters of each candidate model. Then use the validation set to evaluate the performance of several candidate models and select the optimal model. Use the test set to calculate the generalization error of the optimal model. Finally, deploy the optimal model of the improved multilayer perceptron online for real-time prediction of stone automatic grading in production.

[0029] The second object of the present application is achieved by the following technical solution: the stone automatic grading system based on deep learning comprises:

[0030] The data acquisition module is configured to acquire high-speed transportation videos of each batch of stone, down-sample each video, retain a number of pictures for each stage of each video, arrange and name the pictures in the format of "production channel / stone batch / frame number", sample each batch of stone by using the four-part method, complete manual screening experiments, obtain true labels of each batch of stone according to the manual screening experiments, arrange and name the true labels in the format of "production channel / stone batch", form a stone original data set, and output the stone original data set to the stone picture quality evaluation module.

[0031] The stone picture quality evaluation module is configured to evaluate the quality of the pictures and calculate a definition score of the pictures, obtain a definition score ranking of the pictures in the stone original data set, select pictures with high definition scores as a stone data set according to the definition score ranking, and output the stone data set to the stone grading feature extraction module.

[0032] The stone grading feature extraction module is configured to effectively extract initial stone grading feature information of the pictures selected by the stone picture quality evaluation module, and generate initial stone grading feature results of all the pictures in the stone data set.

[0033] The stone grading data fitting module is configured to fit the initial stone grading feature results extracted by the stone grading feature extraction module, compensate for stone grading errors caused by missing stone depth information, and obtain error-corrected stone grading feature prediction results.

[0034] Further, the data acquisition module specifically performs the following operations:

[0035] First, the video of the stone transportation process on the belt is collected by an industrial camera, and the complete transportation process of each batch of stone is stored as a video. Then, the images of each video are sliced at a specific time interval, and the images are named and arranged in the format of "production channel / stone batch / frame number". Meanwhile, each batch of stone is sampled using the quartering method and the manual screening experiment is completed. The true label of each batch of stone is obtained according to the manual screening experiment, and is named and arranged in the format of "production channel / stone batch number". The quartering method refers to grinding the stone sample according to the measurement requirements, passing through a sieve with a specific aperture, then mixing and laying flat into a circle, dividing into four equal parts, taking the opposite two parts, then dividing again until the required size is reached. The manual screening experiment refers to manual screening by workers using a new standard square hole sandstone sieve. The new standard square hole sandstone sieve is divided into 10 sieve sizes, and the sieve sizes are 53-37.5, 37.5-31.5, 31.5-26.5, 26.5-19.0, 19.0-16.0, 16.0-9.5, 9.5-4.75, 4.75-2.36, 2.36-bottom disc, with the sieve unit being mm. Finally, the true label of each batch of stone is mapped one-to-many with the multiple images obtained by slicing the transportation video of each batch of stone, and the stone original data set is prepared.

[0036] Further, the stone picture quality evaluation module is a residual neural network based on multi-level features, which mainly aims to eliminate abnormal frame data in the stone original data set, including frame data with full image blur, large-scale ghosting, extreme lighting, and incomplete shooting during the acquisition process; the residual neural network based on multi-level features includes a base learner and an integrated learner. The base learner is composed of a residual convolutional neural network, which mainly aims to learn different types of hierarchical features in stone pictures and extract initial features of stone pictures. The residual convolutional neural network includes three different stages, each stage including different numbers of residual blocks. The first stage includes one 7*7 convolutional layer, one batch normalization layer, one max-pooling layer, and two residual blocks, aiming to perform initial preprocessing on the stone picture and extract shallow features. The second stage is four residual blocks, aiming to further abstract information and features in the stone picture. The third stage is two residual blocks, aiming to learn and extract high-level semantic information contained in the stone picture. The integrated learner includes three different encoder layers and a Concat operation. The main purpose of the three different encoder layers is to integrate the initial features of the stone picture extracted by the base learner to form the overall representation of the stone picture. The encoder layers all include a 1*1 convolutional layer, a 3*3 convolutional layer, and an average pooling layer, but the channel numbers of the 1*1 convolutional layers are 16, 4, and 4 respectively, the strides of the 3*3 convolutional layers are 7, 4, and 2 respectively, and the padding of the 3*3 convolutional layers is 1, 3, and 2 respectively. The encoder layers aim to perform channel transformation on the representation features of the stone picture after the three stages of the residual convolutional neural network. To ensure data integrity, the Concat operation is used to fuse the hierarchical features after channel transformation of the three different encoder layers, and a fully connected layer is used to map them to 256-dimensional overall features to calculate the stone picture quality definition score. Finally, according to the score, the pictures with scores less than the set threshold score are deleted to form the stone data set, and the stone images in the stone data set are flipped and rotated for data augmentation, including horizontal flipping, vertical flipping, and horizontal and vertical flipping, and rotation with values of -30°, -15°, 15°, and 30°.

[0037] Further, the stone grading feature extraction module is a segmentation module based on digital image processing using the watershed algorithm and concave point detection algorithm, which mainly aims to effectively extract the initial grading feature information of the picture and generate the initial stone grading feature result. The segmentation module includes three parts: stone morphological image processing, stone image segmentation, and stone particle calibration and granularity calculation, which are as follows:

[0038] The first part is a morphological image processing of stone, and the main purpose is to perform a morphological image processing operation on the stone picture to obtain an image that can meet the image segmentation condition, wherein the morphological image processing operation includes stone image preprocessing and stone morphological optimization, the stone image preprocessing is composed of a series of image gray scale transformation, image filtering processing and image binarization processing, and the main purpose is to pre-process the image to enhance the image quality, the image gray scale transformation is to perform brightness transformation processing on the pictures in the stone data set to enhance or weaken the brightness of the image, and then perform gray scale processing on the brightness transformed image to reduce the data amount in the image, the image filtering processing is to use bilateral filtering to perform denoising processing on the stone image after image gray scale transformation while retaining the edge information of the stone particles, and the image binarization processing is to perform fast adaptive thresholding processing on the stone image after image filtering processing to achieve the effect of image simplification, and the stone morphological optimization is composed of a series of erosion and dilation combined opening and closing operation, and the main purpose is to perform morphological operation on the image after stone image preprocessing to achieve the purpose of denoising, smoothing, detecting and analyzing the stone particle image;

[0039] The second part is stone image segmentation, and the main purpose is to effectively segment the stone particles in the image after the morphological image processing of stone, wherein the stone image segmentation adopts a watershed algorithm and a concave point detection algorithm, the watershed algorithm refers to a dam set between each water area in a topographic map, and a gray scale image is regarded as a topographic map, the gray scale value of the pixel points in the gray scale image is regarded as the altitude, and the gray scale value of different pixel points represents different altitudes, the place with low gray scale value has low altitude, and the place with high gray scale value has high altitude, but directly performing watershed algorithm segmentation on the image will lead to over-segmentation phenomenon, and the image edge information is lost, and the problem that the image with large-area adhered stone particles cannot be effectively segmented, therefore, the concave point detection algorithm is introduced to solve the problem, and the combination of the two can effectively segment the adhered particles in the stone image;

[0040] The third part is stone particle calibration and granularity calculation, and the main purpose is to calculate the contour area of each stone particle on the image after stone image segmentation, divide the stones into 10 different sizes of particle diameters according to the contour area, and count the number of stones of the 10 different sizes of particle diameters, wherein the stone particle calibration and granularity calculation calibrate the stone particles on site and statistically analyze the distribution information of the 10 different granularities in the stone to obtain the proportion of each particle size, and the proportions of the 10 different particle sizes correspond to the stone initial grading feature information of a 10-dimensional vector, so as to effectively extract the initial stone grading feature information of the picture, obtain the initial stone grading feature result, and the stone initial grading feature information of the 10-dimensional vector corresponds to the 10 screen hole sizes of the new standard square hole sand screen.

[0041] Further, the stone grading data fitting module is an improved multilayer perceptron, and the main purpose is to solve the problem that the surface stone particle size distribution obtained by the image segmentation module cannot obtain the deep particle size distribution of the stone, and the improved multilayer perceptron can predict the overall stone particle size distribution under the condition of stone stacking on the belt through the surface stone particle size distribution. The initial stone grading feature results of all pictures generated by the stone grading feature extraction module are data fitting correction errors. The improved multilayer perceptron is a multilayer neural network with hidden states, which is composed of an input layer, a hidden variable with hidden states, and an output layer. The specific conditions are as follows:

[0042] The first part is the input layer, and the number of neurons is d, which corresponds to the d-dimensional vector of the initial stone grading feature information generated by the stone grading feature extraction module. At time step t, a small batch of initial stone grading feature samples X t ∈R n×d , where R is a real number, n is the batch size, d is the number of neurons, and also the number of input features;

[0043] The second part is the hidden variable H t with hidden states, which is determined by the input layer of the current time step, that is, the input X t of the small batch of initial stone grading feature samples at time step t, and the hidden variable H t-1 of the previous time step t-1. For a small batch of n initial stone grading feature sequence samples, X t Each row corresponds to the initial stone grading feature sample at time step t from the sequence. Unlike the original multilayer perceptron, the improved multilayer perceptron saves the hidden variable H t-1 of the previous time step and introduces a new weight parameter W hh ∈R h×h to describe how to use the hidden variable of the previous time step in the current time step. The process is as follows formula (1):

[0044] H t =φ(X t W xh +H t-1 W hh +b h ) (1)

[0045] In the formula, X t ∈R n×d is the input of the small batch of initial stone grading feature samples at time step t, n is the batch size, d is the number of input features, h is the number of hidden units, φ is the nonlinear activation function ReLU, and the weight parameters are W xh ∈R d×h and Whh ∈R h×h , W xh ∈R d×h describes how to use the input values of the current time step in the current time step, the bias unit is b h ∈R 1×h , b h ∈R 1×h describes how to use the bias unit in the current time step, the output is the hidden variable H of time step t t , H t ∈R n ×h ;

[0046] The third part is the output layer, which is a fully connected layer. The neurons in the output layer are also fully connected with each of the neurons in the hidden layer. The number of neurons in this layer is q, which corresponds to the q sieve hole sizes of the new standard square hole sand and gravel sieve. The process is as follows formula (2):

[0047] O t = H t W hq + b q (2)

[0048] In the formula, H t ∈R n×h is the hidden variable of time step t, the batch size is n, the number of features h is the number of hidden units of the hidden variable, the number of output units is q, the weight parameters and bias parameters of the output layer are W hq ∈R h×q and b q ∈R 1×q , and the output of the output layer is O t , O t ∈R n×q ;

[0049] The stone grading feature extraction module extracts the initial stone grading features of all pictures of the stone data set and the true labels obtained by the artificial screening experiment in the corresponding data collection module as the data set of the improved multilayer perceptron, and divides the training set, the validation set and the test set. Then construct several selectable improved multilayer perceptron candidate models. In each candidate model, use the cross-entropy loss function to calculate the gap between the initial stone grading features and the true labels on the training set, and use back propagation to update the gradient to train the improved multilayer perceptron candidate model. Get the optimal weight parameters of each candidate model. Then use the validation set to evaluate the performance of several candidate models and select the optimal model. Use the test set to calculate the generalization error of the optimal model. Finally, deploy the optimal model of the improved multilayer perceptron online for real-time prediction of stone automatic grading in production.

[0050] Compared with the prior art, the present application has the following advantages and beneficial effects:

[0051] 1、 The present application provides a residual neural network based on multi-level features, which extracts the visual information features of the shallow and deep layers in the stone photo through the residual neural network, simultaneously fuses the multi-level visual information features in the residual neural network, obtains the features with excellent expression ability, and then realizes the task of screening reasonable frames in the stone picture quality evaluation module.

[0052] 2、 The present application provides an improved multi-layer perceptron, which can learn the time sequence information features of the stone picture through the multi-layer perceptron with hidden variables of hidden states, better data fit the initial stone grading features extracted by the stone grading feature extraction module, make up the stone grading error caused by the lack of stone depth information, further improve the accuracy of stone grading, and further improve the stone grading effect.

[0053] 3、 The present application uses the stone picture quality evaluation module, the stone grading feature extraction module and the stone grading data fitting module based on deep learning and digital image processing technology, has higher accuracy and better generalization ability in different backgrounds, can accurately detect different batches of stone, uses the visual and time sequence information of the stone transportation video, and can solve the automatic prediction of stone grading in the field of deep learning. BRIEF DESCRIPTION OF DRAWINGS

[0054] Figure 1 It is the architecture diagram of the method of the present application.

[0055] Figure 2 It is the residual neural network structure diagram of the present application.

[0056] Figure 3 It is the stone grading feature extraction module structure diagram of the present application.

[0057] Figure 4 It is the improved multi-layer perceptron structure diagram of the present application.

[0058] Figure 5 It is the architecture diagram of the system of the present application. DETAILED DESCRIPTION

[0059] The present application will be further described in detail below in combination with embodiments and drawings, but the embodiments of the present application are not limited thereto.

[0060] Example 1

[0061] The embodiment is implemented under the Pytorch deep learning framework, and the computer configuration adopts: Intel(R) Core(TM) i7-7700K CPU, 16 GB memory, NVIDIA GeForce GTX 1050Ti graphics card, and Windows operating system. The embodiment discloses a stone automatic grading method based on deep learning. The method is based on deep learning and digital image processing technology to realize the automatic grading task of stone in the industrial belt high-speed transportation scene. Specifically, three modules are adopted, namely, a stone picture quality evaluation module, a stone grading feature extraction module, and a stone grading data fitting module. The overall architecture is as shown in Figure 1 The stone picture quality evaluation module is a residual neural network based on multi-level features, which is used for evaluating and screening high-quality dynamic frames. The stone grading feature extraction module is a segmentation module based on digital image processing, which adopts a watershed algorithm and a concave point detection algorithm, and is used for extracting stone grading feature information from the pictures screened by the stone picture quality evaluation module. The stone grading data fitting module is an improved multi-layer perceptron, which is used for data fitting of the initial stone grading feature results extracted by the stone grading feature extraction module, and makes up for the stone grading error caused by the lack of stone depth information, and realizes error correction.

[0062] The specific implementation of the stone automatic grading method includes the following steps:

[0063] 1) First, record the whole process of each batch of stone transportation by industrial camera installed on the belt, and store each complete stone transportation process as a video. Then use ffpmeg to slice each video, store an image every 5 seconds, and arrange them in order according to the format "production channel / stone batch / frame number". At the same time, workers sample the stones on the belt at the same interval according to the quartile method, and conduct manual screening experiments on the stones through the new standard square hole sand screen. The proportion of 10 types of each batch of stone grading obtained is packaged into a real label in the form of a tensor, and arranged in order according to the format "production channel / stone batch number"; the quartile method refers to grinding the stone sample according to the measurement requirements, passing through a specific aperture sieve, then mixing and laying out in a circle, dividing into four equal parts, taking the opposite two parts, then dividing again until the required size is reached; the manual screening experiment refers to manual screening by workers through the new standard square hole sand screen, which is divided into 10 sieve sizes, with sieve sizes of 53-37.5, 37.5-31.5, 31.5-26.5, 26.5-19.0, 19.0-16.0, 16.0-9.5, 9.5-4.75, 4.75-2.36, 2.36-bottom disc, with sieve size unit of mm; finally, map the real label of each batch of stone to the multiple images obtained by slicing the video of each batch of stone transportation, and save them to the database as the stone original data set.

[0064] 2) Input the pictures of the stone original data set into the stone picture quality evaluation module, evaluate the quality of the pictures and calculate the sharpness score, obtain the picture sharpness score ranking of the stone original data set, and select the pictures with higher sharpness score as the stone data set according to the sharpness score ranking; wherein the specific situation of the stone picture quality evaluation module is as follows:

[0065] The stone picture quality evaluation module is a residual neural network based on multi-level features, and the network structure is shown in Figure 2 The network is an end-to-end structure, which can be divided into a base learner and an ensemble learner. The base learner is composed of a resnet18 residual convolutional neural network, which mainly aims to learn different types of hierarchical features in stone pictures and extract initial features of stone pictures; the ensemble learner includes three Encoder encoder layers and a Concat operation, which aims to transform the channel of the feature representation of the stone picture after three stages of residual convolutional neural network. The specific structure of each part of the network is as follows:

[0066] The base learner includes three different stages, each stage includes different number of residual blocks, the residual block is composed of a 1*1 convolution layer, two 3*3 convolution layers, two batch normalization layers and two ReLU activation functions, the first stage includes a 7*7 convolution layer, a batch normalization layer, a max pooling layer and two residual blocks, shallow feature information of the stone picture is extracted through convolution operation, the second stage includes four consecutive residual block operations, which aims to further abstract the information and features in the stone picture, the third stage is two residual blocks, which aims to learn and extract high-level semantic information contained in the stone picture, the feature information in the field of view is extracted through convolution operation, and the field of view is expanded through continuous convolution to obtain the relationship between different feature values, and the down-sampling operation reduces the resolution of the feature map, thereby saving the loss of calculation amount and memory.

[0067] The integrated learner includes three Encoder encoder layers, the main purpose of which is to integrate the initial features of the stone picture extracted by the base learner to form the overall representation of the stone picture, wherein the three Encoder encoder layers all include a 1*1 convolution layer, a 3*3 convolution layer and an average pooling layer, but are not all the same, the channel numbers of the 1*1 convolution layers are 16, 4 and 4 respectively, the strides of the 3*3 convolution layers are 7, 4 and 2 respectively, and the padding of the 3*3 convolution layers is 1, 3 and 2 respectively. In order to ensure the integrity of the data, the Concat operation is used to complete the fusion of the hierarchical features after the channel transformation of the three Encoder encoder layers, and the FC layer is used to map them to 256-dimensional overall features, calculate the stone picture quality definition score, and finally delete the pictures with scores less than the set threshold score, the threshold is set to 60, form the stone data set, and perform flip and rotation operations on the stone images in the stone data set to perform data augmentation, wherein the flip includes horizontal flip, vertical flip and horizontal and vertical flip, and the rotation uses the values of-30°, -15°, 15° and 30°.

[0068] 3) The stone data set obtained by the stone picture quality evaluation module is used as the input data of the stone grading feature extraction module, and the initial stone grading feature information of the picture is effectively extracted to generate the initial stone grading feature result, the stone grading feature extraction module includes stone morphological image processing, stone image segmentation and stone particle calibration and granularity calculation three parts, the structure is shown in Figure 3 The specific situation is as follows:

[0069] The first part is a morphological image processing of stone, and the main purpose is to perform a morphological image processing operation on a stone picture to obtain an image meeting the image segmentation condition, wherein the morphological image processing operation comprises stone image preprocessing and stone morphological optimization, the stone image preprocessing is composed of a series of operations such as image gray scale transformation, image filtering processing and image binarization processing, and the main purpose is to pre-process the image to enhance the image quality, the image gray scale transformation is to perform brightness transformation processing on the pictures in the stone data set to enhance or weaken the brightness of the image, then the image after brightness transformation is subjected to gray scale processing to reduce the data amount in the image, the image filtering processing is to use bilateral filtering to perform denoising processing on the stone image after image gray scale transformation while retaining the edge information of the stone particles, and the image binarization processing is to perform fast adaptive thresholding processing on the stone image after image filtering processing to achieve the effect of image simplification; the stone morphological optimization is composed of a series of erosion and dilation combined opening and closing operations, and the main purpose is to perform morphological operation on the image after stone image preprocessing to achieve the purpose of denoising, smoothing, detecting and analyzing the stone particle image;

[0070] The second part is stone image segmentation, and the main purpose is to effectively segment the stone particles in the image after the stone morphological image processing, wherein the stone image segmentation adopts a watershed algorithm and a concave point detection algorithm, the watershed algorithm refers to a dam set between each water area in a topographic map, and a gray scale image is regarded as a topographic map, the gray scale value of a pixel point in the gray scale image is regarded as an altitude, and different gray scale values of pixel points represent different altitudes, the place with low gray scale value has low altitude, and the place with high gray scale value has high altitude, but directly performing the watershed algorithm on the image will lead to over-segmentation phenomenon, losing the edge information of the image, and unable to effectively segment the image with large-area adhesion of stone particles, therefore, the concave point detection algorithm is introduced to solve the problem, and experiments show that the combination of the two can effectively segment the adhesion particles in the stone image;

[0071] The third part is stone particle calibration and granularity calculation, and the main purpose is to calculate the contour area of each stone particle on the image after the stone image segmentation, divide the stones into 10 different sizes of particle diameters according to the contour area, and count the number of stones of the 10 different sizes of particle diameters, wherein the stone particle calibration and granularity calculation calibrate the stone particles on site and statistically analyze the distribution information of the 10 different sizes of particles in the stone to obtain the proportion of each particle size, and the proportions of the 10 different particle sizes correspond to the stone initial grading feature information of a 10-dimensional vector, so as to effectively extract the initial stone grading feature information of the picture, obtain the initial stone grading feature result, and the stone initial grading feature information of the 10-dimensional vector corresponds to the 10 sieve hole sizes of the new standard square hole sand screen.

[0072] 4) The stone grading feature extraction module generates 6000 stone initial grading feature information 10-dimensional vector input stone grading data fitting module, the initial stone grading feature extraction module generated by the stone grading feature correction error, get further accurate prediction results, wherein the improved multilayer perception is a multilayer neural network with hidden state, which is composed of input layer, hidden variable with hidden state and output layer, and the network structure is shown in Figure 4 The specific situation is as follows:

[0073] The first part is the input layer, and the number of neurons is d=10, which corresponds to the input of the 10-dimensional feature vector of the initial stone grading information. At time step t, a small batch of initial stone grading feature samples X t ∈R n×d , where R is a real number, n is the batch size 6000, and d is equal to 10, which is the input feature number;

[0074] The second part is the hidden variable H t with hidden state, which is determined by the input of the input layer at the current time step, i.e. the input X t of the small batch of initial stone grading feature samples at the current time step t and the hidden variable H t-1 at the previous time step t-1. For a small batch of n initial stone grading feature sequence samples, each row of X t corresponds to the initial stone grading feature sample at time step t from the sequence. Unlike the original multilayer perception, the improved multilayer perception saves the hidden variable H t-1 at the previous time step and introduces a new weight parameter W hh ∈R h×h to describe how to use the hidden variable at the previous time step in the current time step, and the process is as follows formula (1):

[0075] H t =φ(X t W xh +H t-1 W hh +b h ) (1)

[0076] In the formula, X t ∈R n×d is the input of the small batch of initial stone grading feature samples at time step t, n is the batch size 6000, d is the input feature number 10, h is the number of hidden units, φ is the nonlinear activation function ReLU, and the weight parameters are W xh ∈R d ×h and W hh ∈R h×h , Wxh ∈R d×h b h ∈R 1×h , b h ∈R 1×h H t , b t ∈R n×h ;

[0077] The third part is the output layer, which is a fully connected layer. The neurons in the output layer are also fully connected with the neurons in the hidden layer. The number of neurons in the output layer is q, which is equal to 10, corresponding to the 10 sieve sizes of the new standard square-hole sand and gravel sieve. The process is as follows formula (2):

[0078] O t = H t W hq + b q (2)

[0079] In the formula, H t ∈R n×h is the hidden variable of time step t, n is the batch size, h is the number of hidden units of the hidden variable, q is the number of output units, and W hq ∈R h×q and b q ∈R 1×q are the weight parameters and bias parameters of the output layer, respectively. The output of the output layer is O t , and O t ∈R n×q ;

[0080] The initial stone grading features of all pictures in the stone data set extracted in step 3) and the true labels obtained from the artificial screening experiment in step 1) are used as the data set of the improved multilayer perceptron, and the training set, validation set and test set are divided. Then, 8 selectable improved multilayer perceptron candidate models are constructed. In each candidate model, the cross-entropy loss function is used to calculate the difference between the initial stone grading features and the true labels on the training set. Gradient update training of the improved multilayer perceptron candidate model is performed using back propagation, and the optimal weight parameters of each candidate model are obtained. Then, the validation set is used to evaluate the training error performance of the 8 candidate models, and the optimal model is selected from them. The test set is used to calculate the generalization error of the optimal model. Finally, the optimal model of the improved multilayer perceptron is deployed online for real-time prediction of stone automatic grading in enterprise production. The experimental results show that the error is controlled within 7%, meeting the production requirements of the enterprise for concrete stone grading.

[0081] Embodiment 2

[0082] The embodiment discloses a stone automatic grading system based on deep learning, referring to Figure 5 As shown in the figure, the system comprises the following functional modules:

[0083] The data acquisition module is used for collecting the high-speed belt transportation video of each batch of stone, downsampling each video, retaining a number of pictures for each stage of each video, naming and arranging in the format of "production channel / stone batch / frame serial number", sampling each batch of stone by four-point method and completing manual screening experiment, obtaining the true label of each batch of stone according to the manual screening experiment, naming and arranging in the format of "production channel / stone batch", forming a stone original data set;

[0084] The stone picture quality evaluation module is used for evaluating the quality of the picture and calculating the definition score, obtaining the picture definition score sorting of the stone original data set, and selecting the picture with high definition score as the stone data set according to the definition score sorting;

[0085] The stone grading feature extraction module is used for effectively extracting the initial stone grading feature information of the picture screened by the stone picture quality evaluation module, and generating the initial stone grading feature result of all pictures in the stone data set;

[0086] The stone grading data fitting module is used for data fitting on the initial stone grading feature result extracted by the stone grading feature extraction module, making up the stone grading error caused by the lack of stone depth information, and obtaining the error corrected stone grading feature prediction result.

[0087] Further, the data acquisition module specifically performs the following operations:

[0088] First, the whole process of each batch of stone transportation is recorded by an industrial camera installed on the belt, and each complete stone transportation process is stored as a video. Then, ffpmeg is used to slice each video, and every 5 seconds, an image is stored, and the images are arranged in order according to the format "production channel / stone batch / frame number". At the same time, workers sample the stones on the belt according to the quartile method at the same interval, and conduct manual screening experiments on the stones by using a new standard square hole sandstone sieve. The proportion of 10 classifications of each batch of stone is packaged into a real label in the form of a tensor, and arranged in order according to the format "production channel / stone batch number". The quartile method refers to grinding the stone sample according to the measurement requirements, passing through a sieve with a specific aperture, then mixing and laying out in a circle, dividing into four equal parts, taking the opposite two parts, then dividing again until the required size is reached. The manual screening experiment refers to manual screening by workers using a new standard square hole sandstone sieve. The new standard square hole sandstone sieve is divided into 10 sieve sizes, with sieve sizes of 53-37.5, 37.5-31.5, 31.5-26.5, 26.5-19.0, 19.0-16.0, 16.0-9.5, 9.5-4.75, 4.75-2.36, 2.36-bottom disc. The sieve size unit is mm. Finally, the real label of each batch of stone is mapped one-to-many with the multiple images obtained by slicing the video of each batch of stone transportation, and saved to the database to form the stone original data set.

[0089] Further, the stone picture quality evaluation module is a residual neural network based on multi-level features, and the network structure is shown in Figure 2 The network is an end-to-end structure, which can be divided into a base learner and an ensemble learner. The base learner is composed of a resnet18 residual convolutional neural network, and its main purpose is to learn different types of hierarchical features in stone pictures and extract initial features of stone pictures. The ensemble learner includes three Encoder encoder layers and a Concat operation, which aims to transform the channel of the feature representation of the stone picture after three stages of residual convolutional neural network. The specific structure of each part of the network is as follows:

[0090] The base learner includes three different stages, each stage includes different number of residual blocks, the residual block is composed of a 1*1 convolution layer, two 3*3 convolution layers, two batch normalization layers and two ReLU activation functions, the first stage includes a 7*7 convolution layer, a batch normalization layer, a max pooling layer and two residual blocks, shallow feature information of the stone picture is extracted through convolution operation, the second stage includes four consecutive residual block operations, aiming to further abstract the information and features in the stone picture, the third stage is two residual blocks, aiming to learn and extract high-level semantic information contained in the stone picture, feature information in the field of view is extracted through convolution operation, and the field of view is expanded through continuous convolution to obtain the relationship between different feature values, and the down-sampling operation reduces the resolution of the feature map, thereby saving the loss of calculation amount and memory.

[0091] The integrated learner includes three Encoder encoder layers, the main purpose of which is to integrate the initial features of the stone picture extracted by the base learner to form the overall representation of the stone picture, wherein the three Encoder encoder layers all include a 1*1 convolution layer, a 3*3 convolution layer and an average pooling layer, but are not all the same, the channel numbers of the 1*1 convolution layers are 16, 4 and 4 respectively, the strides of the 3*3 convolution layers are 7, 4 and 2 respectively, and the padding of the 3*3 convolution layers is 1, 3 and 2 respectively. In order to ensure the integrity of the data, the Concat operation is used to complete the fusion of the hierarchical features after the channel transformation of the three Encoder encoder layers, and the FC layer is used to map them to 256-dimensional overall features, calculate the stone picture quality definition score, and finally delete the pictures with scores less than the set threshold score, the threshold is set to 60, form the stone data set, and do flip and rotation operations on the stone images in the stone data set to perform data augmentation, wherein the flip includes horizontal flip, vertical flip and horizontal and vertical flip, and the rotation uses the values of-30°, -15°, 15° and 30°.

[0092] Further, the stone grading feature extraction module is a segmentation module based on digital image processing using watershed algorithm and concave point detection algorithm, the main purpose of which is to effectively extract the initial stone grading feature information of the picture to generate the initial stone grading feature result, the segmentation module includes stone morphological image processing, stone image segmentation and stone particle calibration and granularity calculation three parts, the structure is shown in Figure 3 The specific case is as follows:

[0093] The first part is a morphological image processing of stone, and the main purpose is to perform a morphological image processing operation on a stone picture to obtain an image that can meet the image segmentation condition, wherein the morphological image processing operation includes stone image preprocessing and stone morphological optimization, the stone image preprocessing is composed of a series of operations such as image gray scale transformation, image filtering processing and image binarization processing, and the main purpose is to pre-process the image to enhance the image quality, the image gray scale transformation is to perform brightness transformation processing on the pictures in the stone data set to enhance or weaken the brightness of the image, then the image after brightness transformation is subjected to gray scale processing to reduce the data amount in the image, the image filtering processing is to use bilateral filtering to perform denoising processing on the stone image after image gray scale transformation while retaining the edge information of the stone particles, and the image binarization processing is to perform fast adaptive thresholding processing on the stone image after image filtering processing to achieve the effect of image simplification; the stone morphological optimization is composed of a series of erosion and dilation combined opening and closing operations, and the main purpose is to perform morphological operation on the image after stone image preprocessing to achieve the purpose of denoising, smoothing, detecting and analyzing the stone particle image;

[0094] The second part is stone image segmentation, and the main purpose is to effectively segment the stone particles in the image after the stone morphological image processing, wherein the stone image segmentation adopts a watershed algorithm and a concave point detection algorithm, the watershed algorithm refers to a dam set between each water area in a topographic map, and a gray scale image is regarded as a topographic map, the gray scale value of a pixel point in the gray scale image is regarded as an altitude, and different gray scale values of pixel points represent different altitudes, the place with low gray scale value has low altitude, and the place with high gray scale value has high altitude, but directly performing watershed algorithm segmentation on the image will lead to over-segmentation phenomenon, loss of image edge information, and the problem that the image with large-area adhesion of stone particles cannot be effectively segmented, therefore, the concave point detection algorithm is introduced to solve the problem, and experiments show that the combination of the two can effectively segment the adhesion particles in the stone image;

[0095] The third part is stone particle calibration and granularity calculation, and the main purpose is to calculate the contour area of each stone particle on the image after stone image segmentation, divide the stones into 10 different sizes of particle diameters according to the contour area, and count the number of stones of the 10 different sizes of particle diameters, wherein the stone particle calibration and granularity calculation calibrate the stone particles on site and statistically analyze the distribution information of the 10 different sizes of particles in the stone to obtain the proportion of each particle size, and the proportions of the 10 different particle sizes correspond to the stone initial grading feature information of a 10-dimensional vector, so as to effectively extract the initial stone grading feature information of the picture, obtain the initial stone grading feature result, and the stone initial grading feature information of the 10-dimensional vector corresponds to the 10 sieve hole sizes of the new standard square hole sand screen.

[0096] Further, the stone grading data fitting module is an improved multilayer perceptron, and the main purpose is to solve the problem that the surface stone particle size distribution obtained by the image segmentation of the stone image by the segmentation module cannot obtain the deep particle size distribution of the batch of stones. The improved multilayer perceptron can predict the overall stone particle size distribution under the condition of stone stacking on the belt through the surface stone particle size distribution, and correct the error of the initial stone grading feature results of all the pictures generated by the stone grading feature extraction module. In this embodiment, the 10-dimensional vector of the initial stone grading feature information of 6000 stones generated by the stone grading feature extraction module is input into the stone grading data fitting module, the initial stone grading feature results generated by the stone grading feature extraction module are corrected by data fitting, and a further accurate prediction result is obtained. The improved multilayer perceptron is a multilayer neural network with hidden states, which is composed of an input layer, a hidden variable with hidden states, and an output layer. The network structure is shown in Figure 4 , and the specific conditions are as follows:

[0097] The first part is the input layer, and the number of neurons is d=10, which corresponds to the input of the 10-dimensional feature vector of the initial stone grading information. At time step t, a small batch of initial stone grading feature samples X t ∈R n×d , where R is a real number, n is the batch size 6000, and d is equal to 10, which is the input feature number;

[0098] The second part is the hidden variable H t with hidden states, which is determined by the input layer of the current time step, i.e. the input X t of the small batch of initial stone grading feature samples at the current time step t and the hidden variable H t-1 of the previous time step t-1. For a small batch of n initial stone grading feature sequence samples, each row of X t corresponds to an initial stone grading feature sample at time step t from the sequence. Unlike the original multilayer perceptron, the improved multilayer perceptron saves the hidden variable H t-1 of the previous time step and introduces a new weight parameter W hh ∈R h×h to describe how to use the hidden variable of the previous time step in the current time step. The process is as follows formula (1):

[0099] H t =φ(X t W xh +H t-1 W hh +b h ) (1)

[0100] In the formula, X t∈R n×d is the input of the initial stone batch classification feature sample at time step t, n is the batch size 6000, d is the number of input features 10, h is the number of hidden units, φ is the nonlinear activation function ReLU, and the weight parameters are W xh ∈R d ×h and W hh ∈R h×h , W xh ∈R d×h describes how to use the input value of the current time step in the current time step, and the bias unit is b h ∈R 1×h , b h ∈R 1×h describes how to use the bias unit in the current time step, and the output is the hidden variable H t at time step t t ∈R n×h ;

[0101] The third part is the output layer, which is a fully connected layer. The neurons in the output layer are also fully connected with each neuron in the hidden layer. The number of neurons in this layer is q, which is equal to 10, corresponding to the 10 sieve sizes of the new standard square hole sandstone screen. The process is as follows formula (2):

[0102] O t = H t W hq +b q (2)

[0103] In the formula, H t ∈R n×h is the hidden variable at time step t, n is the batch size, h is the number of hidden units of the hidden variable, q is the number of output units, and the weight parameters and bias parameters of the output layer are W hq ∈R h×q and b q ∈R 1×q , respectively. The output of the output layer is O t , O t ∈R n×q ;

[0104] The initial stone grading features of all pictures of the stone data set are extracted by the stone grading feature extraction module, and the true labels obtained by manual screening experiments in the corresponding data acquisition module are taken as the data set of the improved multilayer perceptron, and the training set, the validation set and the test set are divided, then 8 selectable improved multilayer perceptron candidate models are constructed, the difference between the initial stone grading features and the true labels on the training set is calculated in each candidate model using the cross-entropy loss function, and the gradient update training of the improved multilayer perceptron candidate model is carried out using back propagation, the optimal weight parameters of each candidate model are obtained, then the training error performance of the 8 candidate models is evaluated using the validation set and the optimal model is selected from them, the generalization error of the optimal model is calculated using the test set, finally the improved multilayer perceptron optimal model is deployed online, which is used for real-time prediction of stone automatic grading in enterprise production, and the experimental results show that the error is controlled within 7%, which meets the production requirements of the enterprise for concrete stone grading.

[0105] In summary, after adopting the above scheme, the present application provides a new method and system for stone grading, uses deep learning as an effective tool for stone grading, can effectively solve the problem that computers are difficult to automatically and accurately identify stone grading, effectively promotes the development of computer-aided automatic stone grading, has practical popularization value, and is worth popularizing.

[0106] The above embodiments are preferred embodiments of the present application, but the embodiments of the present application are not limited by the above embodiments, and any changes, modifications, substitutions, combinations and simplifications made without departing from the spirit and principles of the present application should be equivalent replacement methods, and are all included in the protection scope of the present application.

Claims

1. A method for automatic stone classification based on deep learning, characterized in that, The method is based on deep learning and digital image processing technology to realize the automatic grading task of stone in industrial belt high-speed transportation scene, which specifically adopts three modules, namely stone picture quality evaluation module, stone grading feature extraction module and stone grading data fitting module; wherein, the stone picture quality evaluation module is a residual neural network based on multi-level features, which is used for evaluating and screening high-quality dynamic frames; the stone grading feature extraction module is a segmentation module based on digital image processing, which adopts watershed algorithm and concave point detection algorithm, and is used for extracting stone grading feature information from the pictures screened by the stone picture quality evaluation module; the stone grading data fitting module is an improved multilayer perceptron, which is used for data fitting of the initial stone grading feature results extracted by the stone grading feature extraction module, making up for the stone grading error caused by the lack of stone depth information, and realizing error correction; The specific implementation of the stone automatic grading method includes the following steps: 1) Collect each batch of stone in the belt high-speed transportation video, downsample each video, keep several pictures for each stage of each video, name and arrange them in the format of "production channel / stone batch / frame number", at the same time, sample each batch of stone by four division method and complete manual screening experiment, get the true label of each batch of stone according to the manual screening experiment, name and arrange them in the format of "production channel / stone batch number", form the stone original data set; 2) input the pictures of stone original data set into stone picture quality evaluation module, evaluate the quality of the pictures and calculate the definition score, get the definition score sorting of stone original data set, select the pictures with high definition score as stone data set according to the definition score sorting; The stone picture quality evaluation module is a residual neural network based on multi-level features, and the main purpose is to eliminate abnormal frame data in the stone original data set, including frame data with full picture blur, large-scale ghosting, extreme lighting and incomplete shooting in the acquisition process; the residual neural network based on multi-level features includes a base learner and an integrated learner, the base learner is composed of a residual convolutional neural network, and the main purpose is to learn different types of hierarchical features in stone pictures and extract initial features of stone pictures; wherein the residual convolutional neural network includes three different stages, each stage includes different number of residual blocks; the first stage includes one 7*7 convolutional layer, one batch normalization layer, one maximum pooling layer and two residual blocks, aiming at initial preprocessing of stone pictures and shallow feature extraction; the second stage is four residual blocks, aiming at further abstracting information and features in stone pictures; the third stage is two residual blocks, aiming at learning and extracting high-level semantic information contained in stone pictures; the integrated learner includes three different encoder layers and a Concat operation, wherein the main purpose of the three different encoder layers is to integrate the initial features of stone pictures extracted by the base learner to form the overall representation of the stone pictures, the encoder layers all include a 1*1 convolutional layer, a 3*3 convolutional layer and an average pooling layer, but the difference is that the channel numbers of the 1*1 convolutional layer are 16, 4 and 4 respectively, the strides of the 3*3 convolutional layer are 7, 4 and 2 respectively, and the padding of the 3*3 convolutional layer is 1, 3 and 2 respectively, the encoder layers aim to transform the channel of the representation features of the stone pictures after the three stages of the residual convolutional neural network; in order to ensure the integrity of the data, the Concat operation is used to fuse the hierarchical features after the channel transformation of the three different encoder layers, and a fully connected layer is used to map them to 256-dimensional overall features, calculate the stone picture quality definition score, finally delete the pictures with score less than the set threshold score according to the score, form the stone data set, and do flip and rotation operation on the stone images in the stone data set to expand the data, wherein the flip includes horizontal flip, vertical flip and horizontal and vertical flip, the rotation uses 、 、 and ​ 3) take the stone data set obtained in step 2) as the input data of stone grading feature extraction module, effectively extract the initial stone grading feature information of the pictures, generate the initial stone grading feature results of all pictures in stone data set; 4) input the initial stone grading feature results of all pictures generated by stone grading feature extraction module into stone grading data fitting module, correct the error of initial stone grading feature results by data fitting, get the stone grading feature prediction results after error correction.

2. The deep learning-based automatic stone classification method according to claim 1, characterized in that, In step 1), first, the video of the stone transportation process on the belt is collected by an industrial camera, and each batch of stone transportation process is stored as a video, then the images of each video are sliced at a specific time interval, and the images are named and arranged in the format of "production channel / stone batch / frame number", meanwhile, each batch of stone is sampled by the quartering method and the manual screening experiment is completed, the true label of each batch of stone is obtained according to the manual screening experiment, and is named and arranged in the format of "production channel / stone batch number"; wherein, the quartering method refers to grinding the stone sample according to the measurement requirements, passing through a sieve with a specific aperture, then mixing and laying flat into a circle, dividing into four equal parts, taking the opposite two parts, then dividing again until the required size is reached; the manual screening experiment refers to manual screening by workers through new standard square hole sandstone screen, which is divided into 10 screen sizes, and the screen sizes are 53~37.5, 37.5~31.5, 31.5~26.5, 26.5~19.0, 19.0~16.0, 16.0~9.5, 9.5~4.75, 4.75~2.36, 2.36~bottom disc, the screen unit is mm, finally, the true label of each batch of stone is mapped to the multiple images obtained by slicing the video of each batch of stone transportation, and the stone original data set is made.

3. The deep learning-based automatic stone classification method according to claim 1, characterized in that, In step 3), the stone grading feature extraction module is a segmentation module based on digital image processing using watershed algorithm and concave point detection algorithm, its main purpose is to effectively extract the initial grading feature information of the stone picture, and generate the initial stone grading feature result, the segmentation module includes three parts of stone morphological image processing, stone image segmentation and stone particle calibration and particle size calculation, the specific conditions are as follows: The first part is stone morphological image processing, its main purpose is to perform morphological image processing operation on the stone picture to obtain an image that meets the image segmentation condition, wherein the morphological image processing operation includes stone image preprocessing and stone morphological optimization, the stone image preprocessing is composed of image gray scale transformation, image filtering processing and image binarization processing, its main purpose is to preprocess the image to enhance the image quality, the image gray scale transformation is to perform brightness transformation processing on the pictures in the stone data set to enhance or weaken the brightness of the image, then the image after brightness transformation is processed by gray scale to reduce the data amount in the image, the image filtering processing is to use bilateral filtering to denoise the stone image after image gray scale transformation while preserving the edge information of the stone particles, the image binarization processing is to perform fast adaptive thresholding processing on the stone image after image filtering processing to achieve the effect of image simplification; the stone morphological optimization is composed of a series of erosion and dilation combined opening and closing operation, its main purpose is to perform morphological operation on the image after stone image preprocessing to achieve the purpose of denoising, smoothing, detecting and analyzing the stone particle image; The second part is stone image segmentation, and the main purpose is to effectively segment the stone particles in the image after the stone morphological image processing, wherein the stone image segmentation is achieved by using a watershed algorithm and a concave point detection algorithm; the watershed algorithm refers to a dam set between each water area in a topographic map, and a gray-scale image is regarded as a topographic map, and the gray value of a pixel point in the gray-scale image is regarded as an altitude, and different gray values of pixel points represent different altitudes, and the place with low gray value has low altitude, and the place with high gray value has high altitude; however, directly performing the watershed algorithm on the image may cause over-segmentation, loss of image edge information, and ineffective segmentation of the image with large-area adhered stone particles, and therefore, the concave point detection algorithm is introduced to solve the problem, and the combination of the two can effectively segment the adhered particles in the stone image; The third part is stone particle calibration and granularity calculation, and the main purpose is to calculate the contour area of each stone particle on the image after the stone image segmentation, divide the stones into 10 different sizes according to the contour area, and count the number of stones in the 10 different sizes, wherein the stone particle calibration and granularity calculation calibrate the stone particles on site, count and analyze the distribution information of the 10 different sizes of the stone, obtain the proportion of each size of particles, and correspond the proportions of the 10 different sizes of particles to the stone initial classification feature information of a 10-dimensional vector, so as to effectively extract the initial stone classification feature information of the image, obtain the initial stone classification feature result, and the stone initial classification feature information of the 10-dimensional vector corresponds to the 10 screen hole sizes of the new standard square-hole sandstone screen.

4. The deep learning-based automatic stone classification method according to claim 1, characterized in that, In step 4), the stone classification data fitting module is an improved multilayer perception machine, and the main purpose is to solve the problem that the stone image segmentation of the segmentation module cannot obtain the deep granularity distribution of the stone from the surface granularity distribution of the stone, the improved multilayer perception machine can predict the overall stone granularity distribution under the condition of the stone stacking on the belt through the surface granularity distribution of the sandstone, correct the error of the initial stone classification feature result of all images generated by the stone classification feature extraction module, and the improved multilayer perception machine is a multilayer neural network with hidden states, which is composed of an input layer, a hidden variable with hidden states and an output layer, and the specific conditions are as follows; The first part is an input layer, the number of neurons of which is d, and respectively corresponds to the d-dimensional vector of the initial stone grading feature information generated by the stone grading feature extraction module, and there is a small batch of initial stone grading feature samples input at time step t wherein R is a real number, n is a batch size, d is the number of neurons, and is also the number of input features; The second part is a hidden variable with hidden state , which is determined by the input layer of the current time step, that is, the input of the initial stone grading feature sample of the current time step t small batch The hidden variable of the previous time step t-1 Decided, for n initial stone grading feature sequence samples in small batches, Each row corresponds to the initial stone grading feature sample at time step t from the sequence, unlike the original multilayer perceptron, the improved multilayer perceptron saves the hidden variable of the previous time step And introduce a new weight parameter To describe how to use the hidden variable of the previous time step in the current time step, the process is as follows formula (1): （1）； wherein, is the input of the initial stone batch classification feature sample at time step t, n is the batch size, d is the number of input features, h is the number of hidden units, is the nonlinear activation function ReLU, and the weight parameters are and , describes how to use the input value of the current time step in the current time step, the bias unit is , describes how to use the bias unit in the current time step, and the output is the hidden variable at time step t , has ; The third part is an output layer, and the output layer is a fully connected layer, and the neurons in the output layer are also fully connected with the neurons in the hidden layer, the number of neurons in the layer is q, and the q screen hole sizes of the new standard square-hole sandstone screen are respectively corresponded, and the process is as follows formula (2): （2）； In the formula, is the hidden variable of time step t, the batch size is n, the number of hidden units of the hidden variable is h, the number of output units is q, and the weight parameter and the bias parameter of the output layer are and respectively. and The output of the output layer is , and ; The initial stone grading features of all pictures of the extracted stone data set in step 3) and the true labels obtained by the artificial screening experiment in step 1) are taken as the data set of the improved multilayer perceptron, and the training set, the validation set and the test set are divided, then a plurality of selectable improved multilayer perceptron candidate models are constructed, the difference between the initial stone grading features and the true labels on the training set is calculated using the cross-entropy loss function in each candidate model, and the gradient update training of the improved multilayer perceptron candidate model is carried out using back propagation, the optimal weight parameters of each candidate model are obtained, then the performance of several candidate models is evaluated using the validation set and the optimal model is selected from them, the generalization error of the optimal model is calculated using the test set, and finally the improved multilayer perceptron optimal model is deployed online for real-time prediction of stone automatic grading in production.

5. Automatic stone grading system based on deep learning, characterized by, It comprises: A data acquisition module is used to acquire high-speed belt transportation videos of each batch of stone, downsample each video, retain a number of pictures for each stage of each video, arrange and name them in the format of "production channel / stone batch / frame sequence number", and at the same time, sample each batch of stone using the four-point method and complete the artificial screening experiment, obtain the true labels of each batch of stone according to the artificial screening experiment, arrange and name them in the format of "production channel / stone batch number", form a stone original data set; A stone picture quality evaluation module is used to evaluate the quality of the pictures and calculate their sharpness scores, obtain the sharpness score ranking of the pictures in the stone original data set, and select pictures with high sharpness scores as the stone data set according to the sharpness score ranking; A stone grading feature extraction module is used to effectively extract the initial stone grading feature information of the pictures screened by the stone picture quality evaluation module, and generate the initial stone grading feature results of all pictures in the stone data set; A stone grading data fitting module is used to fit the initial stone grading feature results extracted by the stone grading feature extraction module, make up for the stone grading errors caused by the lack of stone depth information, and obtain the error-corrected stone grading feature prediction results; The stone picture quality evaluation module is a residual neural network based on multi-level features, and the main purpose is to eliminate abnormal frame data in the stone original data set, including frame data with full picture blur, large-scale ghosting, extreme lighting and incomplete shooting in the acquisition process; the residual neural network based on multi-level features includes a base learner and an integrated learner, the base learner is composed of a residual convolutional neural network, and the main purpose is to learn different types of hierarchical features in stone pictures and extract initial features of stone pictures; wherein the residual convolutional neural network includes three different stages, each stage includes different number of residual blocks; the first stage includes one 7*7 convolutional layer, one batch normalization layer, one maximum pooling layer and two residual blocks, aiming at initial preprocessing of stone pictures and shallow feature extraction; the second stage is four residual blocks, aiming at further abstracting information and features in stone pictures; the third stage is two residual blocks, aiming at learning and extracting high-level semantic information contained in stone pictures; the integrated learner includes three different encoder layers and a Concat operation, wherein the main purpose of the three different encoder layers is to integrate the initial features of stone pictures extracted by the base learner to form the overall representation of the stone pictures, the encoder layers all include a 1*1 convolutional layer, a 3*3 convolutional layer and an average pooling layer, but the difference is that the channel numbers of the 1*1 convolutional layer are 16, 4 and 4 respectively, the strides of the 3*3 convolutional layer are 7, 4 and 2 respectively, and the padding of the 3*3 convolutional layer is 1, 3 and 2 respectively, the encoder layers aim to transform the channel of the representation features of the stone pictures after the three stages of the residual convolutional neural network; in order to ensure the integrity of the data, the Concat operation is used to fuse the hierarchical features after the channel transformation of the three different encoder layers, and a fully connected layer is used to map them to 256-dimensional overall features, calculate the stone picture quality definition score, finally delete the pictures with scores less than the set threshold score according to the score, form the stone data set, and do flip and rotation operation on the stone images in the stone data set to expand the data, wherein the flip includes horizontal flip, vertical flip and horizontal and vertical flip, the rotation uses the value of 、 、 and .

6. The deep learning based automatic stone classification system according to claim 5, characterized in that, The data acquisition module specifically performs the following operations: First, the video of the stone transportation process on the belt is collected by an industrial camera, and each batch of stone transportation process is stored as a video. Then, the images of each video are sliced at a specific time interval, and the images are named and arranged in the format of "production channel / stone batch / frame number". Meanwhile, each batch of stone is sampled using the quartering method and the manual screening experiment is completed. The true label of each batch of stone is obtained according to the manual screening experiment, and is named and arranged in the format of "production channel / stone batch number". The quartering method refers to grinding the stone sample according to the measurement requirements, passing through a specific aperture sieve, then mixing and laying flat into a circle, dividing into four equal parts, taking the opposite two parts, then dividing again until the required size is reached. The manual screening experiment refers to manual screening by workers using a new standard square hole sandstone sieve. The new standard square hole sandstone sieve is divided into 10 sieve sizes, with sieve sizes of 53-37.5, 37.5-31.5, 31.5-26.5, 26.5-19.0, 19.0-16.0, 16.0-9.5, 9.5-4.75, 4.75-2.36, 2.36-bottom disc. The sieve unit is mm. Finally, the true label of each batch of stone is mapped one-to-many with the multiple images obtained by slicing the transportation video of each batch of stone to form the stone original data set.

7. The deep learning based automatic stone classification system according to claim 5, wherein, The stone grading feature extraction module is a segmentation module based on digital image processing using the watershed algorithm and concave point detection algorithm. Its main purpose is to effectively extract the initial grading feature information of the stone image and generate the initial stone grading feature result. The segmentation module includes three parts: stone morphological image processing, stone image segmentation, and stone particle calibration and particle size calculation. The specific conditions are as follows: The first part is stone morphological image processing, which mainly aims to perform morphological image processing operations on the stone image to obtain an image that meets the image segmentation conditions. The morphological image processing operations include stone image preprocessing and stone morphological optimization. Stone image preprocessing is composed of image grayscale transformation, image filtering processing, and image binarization processing. Its main purpose is to preprocess the image to enhance the image quality. Image grayscale transformation is to perform brightness transformation on the stone image in the stone data set to enhance or weaken the brightness of the image. Then the brightness transformed image is grayed to reduce the data amount in the image. Image filtering processing uses bilateral filtering to denoise the stone image after image grayscale transformation while preserving the edge information of the stone particles. Image binarization processing uses fast adaptive thresholding to process the stone image after image filtering to achieve image simplification. Stone morphological optimization is composed of a series of erosion and dilation combined opening and closing operations. Its main purpose is to perform morphological operations on the image after stone image preprocessing to achieve the purpose of denoising, smoothing, detecting, and analyzing the stone particle image. The second part is stone image segmentation, and the main purpose is to effectively segment the stone particles in the image after the stone morphological image processing, wherein the stone image segmentation is achieved by using a watershed algorithm and a concave point detection algorithm. The third part is stone particle calibration and granularity calculation, and the main purpose is to calculate the contour area of each stone particle on the image after the stone image segmentation, divide the stones into 10 different sizes according to the contour area, and count the number of stones of the 10 different sizes.

8. The deep learning based automatic stone classification system according to claim 5, wherein, The stone grading data fitting module is an improved multilayer perception machine, and the main purpose is to solve the problem that the surface stone granularity distribution obtained by the segmentation module cannot obtain the deep granularity distribution of the stone. The first part is an input layer, the number of neurons of which is d, and respectively corresponds to the d-dimensional vector of the initial stone grading feature information generated by the stone grading feature extraction module, and there is a small batch of initial stone grading feature samples input at the time step t wherein R is a real number, n is the batch size, d is the number of neurons, and is also the number of input features; The second part is a hidden variable with hidden state , which is determined by the input layer of the current time step, that is, the input of the initial stone grading feature sample of the current time step t small batch and the hidden variable of the previous time step t-1 , for n initial stone grading feature sequence samples in a small batch Each row corresponds to the initial stone grading feature sample at time step t from the sequence, unlike the original multilayer perceptron, the improved multilayer perceptron saves the hidden variable of the previous time step And introduce a new weight parameter To describe how to use the hidden variable of the previous time step in the current time step, the process is as follows formula (1): （1）； In the formula, is the input of the initial stone batch classification feature sample at time step t, n is the batch size, d is the number of input features, h is the number of hidden units, is a nonlinear activation function ReLU, and the weight parameter has and , describes how to use the input value of the current time step in the current time step, the bias unit is , describes how to use the bias unit in the current time step, and the output is the hidden variable of time step t , has ; The third part is the output layer, which is a fully connected layer, and the neurons in the output layer and the neurons in the hidden layer are also fully connected, and the number of neurons in this layer is q, which corresponds to the q sieve hole sizes of the new standard square hole sandstone sieve, and the process is as follows formula (2): （2）； In the formula, is the hidden variable of time step t, the batch size is n, the number of hidden units of the hidden variable is h, the number of output units is q, and the weight parameter and the bias parameter of the output layer are and respectively and The output of the output layer is , and ; The initial stone grading features of all pictures of the stone data set are extracted by the stone grading feature extraction module, and the true labels obtained by manual screening experiments in the corresponding data collection module are used as the data set of the improved multilayer perceptron, and the training set, the validation set and the test set are divided, then a plurality of selectable improved multilayer perceptron candidate models are constructed, the difference between the initial stone grading features and the true labels on the training set is calculated in each candidate model using the cross-entropy loss function, and the gradient update training of the improved multilayer perceptron candidate model is carried out by back propagation, the optimal weight parameters of each candidate model are obtained, then the performance of several candidate models is evaluated using the validation set and the optimal model is selected from them, the generalization error of the optimal model is calculated using the test set, and finally the improved multilayer perceptron optimal model is deployed online for real-time prediction of stone automatic grading in production.

Images