Complex background material tracking method
By adding unique id encoding to the material surface and combining CNN and LSTM networks for feature extraction and sorting, the problems of low recognition accuracy and high labor costs of material tracking in complex backgrounds are solved, and efficient material identification and management are achieved, reducing Deep learning algorithm implementation time.
Patent Information
- Application Number
- CN202411264778.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-10
- Publication Date
- 2024-12-27
AI Technical Summary
In complex background environments, there are problems such as difficulty in marking, low recognition accuracy, difficulty in material management, large labor costs, unclear material consumption data and inability to scientifically control. Traditional deep learning algorithms collect data in a time-consuming and labor-intensive manner and insufficient accuracy.
Add unique id encoding to the material surface, adjust the light source and camera to take images, combine CNN and LSTM networks to extract and sort features, generate initial models, and update model parameters through incremental learning, and use image features and gradient information for error detection Exclusion and weight correction to achieve accurate tracking and identification of materials.
It improves the accuracy of material recognition in high temperature and complex backgrounds, reduces the implementation time of deep learning algorithms, saves manual labeling costs, and speeds up the recognition speed and accuracy through online data updates.
Smart Images
Figure CN119205842A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to a method for tracking materials in complex backgrounds, belonging to the technical field of vision. Background Art
[0002] With the rapid development and application of deep learning, the visual field has gotten rid of many limitations of traditional machine vision and has been widely used in every corner of the industry after the addition of deep learning. However, when tracking materials in complex production environments such as high temperature, changing background environment, and severe wear, there are often problems such as difficulty in marking, low recognition accuracy, difficulty in material management, high labor costs, unclear material consumption data, and inability to scientifically control materials. In addition, directly using traditional algorithms to collect data for labeling and training models is time-consuming and labor-intensive, and the accuracy is insufficient.
[0003] Based on this, the present invention is proposed. Summary of the invention
[0004] In view of the shortcomings of the prior art, the present invention provides a method for tracking materials in a complex background. The specific technical solution is as follows:
[0005] A complex background material tracking method includes the following steps:
[0006] Step 1: Select appropriate pigments and add at least one unique ID code to the surface of the material to be tracked;
[0007] Step 2: Adjust the light source and the camera until the clarity of the image of the ID code captured by the camera meets the preset requirements;
[0008] Step 3: Use characters similar to the surface ID code of the material to be tracked and randomly combine these similar characters, and randomly generate a certain distance between the characters to generate first character data;
[0009] Collect the original image of the surface of the material to be tracked and extract the surface features, and after image processing, fuse it with the first character data to obtain the second character data;
[0010] The second character data and the original image of the material surface are divided into a certain proportion and then fused with the collected original image of the material surface at a relative position;
[0011] Take each generated feature map after fusion and its corresponding ID code as input, build a CNN network and extract features, hide some features according to the actual situation, then input it into the LSTM network and sort the characters, output the predicted ID label, and thus generate the initial model M0;
[0012] Step 4: Deploy the initial model M0 to the environment to collect images of materials with actual ID codes and predict new ID labels; on the same material, when the same ID code is present and the number of the same ID codes is greater than 1, send all ID labels and image data with ID codes to the initial model M0 for incremental learning and update model parameters; at the same time, when the initial model M0 starts predicting, the gradient information of the character area is obtained in the last layer of the CNN network;
[0013] The weight of the characters in the image area is recorded as input information, and the weight is also recorded when the same ID code is tracked next time, and this process is repeated, and the feature data of the ID code is used as a feature vector representing its erosion status; when performing incremental learning training, this feature vector is added to the character features in each ID code, and the model parameters are updated after the input time series is sorted. The updated model is the M1 model;
[0014] Step 5. In the final prediction stage, the possible probability of each character is predicted for each position of the ID tag. The prediction result is in matrix form. After softmax processing, it is spliced into a complete ID tag. Finally, the ID tag is transmitted to the corresponding position of the database for self-increment.
[0015] As a further improvement, in step three, the image processing includes opening and closing operations, adding noise, affine transformation, perspective change, and adding mask operations.
[0016] As a further improvement, in step three, the fusion with the first character data is performed using a mask.
[0017] As a further improvement, the fusion algorithm formula of the second character data and the original image of the material surface is divided into a certain proportion and then fused with the collected original image of the material surface at a relative position is:
[0018]
[0019] is a matrix A, which is a 3-channel feature map of the second character data surface; A 1 represents the value of the relative position on a single channel, and β represents the number of values on a single channel;
[0020] is the matrix G, which is the feature vector extracted from the three channels of RGB on the original image surface of the material surface; (s1, t1), (s2, t2), (s3, t3) correspond to the coordinates of the three channels of RGB on the original image surface of the material surface respectively; G (s1,t1) , G (s2,t2) , G (s3,t3)It represents the coordinates (s1, t1), (s2, t2), (s3, t3) multiplied by the eigenvector;
[0021] is the matrix λ, which is the feature map of the training data to be input into the initial model M0. 1 , 2 , 3 Represents the eigenvalue.
[0022] As a further improvement, in step 4, for the characters corresponding to the same ID code, the gradient vector g is obtained by accumulating the gradient information captured each time. n , for the entire ID code, the corresponding two-dimensional gradient matrix g is obtained m×n ;
[0023]
[0024] in, is the two-dimensional gradient matrix g m×n , m is the number of characters in an ID code, and n is the number of the same ID code; is the image feature matrix, It is the gradient value of each channel at each position in the image feature matrix;
[0025] is the gradient matrix, is the gradient value of each channel at each position in the gradient matrix; μ is the coefficient for adjusting the weight of the gradient matrix and the image feature matrix;
[0026] After predicting the new ID tag, the data will enter the M1 model and be compared with other ID tags on the same material. When the number of identical ID tags identified on the same material is greater than or equal to 2, the incorrectly identified ID tag on the material and the features obtained for each character are input into the CNN network for feature extraction. The real material ID code is used as the target value for training, and a softmax process is performed on the predicted weight of each character to obtain all probability values. At the same time, the probability relationship matrix of all possible characters and real characters is shared with each character. The prediction result of each character can take into account the probability relationship of other characters that may be misdetected, thereby excluding the wrong characters.
[0027]
[0028] Where D is the matrix of all id labels on the same material, W is the Embedding parameter matrix generated on all id labels using the Glove method, p represents the association between the characters at the same position of all id labels on the same material, i is the number of characters in each id label, q is the association between id labels on the same material, j is the number of id labels on each material, Y is the association between the final prediction value and p and q, and d is the dimension of p and q; multiple associations are obtained for each material's id label, and the final weight Z is obtained through softmax, and then the final weight Z is assigned to the characters at each position in each id label;
[0029] Compared with the predicted correct ID label, its weight is corrected by:
[0030] {[g(y)-g(z)]*λ,g(x)*λ}=Y;
[0031] g(y) is the regular gradient actually captured, g(z) is the predicted regular gradient, g(x) is the image gradient, λ represents the eigenvalue; Y is the final predicted value;
[0032] The purpose of the correction is to minimize the difference between the two gradients g(y) and g(z), that is, after continuous additional weight correction, the predicted value is closer to the final target value;
[0033] The loss function Loss is divided into two parts: the error between the predicted ID label and the actual ID code is one part, and the error between g(y) and g(z) is one part each time. 1 Calculate the loss of character order, LOSS 2 The error between the target value and the predicted value is calculated based on the character at each position; where LOSS is calculated 1 When the loss is made, the input sequence is guaranteed to be greater than the label sequence;
[0034] Loss=Loss 1 +Loss 2 ;
[0035]
[0036] Among them, p(v|h) is the probability of the output sequence obtained for a given input h at any time; S is the training set, and v is the set of all characters contained in the id code;
[0037]
[0038] Where f is 0 or 1, p(f α) after softmax processing, the value range is (0,1); N is the width of each generated feature map, which contains two parts: characters and background, and i is the number of characters in each id label.
[0039] Beneficial effects of the present invention:
[0040] 1. The present invention uses image recognition methods and captures the rules of image background interference to identify ID codes and thus accurately track materials. It is also a deep learning method based on the combination of optical character recognition and false detection capture.
[0041] 2. The present invention aims at the problem that materials are difficult to control under high temperature and complex background environment in the industrial field. By adding ID code on the surface of materials, using a camera to collect the original image of the surface of materials, processing the original image, and establishing an initial model M0 and predicting a new ID tag; in the final prediction stage, the possible probability of each character is predicted for each position of the ID tag, and the prediction result is in the form of a matrix. After softmax processing, it is spliced into a complete ID tag, and finally the ID tag is transmitted to the corresponding position of the database for self-increment; the ID tag of each material is sent to the back-end database for unified management. The present invention can improve the recognition accuracy of character recognition when it is seriously interfered by external background, and speed up the implementation of the recognition algorithm.
[0042] 3. By collecting data online and continuously updating the model for online incremental learning, the cost of manual labeling can be saved and the time for implementing deep learning algorithms can be greatly reduced.
[0043] 4. Image feature engineering is combined with time series problems. Not only image features are used, but also the degree of character erosion and its correlation with the overall data are considered, which can further improve the recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] Figure 1 It is a simplified flow chart of the present invention;
[0045] Figure 2 This is the flowchart from when the algorithm starts to enter the automatic upgrade iteration part to predicting the final result. DETAILED DESCRIPTION
[0046] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.
[0047] Example 1
[0048] like Figure 1 , 2As shown, the complex background material tracking method comprises the following steps:
[0049] Step 1: Select appropriate pigments to add at least one unique ID code to the surface of the material to be tracked.
[0050] Step 2: Use a suitable camera and light source according to the type of material and the color of the pigment to be tracked, and adjust the light source and camera until the clarity of the image of the ID code captured by the camera meets the preset requirements.
[0051] Step 3: Generate initial data and initial model.
[0052] 1) Use characters similar to the ID code (abbreviated as ID) on the surface of the material to be tracked and randomly combine these similar characters, randomly generate a certain distance between the characters, and generate the first character data. The purpose of this step is to improve the adaptability of the initial model to the ID code, and also allow the initial model to recognize most characters when it is just launched. The definition of "similar" is: similar to the font and color of the characters on the actual material. If calculated through the similarity algorithm, a similarity of more than 80% can be determined to be similar.
[0053] 2) Collect the original image of the surface of the material to be tracked and extract the surface features. After opening and closing operations, adding noise, affine transformation, perspective change, and adding mask operations, fuse it with the first character data to obtain the second character data; use mask for fusion. Fusion can turn the original layer into pixel-level fusion, which can further improve the adaptability of the initial model.
[0054] 3) The second character data and the original image of the material surface are divided into a certain ratio (based on the actual number of characters used in the entire character set and the number of characters used on a single material; for example, when the original image data is insufficient, it can be divided into 2:1), and then fused with the collected original image of the material surface at a relative position.
[0055] The fusion algorithm formula is:
[0056]
[0057] is a matrix A, which is a 3-channel feature map of the second character data surface; A 1 represents the value of the relative position on a single channel, and β represents the number of values on a single channel;
[0058] is the matrix G, which is the feature vector extracted from the three RGB channels of the original image surface of the material surface; (s1, t1), (s2, t2), (s3, t3) correspond to the coordinates of the three RGB channels of the original image surface of the material surface, G (s1,t1) , G (s2,t2) , G (s3,t3) It represents the coordinates (s1, t1), (s2, t2), (s3, t3) multiplied by the eigenvector;
[0059] is the matrix λ, which is the feature map of the training data to be input into the initial model M0. 1 , 2 , 3 represents the eigenvalue;
[0060] Take each generated feature map after fusion and its corresponding ID code as input, build a CNN network and extract features, hide some features according to actual conditions (to increase the robustness of the model), then input it into the LSTM network and sort the characters, output the predicted ID label, and thus generate the initial model M0.
[0061] Step 4. Deploy the initial model M0 to the environment to collect actual material images with ID codes and predict new ID labels. When the same material has the same ID code and the number of the same ID codes is greater than 1, send all ID labels and image data with ID codes to the initial model M0 for incremental learning and update the model parameters. At the same time, when the initial model M0 starts predicting, the gradient information of the character area is obtained in the last layer of the CNN network.
[0062]
[0063] Among them, a is the number of ID codes on each material that needs to be tracked, x is each generated feature map, c is the channel of the image, K is the last layer feature matrix of the initial model M0; g(xc) is the gradient information.
[0064] The weight of the characters in the image area is recorded as input information, and the next time the same ID code is tracked, its weight is also recorded, and this is repeated, and the feature data of this ID code is used as a feature vector representing its erosion status. When performing incremental learning training, this feature vector is added to the character features in each ID code, and the model parameters are updated after the input time series is sorted. The updated model is the M1 model.
[0065] For characters with the same ID code, the gradient vector g is obtained by accumulating the gradient information captured each time n , for the entire ID code, the corresponding two-dimensional gradient matrix g is obtainedm×n .
[0066]
[0067] in, is the two-dimensional gradient matrix g m×n ,
[0068] m is the number of characters in an id code, and n is the number of the same id code;
[0069] is the image feature matrix, It is the gradient value of each channel at each position in the image feature matrix;
[0070] is the gradient matrix, and Figure 2 The matrix y in is the gradient value of each channel at each position in the gradient matrix;
[0071] μ is the coefficient for adjusting the weights of the gradient matrix and the image feature matrix. After predicting a new ID tag, the data will enter the M1 model and be compared with other ID tags on the same material. When the number of identical ID tags identified on the same material is greater than or equal to 2, the incorrectly identified ID tag on the material and the features obtained for each character are input into the CNN network for feature extraction. The real material ID code is used as the target value for training, and the predicted weight of each character is softmax processed to obtain all probability values. At the same time, the probability relationship matrix of all possible characters and real characters is shared with each character. The prediction result of each character can take into account the probability relationship of other characters that may be misdetected, thereby excluding the wrong characters.
[0072]
[0073] Among them, D is the matrix composed of all id tags on the same material, W is the Embedding parameter matrix generated on all id tags using the Glove method, p represents the association between the characters at the same position of all id tags on the same material, and i is the number of characters in each id tag. q is the association between id tags on the same material, j is the number of id tags on each material, Y is the association between the final prediction value and p and q, and d is the dimension of p and q. For each material's id tag, multiple associations are obtained and the final weight Z is obtained through softmax. The final weight Z is then assigned to the character at each position in each id tag.
[0074] Compared with the predicted correct ID label, its weight is corrected by:
[0075] {[g(y)-g(z)]*λ,g(x)*λ}=Y.
[0076] g(y) is the regular gradient actually captured, g(z) is the predicted regular gradient, g(x) is the image gradient, λ represents the eigenvalue; Y is the final predicted value.
[0077] The purpose of the correction is to minimize the difference between the two gradients g(y) and g(z), that is, after continuous additional weight correction, the predicted value is closer to the final target value.
[0078] The loss function Loss is divided into two parts: the error between the predicted ID label and the actual ID code is one part, and the error between g(y) and g(z) is one part each time. 1 Calculate the loss of character order, LOSS 2 The error between the target value and the predicted value is calculated based on the character at each position. 1 The loss ensures that the input sequence is larger than the label sequence.
[0079] Loss=Loss 1 +Loss 2 ;
[0080]
[0081] Among them, p(v|h) is the probability of the output sequence obtained for a given input h at any time; S is the training set, and v is the set of all characters contained in the id code. To make the loss function LOSS 1 To minimize the probability of obtaining the output sequence for each given input h, we need to maximize the probability p(v|h).
[0082]
[0083] Where f is 0 or 1, p(f α ) is in the range of (0,1) after softmax processing; N is the width of the entire input image (each generated feature map), which contains both characters and background. i is the number of characters in each id label.
[0084] Step 5. In the final prediction stage, the possible probability of each character is predicted for each position of the ID tag. The prediction result is in matrix form. After softmax processing, it is spliced into a complete ID tag. Finally, the ID tag is transmitted to the corresponding position of the database for self-increment.
[0085] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A method for tracking materials in complex backgrounds, characterized in that: The following steps are involved: Step 1: Select appropriate pigments and add at least one unique ID code to the surface of the material to be tracked; Step 2: Adjust the light source and the camera until the clarity of the image of the ID code captured by the camera meets the preset requirements; Step 3: Use characters similar to the surface ID code of the material to be tracked and randomly combine these similar characters, and randomly generate a certain distance between the characters to generate first character data; Collect the original image of the surface of the material to be tracked and extract the surface features, and after image processing, fuse it with the first character data to obtain the second character data; The second character data and the original image of the material surface are divided into a certain proportion and then fused with the collected original image of the material surface at a relative position; Take each generated feature map after fusion and its corresponding ID code as input, build a CNN network and extract features, hide some features according to the actual situation, then input it into the LSTM network and sort the characters, output the predicted ID label, and thus generate the initial model M0; Step 4: Deploy the initial model M0 to the environment to collect images of materials with actual ID codes and predict new ID labels; on the same material, when the same ID code is present and the number of the same ID codes is greater than 1, send all ID labels and image data with ID codes to the initial model M0 for incremental learning and update model parameters; at the same time, when the initial model M0 starts predicting, the gradient information of the character area is obtained in the last layer of the CNN network; The weight of the characters in the image area is recorded as input information, and the weight is also recorded when the same ID code is tracked next time, and this process is repeated, and the feature data of the ID code is used as a feature vector representing its erosion status; when performing incremental learning training, this feature vector is added to the character features in each ID code, and the model parameters are updated after the input time series is sorted. The updated model is the M1 model; Step 5. In the final prediction stage, the possible probability of each character is predicted for each position of the ID tag. The prediction result is in matrix form. After softmax processing, it is spliced into a complete ID tag. Finally, the ID tag is transmitted to the corresponding position of the database for self-increment.
2. A complex background material tracking method according to claim 1, characterized in that: In step three, the image processing includes opening and closing operations, adding noise, affine transformation, perspective change, and adding mask operations.
3. A complex background material tracking method according to claim 1, characterized in that: In step three, the fusion with the first character data is performed using a mask.
4. The method for tracking complex background materials according to claim 1, characterized in that: The fusion algorithm formula of the second character data and the original image of the material surface divided into a certain proportion and then fused with the collected original image of the material surface at a relative position is: is a matrix A, which is a 3-channel feature map of the second character data surface; A1 represents the value of the relative position on a single channel, and β represents the number of values on a single channel; is the matrix G, which is the feature vector extracted from the three channels of RGB on the original image surface of the material surface; (s1, t1), (s2, t2), (s3, t3) correspond to the coordinates of the three channels of RGB on the original image surface of the material surface respectively; G (s1,t1) , G (s2,t2) , G (s3,t3) It represents the coordinates (s1, t1), (s2, t2), (s3, t3) multiplied by the eigenvector; is the matrix λ, the matrix λ is the feature map of the training data to be input into the initial model M0, and λ1, λ2, and λ3 represent the eigenvalues.
5. The method for tracking complex background materials according to claim 1, characterized in that: In step 4, for the characters corresponding to the same ID code, the gradient vector g is obtained by accumulating the gradient information captured each time n , for the entire ID code, the corresponding two-dimensional gradient matrix g is obtained m×n ; in, is the two-dimensional gradient matrix g m×n , m is the number of characters in an ID code, and n is the number of the same ID code; is the image feature matrix, It is the gradient value of each channel at each position in the image feature matrix; is the gradient matrix, is the gradient value of each channel at each position in the gradient matrix; μ is the coefficient for adjusting the weight of the gradient matrix and the image feature matrix; After predicting the new ID tag, the data will enter the M1 model and be compared with other ID tags on the same material. When the number of identical ID tags identified on the same material is greater than or equal to 2, the incorrectly identified ID tag on the material and the features obtained for each character are input into the CNN network for feature extraction. The real material ID code is used as the target value for training, and a softmax process is performed on the predicted weight of each character to obtain all probability values. At the same time, the probability relationship matrix of all possible characters and real characters is shared with each character. The prediction result of each character can take into account the probability relationship of other characters that may be misdetected, thereby excluding the wrong characters. Pi=(Di)T(W p)T; qj=D jW q; Where D is the matrix of all id labels on the same material, W is the Embedding parameter matrix generated on all id labels using the Glove method, p represents the association between the characters at the same position of all id labels on the same material, i is the number of characters in each id label, q is the association between id labels on the same material, j is the number of id labels on each material, Y is the association between the final prediction value and p and q, and d is the dimension of p and q; multiple associations are obtained for each material's id label, and the final weight Z is obtained through softmax, and then the final weight Z is assigned to the characters at each position in each id label; Compared with the predicted correct ID label, its weight is corrected by: {[g(y)-g(z)]*λ,g(x)*λ}=Y; g(y) is the regular gradient actually captured, g(z) is the predicted regular gradient, g(x) is the image gradient, λ represents the eigenvalue; Y is the final predicted value; The purpose of the correction is to minimize the difference between the two gradients g(y) and g(z), that is, after continuous additional weight correction, the predicted value is closer to the final target value; The loss function Loss is divided into two parts. One part is the error between the predicted ID label and the real ID code, and the other part is the error between g(y) and g(z) corrected each time. LOSS1 calculates the loss of character sequence, and LOSS2 calculates the error between the target value and the predicted value based on the character at each position. When calculating the LOSS1 loss, ensure that the input sequence is greater than the label sequence. Loss = Loss1 + Loss2; Where p(v|h) is the probability of the output sequence obtained for a given input h at any time; S is the training set, and v is the set of all characters contained in the id code; Where f is 0 or 1, p(f α ) after softmax processing, the value range is (0,1); N is the width of each generated feature map, which contains two parts: characters and background, and i is the number of characters in each id label.