Tumbler locking monitoring and alarm method and system based on visual recognition

Through the flip machine lock monitoring method based on visual recognition, the convolutional neural network model is used to realize the precise monitoring of the flip machine locking state, solving the problem of manual monitoring and inaccurate labor in the existing technology, and improving the monitoring efficiency and accuracy.

CN118628987BActive Publication Date: 2024-12-06AUTOMOTIVE ENGINEERING CORPORATION +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411104569.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-13
Publication Date
2024-12-06
Estimated Expiration
2044-08-13

AI Technical Summary

Technical Problem

In the prior art, the locking monitoring of the flip machine relies on manual on-site monitoring, which consumes manpower and cannot guarantee the timeliness and accuracy of the monitoring. It is impossible to accurately monitor the locking mechanism of the flip machine, which increases labor costs.

Method used

The flip machine lock monitoring and alarm method based on visual recognition is adopted. By obtaining training image data, a data set is formed and the basic convolutional neural network model is trained, and the test data set is used to test the model. If the test results meet the actual results, it will be deployed to the production site. Use real-time production data to detect, mark and display the detection results to achieve accurate monitoring of the locking status of the flip machine.

Benefits of technology

Through the combination of visual camera and convolutional neural network model, high-speed and accurate monitoring of flip locking is achieved, reducing labor costs and improving monitoring accuracy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118628987B_ABST
    Figure CN118628987B_ABST
Patent Text Reader

Abstract

The present invention relates to the field of equipment monitoring and image analysis, and discloses a method and system for monitoring and alarming the locking of a turning machine based on visual recognition, by acquiring training image data, combining the training image data with a basic convolutional neural network model, obtaining an applied convolutional neural network model after training, applying the applied convolutional neural network model to the production site, obtaining detection results in combination with actual conditions, and displaying the detection results. The problem of being unable to accurately monitor the locking mechanism of the turning machine, excessively consuming manpower, and increasing labor costs is solved. The advantage is that the on-site image is obtained by real-time shooting with a visual camera, the on-site image is combined with the training model, and high-speed and accurate on-site image judgment is completed through system processing, so as to realize the monitoring and alarming of the locking of the turning machine.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of equipment monitoring and image analysis, and in particular to a locking monitoring and alarm method and system for a turnover machine based on visual recognition. Background Art

[0002] The flip machine is used to flip automotive equipment such as car shells from one position to another for subsequent processing, inspection or loading; during or after the flipping process, it is necessary to ensure that the automotive equipment is locked and maintained in the predetermined position to prevent it from accidentally moving or sliding.

[0003] In the prior art, on-site monitoring needs to be performed manually in different time periods. This monitoring method is too labor-intensive and cannot guarantee the timeliness and accuracy of the monitoring. Summary of the invention

[0004] To this end, the present invention provides a tilting machine locking monitoring and alarm method and system based on visual recognition, which is used to solve the problems of being unable to accurately monitor the tilting machine locking mechanism, excessive consumption of manpower, and increased labor costs.

[0005] In order to achieve the above-mentioned objectives, the present invention provides the following technical solutions: a method and system for monitoring and alarming the locking of a turning machine based on visual recognition, obtaining training image data, forming a data set according to the training image data, annotating the data set and dividing it into a training data set and a test data set; using the training data set to train a basic convolutional neural network model to obtain an applied convolutional neural network model, using the test data set to test the applied convolutional neural network model to obtain a test result, comparing the test result with the actual result, if the test result does not match the actual result, performing incremental training on the applied convolutional neural network model, if the test result matches the actual result, deploying the applied convolutional neural network model at a production site; obtaining real-time production data at the production site, inputting the real-time production data into the applied convolutional neural network model, obtaining a detection result, annotating the detection result on a data image and displaying the data image.

[0006] As a preferred technical solution of the present invention, image annotation: screening training images according to the locking state of the flip machine, using rectangular frames and manually annotating through labels; image vectorization: reading the training images, converting the training images into three-dimensional arrays; image standardization: adjusting the dimensional order of the three-dimensional array, adjusting the number of channels to the first place; dividing the pixel values ​​of the training images by 255, normalizing the training images; and unifying the resolution size of the training images.

[0007] As a preferred technical solution of the present invention, the locking state of the flip machine is divided into upper locked, upper unlocked warning, upper unlocked, lower locked and lower unlocked, wherein the upper locked, upper unlocked warning and upper unlocked are set as locking hook states, and the lower locked and lower unlocked are set as reverse stop rod states; the training images are screened according to the locking state of the flip machine, and the number of corresponding images for each state is not less than 200.

[0008] As a preferred technical solution of the present invention, the label includes category class, center point x-axis coordinate cx, center point y-axis coordinate cy, rectangular frame height h and rectangular frame width w; the data set is divided into the training data set and the test data set according to a 5:1 ratio.

[0009] As a preferred technical solution of the present invention, the training process of the basic convolutional neural network model is recorded, and the accuracy, recall rate, average precision and loss value are used as evaluation data. After the evaluation data all reach a value above 0.99 and are maintained for 50 iterations, it is determined that the training of the applied convolutional neural network model is completed, and the model output is obtained, and the model output is interpreted.

[0010] As a preferred technical solution of the present invention, the confidence threshold is set to 0.01, and the prediction boxes with confidence less than 0.01 are screened out; the loU threshold is set to 0.5, and the prediction boxes with loU values ​​greater than 0.5 are regarded as corresponding to the same object; non-maximum suppression is used to eliminate overlapping prediction boxes, and the prediction box with the largest confidence is retained as the best prediction box; after obtaining the best prediction box, argmax(softmax(a0,a1,a2,a3,a4)) is used to obtain the category class.

[0011] The calculation formula of the category class is:

[0012]

[0013] Where: Represents the exponential sum of the likelihood of each category of the model output, using the exponential function Ensure that the sum of the exponents is not 0, Indicates the probability that the model predicts the category to be 0, and the rest of the scores are similar. The argmax function returns The index of the element with the maximum value in . The index corresponds to the same category as the element value, and class represents the category with the maximum probability.

[0014] The calculation formula for comparing the test results is: Calculate the accuracy of the test set P = the number of correct predictions by the model / the total number of test sets.

[0015] The locking monitoring and alarm system of a turnover machine based on visual recognition includes: an acquisition unit, which is used to acquire training image data, form a data set according to the training image data, annotate the data set and divide it into a training data set and a test data set; a training unit, which uses the training data set to train a basic convolutional neural network model to obtain an applied convolutional neural network model, and uses the test data set to test the applied convolutional neural network model to obtain a test result; and an application display unit, which acquires real-time production data at a production site, inputs the real-time production data into the applied convolutional neural network model, obtains a detection result, annotates the detection result on a data image and displays the data image.

[0016] As a preferred technical solution of the present invention, a computer-readable storage medium includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the flip machine locking monitoring and alarm method based on visual recognition as described in any one of items 1 to 7 of the above contents.

[0017] As a preferred technical solution of the present invention, an electronic device includes: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include a method for executing the flip machine locking monitoring and alarm method based on visual recognition as described in any one of items 1 to 7 of the above contents.

[0018] Compared with the prior art, the present invention provides a system with the following beneficial effects: obtaining on-site images through real-time shooting with a visual camera, combining the on-site images with training models and completing high-speed and accurate on-site image judgment through system processing, thereby realizing monitoring and alarming of the locking of the turning machine, greatly improving monitoring accuracy and reducing labor costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a system flow chart of the present invention. DETAILED DESCRIPTION

[0020] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0022] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0023] A method for monitoring locking alarm of a turning machine based on visual recognition, the method comprising the following steps:

[0024] S100, obtaining training image data, forming a data set according to the training image data, annotating the data set and dividing it into a training data set and a test data set;

[0025] S200, using the training data set to train the basic convolutional neural network model to obtain the applied convolutional neural network model, using the test data set to test the applied convolutional neural network model to obtain the test result, comparing the test result with the actual result, if the test result does not match the actual result, then incrementally training the applied convolutional neural network model, if the test result matches the actual result, then deploying the applied convolutional neural network model at the production site;

[0026] S300, acquiring real-time production data at the production site, inputting the real-time production data into a convolutional neural network model, obtaining detection results, marking the detection results on a data image, and displaying the data image.

[0027] Specifically, step S100 further includes:

[0028] S110, image annotation: the locking state of the flip machine is divided into five states: upper locked, upper unlocked warning, upper unlocked, lower locked and lower unlocked. Among them, the upper locked, upper unlocked warning and upper unlocked states are divided into locking hook states, and the lower locked and lower unlocked states are divided into reverse stop rod states.

[0029] The data set is screened in the training image data according to the above status, and the number of each training image is no less than two hundred; the data set is manually annotated with a rectangular box, and each training image is labeled, where the label includes category class, center point x-axis coordinate cx, center point y-axis coordinate cy, rectangular box height h, rectangular box width w, and the data set is divided into a training data set and a test data set in a ratio of 5:1.

[0030] S120, image vectorization: reading training image data, and converting the training image data into a three-dimensional array, wherein the RGB value information of each pixel in the three-dimensional array represents information of a feature in the three-dimensional array vector.

[0031] S130, image standardization: adjust the dimensional order of the three-dimensional array and adjust the number of channels to the first place; divide the pixel values ​​of the training images by 255 to normalize the training images; unify the resolution size of the training images to achieve image standardization.

[0032] Specifically, step S200 further includes:

[0033] S210, model incremental training: train the basic convolutional neural network model on the training data set and record the training process.

[0034] Further:

[0035] S211, model incremental training is the second and subsequent training. In addition to possible adjustments to the training data set, it is also necessary to add training image data to the training data set and annotate it to address the shortcomings of the previous training.

[0036] S212, model training process: accuracy (precision), recall rate (recall), average precision (mAP) and loss value are used as records of the model training process; generally, the basic convolutional neural network model training is confirmed to be completed after the accuracy, recall and average precision standards reach a value of 0.99 or above and maintain 50 iterations.

[0037] S220, model testing: Train the basic convolutional neural network model through the test training set, obtain the model output, and explain the model output.

[0038] Step S220 further includes:

[0039] S221, Model output: The output format of the basic convolutional neural network is [batch_size, anchors, 10], batch_size is the batch number, indicating how many images the model calculates at one time; anchors is the number of anchor boxes, indicating the number of prediction boxes found in this calculation; the last parameter has a complex meaning, (confidence obj, center point x-axis coordinate cx, center point y-axis coordinate cy, rectangular box height h, rectangular box width w, upper lock a0, upper unlock warning a1, upper unlock a2, lower lock a3, lower unlock a4). The meanings of the first five parameters are fixed: confidence obj indicates the credibility of the target object in this prediction box; cx, cy, h, w indicate the position and size of the rectangular box. The other parameters constitute the one-hot code of the target detection category.

[0040] S222. Explain the model output: set the confidence threshold to 0.01, and filter out prediction boxes with confidence less than 0.01; set the IoU value to 0.5, and the prediction boxes with IoU values ​​greater than 0.5 are considered to correspond to the same object; use non-maximum suppression (nms) to eliminate overlapping prediction boxes, leaving the prediction box with the highest confidence as the best prediction box; after obtaining the best prediction box, use argmax(softmax(a0,a1,a2,a3,a4)) to get the category class.

[0041] The application formula of category class is as follows:

[0042]

[0043] Where: Represents the exponential sum of the likelihood of each category of the model output, using the exponential function Ensure that the sum of the exponents is not 0, Indicates the probability that the model predicts the category to be 0, and the rest of the scores are similar. The argmax function returns The index of the element with the maximum value in . The index is the same as the category corresponding to the element value, and class represents the category with the maximum probability.

[0044] S230. Compare the model test results with the actual results: calculate the accuracy of the test set P = the number of correct model predictions / the total number of test sets.

[0045] S240, P> : Set the accuracy threshold = 0.99, if P does not exceed the threshold , then return to step S210 to train the model incrementally; if P exceeds , then go to the next step S250.

[0046] S250, model deployment: deploy the convolutional neural network model to the flip machine locking detection system to confirm that the system operates normally.

[0047] Step S300 further includes:

[0048] S310, obtaining image data of the locking device of the turning machine and the turning machine number during the production process;

[0049] S320: After the training image data obtained in the production process is processed in step S100, it is input into the deployed model, and the model output is interpreted according to steps S221 and S222, and then it is determined whether the locking device of the turning machine is locked according to Table 1:

[0050] Table 1 Correspondence table between model prediction results and system output results

[0051]

[0052] S330, marking the detection result, the flip machine number, and the determination result on the image data and displaying them.

[0053] S340: If the on-site personnel find that the model detection standard does not fit the actual situation well, they can make modification suggestions to the R&D personnel. After discussion, return to step S210 to perform model incremental training to complete the method cycle of the locking monitoring of the turning machine.

[0054] This embodiment also includes a tilting machine locking monitoring system based on visual recognition, and a device that cooperates with the system.

[0055] A visual camera is fixedly arranged on the pry body, and the visual camera is used to take pictures of the locking hook, and includes a background processing module with the above method built in, and the visual camera can upload the pictures to the background processing module.

[0056] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A locking monitoring and alarm method for a turning machine based on visual recognition, characterized in that: The following steps are involved: Acquire training image data, form a data set according to the training image data, annotate the data set and divide it into a training data set and a test data set, wherein annotating the data set includes: Image annotation: Filter training images according to the locking status of the flip machine, use rectangular boxes and manually annotate them with labels; Image vectorization: reading the training image and converting the training image into a three-dimensional array; Image standardization: adjusting the dimensional order of the three-dimensional array and adjusting the number of channels to the first place; dividing the pixel values ​​of the training images by 255 to normalize the training images; and unifying the resolution of the training images; The locking states of the flip machine are divided into upper locking, upper unlocking warning, upper unlocking, lower locking and lower unlocking, wherein the upper locking, upper unlocking warning and upper unlocking are set as locking hook states, and the lower locking and lower unlocking are set as reverse stopper lever states; the training images are screened according to the locking states of the flip machine, and the number of corresponding images for each state is not less than 200; Using the training data set to train a basic convolutional neural network model to obtain an applied convolutional neural network model, using the test data set to test the applied convolutional neural network model to obtain a test result, comparing the test result with the actual result, if the test result does not match the actual result, performing incremental training on the applied convolutional neural network model, if the test result matches the actual result, deploying the applied convolutional neural network model on the production site; Acquire real-time production data at the production site, the real-time production data including image data of the locking device of the flip machine and the flip machine number; annotate the image data of the locking device of the flip machine and input it into the applied convolutional neural network model, and interpret the model output of the applied convolutional neural network model, wherein the output format is [batch_size, anchors, 10], batch_size is the batch number, indicating how many images the model calculates at one time; anchors is the number of anchor boxes, indicating the number of prediction boxes found in this calculation; the last parameter includes the confidence obj, the x-axis coordinate cx of the center point, the y-axis coordinate cy of the center point, the height h of the rectangular box, the width w of the rectangular box, the upper locking a0, the upper unlocking warning a1, the upper unlocking a2, the lower locking a3, and the lower unlocking a4; the confidence obj indicates the credibility of the existence of the target object in this prediction box; cx, cy, h, and w indicate the position and size of the rectangular box; the other parameters constitute the one-hot code of the target detection category; Set the confidence threshold to 0.01 to filter out prediction boxes with confidence less than 0.01; set the IoU value to 0.5, and the prediction boxes with IoU values ​​greater than 0.5 are considered to correspond to the same object; use non-maximum suppression (nms) to eliminate overlapping prediction boxes, leaving the prediction box with the highest confidence as the best prediction box; after obtaining the best prediction box, use argmax(softmax(a0,a1,a2,a3,a4)) to get the category class and obtain the detection result. It is determined whether the locking device of the turning machine is locked to obtain a determination result, and the detection result, the turning machine number and the determination result are marked on the data image and the data image is displayed.

2. The method for monitoring and alarming the locking of a turning machine based on visual recognition according to claim 1 is characterized in that: The label includes category class, center point x-axis coordinate cx, center point y-axis coordinate cy, rectangular frame height h and rectangular frame width w; the data set is divided into the training data set and the test data set according to a 5:1 ratio.

3. The method for monitoring and alarming locking of a turning machine based on visual recognition according to claim 1 is characterized in that: The using the training data set to train a basic convolutional neural network model to obtain an applied convolutional neural network model comprises: The training process of the basic convolutional neural network model is recorded, and the accuracy, recall rate, average precision and loss value are used as evaluation data. After the evaluation data all reach a value above 0.99 and are maintained for 50 iterations, it is determined that the training of the applied convolutional neural network model is completed, and the model output is obtained, and the model output is interpreted.

4. The method for monitoring and alarming the locking of a turning machine based on visual recognition according to claim 3 is characterized in that: The calculation formula of the category class is: Where: Represents the exponential sum of the likelihood of each category of the model output, using the exponential function Ensure that the sum of the exponents is not 0, Indicates the probability that the model predicts the category to be 0, and the rest of the scores are similar. The argmax function returns The index of the element with the maximum value in . The index corresponds to the same category as the element value, and class represents the category with the maximum probability. The calculation formula for comparing the test results is: Calculate the accuracy of the test set P = the number of correct predictions by the model / the total number of test sets.

5. The locking monitoring and alarm system of the turning machine based on visual recognition is characterized by: include: An acquisition unit is used to acquire training image data, form a data set according to the training image data, annotate the data set and divide it into a training data set and a test data set, wherein annotating the data set includes: Image annotation: Filter training images according to the locking status of the flip machine, use rectangular boxes and manually annotate them with labels; Image vectorization: reading the training image and converting the training image into a three-dimensional array; Image standardization: adjusting the dimensional order of the three-dimensional array and adjusting the number of channels to the first place; dividing the pixel values ​​of the training images by 255 to normalize the training images; and unifying the resolution of the training images; The locking states of the flip machine are divided into upper locking, upper unlocking warning, upper unlocking, lower locking and lower unlocking, wherein the upper locking, upper unlocking warning and upper unlocking are set as locking hook states, and the lower locking and lower unlocking are set as reverse stopper lever states; the training images are screened according to the locking states of the flip machine, and the number of corresponding images for each state is not less than 200; A training unit, using the training data set to train a basic convolutional neural network model to obtain an applied convolutional neural network model, and using the test data set to test the applied convolutional neural network model to obtain a test result; The application display unit obtains real-time production data at the production site, and the real-time production data includes the image data of the locking device of the flip machine and the flip machine number; the image data of the locking device of the flip machine is annotated, and input into the application convolutional neural network model, and the model output of the application convolutional neural network model is interpreted, wherein the output format is [batch_size, anchors, 10], batch_size is the batch number, indicating how many images the model calculates at one time; anchors is the number of anchor boxes, indicating the number of prediction boxes found in this calculation; the last parameter includes the confidence obj, the x-axis coordinate cx of the center point, the y-axis coordinate cy of the center point, the height h of the rectangular box, the width w of the rectangular box, the upper locking a0, the upper unlocking warning a1, the upper unlocking a2, the lower locking a3, and the lower unlocking a4; the confidence obj indicates the credibility of the existence of the target object in this prediction box; cx, cy, h, and w indicate the position and size of the rectangular box; the other parameters constitute the one-hot code of the target detection category; Set the confidence threshold to 0.01 to filter out prediction boxes with confidence less than 0.01; set the IoU value to 0.5, and the prediction boxes with IoU values ​​greater than 0.5 are considered to correspond to the same object; use non-maximum suppression (nms) to eliminate overlapping prediction boxes, leaving the prediction box with the highest confidence as the best prediction box; after obtaining the best prediction box, use argmax(softmax(a0,a1,a2,a3,a4)) to get the category class and obtain the detection result. It is determined whether the locking device of the turning machine is locked to obtain a determination result, and the detection result, the turning machine number and the determination result are marked on the data image and the data image is displayed.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the flip machine locking monitoring and alarm method based on visual recognition as described in any one of claims 1 to 4.

7. An electronic device, characterized in that: include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include a method for executing the flip machine locking monitoring and alarm method based on visual recognition as described in any one of claims 1 to 4.