Method for detecting missed gluing behavior of screw glue based on deep learning algorithm

Abstract

The invention relates to a deep learning algorithm-based method for detecting a missed gluing behavior of screw glue. The method comprises the following steps of: 1, preprocessing a real-time video of a screw glue gluing station; step 2, performing algorithm identification based on a deep learning algorithm on the preprocessed real-time video in a single-frame and single-frame manner so as to identify categories and positions of three objects including a screw glue box, a hand and a screw glue stick in a screw glue coating station; step 3, performing joint judgment on multi-frame recognition results of algorithm recognition so as to switch among a non-gluing state, a to-be-gluing state and a gluing state of a screw glue gluing station, and performing judgment processing on conditions of each state; and step 4, performing corresponding processing according to whether the screw glue coating omission behavior exists or not output in the step 3: if the screw glue coating omission behavior exists in the screw glue coating station, indicating an alarm device to give an alarm, and recording the time of the current frame of the real-time video.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and specifically to the field of deep learning of artificial intelligence technology, especially a technology that can be applied in a smart factory, using computing equipment and a camera to detect glue leakage on screws in a factory (for example, at workstations, operating tables, etc.). Background Art

[0002] As an important branch of artificial intelligence technology, deep learning technology has made significant progress in recent years and has been widely used in many fields. Deep learning has penetrated into many fields such as speech processing, computer vision, and natural language processing. For example, in some aspects of speech, image, and video recognition, systems based on deep learning have reached or exceeded the human level. The popularity and popularization of deep learning frameworks have made large-scale data processing and model training more efficient. As one of the foundations of deep learning methods, new neural network structures such as Transformer and GPT have achieved remarkable success in language modeling and generation tasks, showing the importance of model structure optimization. With the increase in the cost of labeled data, semi-supervised learning and unsupervised learning using unlabeled data have become a research hotspot, effectively reducing the cost of labeling and improving the generalization ability of the model. Through multi-layer processing based on multi-layer perceptrons with multiple hidden layers, the initial "low-level" feature representation is gradually transformed into a "high-level" feature representation, and complex learning tasks such as classification can be completed with a "simple model". As a type of machine learning, deep learning is the only way to achieve artificial intelligence.

[0003] Screw glue, a glue that perfectly bonds metals, is widely used in various industries and fields for its excellent strength, toughness and durability. In the past, metal bonding was a headache. Traditional adhesives often cannot withstand the hardness of metals, have poor durability, and may even damage metals. The emergence of screw glue solves this problem. It is not only harmless to metals, but its strong bonding and durability can ensure the long-term stability of metal products. The characteristics of screw glue are mainly three points. First, its strength is extremely high and can reach its peak in a short time, so that metal parts are firmly bonded. Secondly, it has excellent toughness and will not become brittle or lose elasticity even in extremely cold and hot environments. Finally, it has excellent durability and will not age or fall off after long-term use. However, at present, screw glue coating is basically manual operation. This manual operation will more or less result in the occurrence of screw glue leakage, which will cause great equipment failure and loss, and even lead to potential safety risks in some applications. Therefore, avoiding the leakage of screw glue is an urgent technical problem that has always existed in the industry and must be solved.

[0004] The industry is always in need of improved screw glue leakage monitoring technology, especially based on artificial intelligence deep learning algorithm identification, to overcome the above and other technical problems.

[0005] The information included in this background section of the present specification, including any references cited herein and any description or discussion thereof, is included for technical reference purposes only and is not to be construed as subject matter that will limit the scope of the present invention. Summary of the invention

[0006] The present invention is proposed in view of the above and other more concepts.

[0007] One of the main objectives of the present invention is to solve the problem of screw glue leakage that may occur when operators perform screw glue coating operations in actual factory scenarios, and to propose a detection technology for screw glue leakage in the factory.

[0008] In view of the above, the present invention aims to provide a technology for monitoring abnormal behavior in the process of screw gluing by using a camera in combination with a deep learning algorithm. When the screws are not glued, timely judgment, warning and correction can be made to prevent the occurrence of screw glue leakage, thereby overcoming the technical difficulties and defects that have always existed in the industry.

[0009] According to one aspect of the present invention, a method for detecting the behavior of screw glue leakage based on a deep learning algorithm is provided, comprising the following steps: step 1, preprocessing the real-time video of the screw glue application station; step 2, performing algorithm recognition based on a deep learning algorithm frame by frame on the preprocessed real-time video to identify the categories and positions of three objects in the screw glue application station: a screw glue box, a hand, and a screw glue stick; step 3, jointly judging the multi-frame recognition results of the algorithm recognition of the real-time video, so as to detect the situation where the screw glue application station is not in the state of leakage. The respective situations of the glue state, the state to be glued and the state being glued are judged and processed, including switching between the no-glue state, the state to be glued and the state being glued at the screw glue coating station, wherein the initial state of the screw glue coating station is set to the no-glue state; step 4, according to whether there is a screw glue leakage behavior at the screw glue coating station output in step 3, corresponding processing is performed, including: if there is a screw glue leakage behavior at the screw glue coating station, the alarm device of the screw glue coating station is instructed to alarm, and the time of the current frame of the real-time video is recorded.

[0010] According to one embodiment, step 1 includes the following sub-steps: step 1-1: manually marking the screw glue box area and the screw area in the real-time video frame; step 1-2: using a camera to collect data of the real-time video of the screw glue coating station area; step 1-3: cutting frames of the collected real-time video data; step 1-4: using a frame difference method to calculate the frame difference between the current frame and the previous frame of the real-time video. If the threshold of the frame difference is lower than the set value, the current frame is not preprocessed; and step 1-5: if the threshold of the frame difference is higher than the set value, the current frame is preprocessed.

[0011] According to one embodiment, step 2 includes the following sub-steps:

[0012] Step 2-1: using a deep learning neural network model, extracting features from the image / video data of the current frame that has been preprocessed in step 1;

[0013] Step 2-2: Put the features extracted in step 2-1 into a feature enhancement module that combines an improved attention mechanism and an enhanced convolutional attention module of a deep learning neural network model to perform feature enhancement; and

[0014] Step 2-3: Send the feature data enhanced by the feature in step 2-2 to a multi-layer perceptron to predict the target position and target category, so as to identify the categories and positions of the three objects, namely, the screw glue box, the hand, and the screw glue stick, in the screw glue applying station.

[0015] According to one embodiment, the improved attention mechanism is expressed by the following formula 1:

[0016]

[0017] Wherein, H is the height of the input tensor of the deep learning neural network model, z is the improved attention value, W is the width of the input tensor of the deep learning neural network model, x is the self-attention function, i is the index of the input tensor width of the deep learning neural network model, and c is the channel index;

[0018] And, the enhanced convolutional attention module is expressed by the following formula 2:

[0019]

[0020] Among them, M is the enhanced convolution attention value, F is the input tensor of the deep learning neural network model, σ is the Sigmoid activation function, W1 is the value of the second dimension of the input tensor of the deep learning neural network model, W0 is the value of the first dimension of the input tensor of the deep learning neural network model, MaxPool is the maximum pooling function, AvgPool is the average pooling function, MLP is the multi-layer perceptron function, max is the maximum value function, avg is the average value function, and c is the channel index.

[0021] According to one embodiment, step 3 includes the following sub-steps:

[0022] Step 3-1: When the screw glue gluing station is in a non-gluing state, determine whether the screw glue box is in the screw glue box area through the detection result of the current frame in step 2; when it is determined that the screw glue box is in the screw glue box area, record the time the screw glue box is in the screw glue box area, and set the time the screw glue box is not in the screw glue box area to 0; when it is determined that the screw glue box is not in the screw glue box area, record the time the screw glue box is not in the screw glue box area, and set the time the screw glue box is in the screw glue box area to 0; if it is determined that the time the screw glue box is in the screw glue box area exceeds a threshold, set the state of the screw glue gluing station to a waiting state;

[0023] Step 3-2: When the screw glue coating station is in the state of waiting for gluing, first, determine whether the screw glue box is in the screw glue box area. When it is determined that the screw glue box is not in the screw glue box area, record the time period that the screw glue box is not in the screw glue box area; when it is determined that the screw glue box is in the screw glue box area, set the time period that the screw glue box is not in the screw glue box area to 0; then, determine whether the hand touches the screw box area according to the detection result of step 2. If it is determined that the hand touches the screw box area, set the state of the screw glue coating station to the gluing state, record the start time of the gluing state, and set whether the hand is in the screw holding state to yes; then, determine the time period that the screw glue box is not in the screw glue box area. If the time period exceeds the threshold and the screw glue coating station is not in the gluing state, set the state of the screw glue coating station to the non-gluing state; and

[0024] Step 3-3: When the screw glue coating station is in the gluing state, judgment and processing are performed based on at least one of the following judgment module items: a logic judgment module for touching the screw glue area, a logic judgment module for touching the screw area, a screw glue missing coating time judgment module, and a gluing state end judgment module.

[0025] According to one embodiment, step 3-3 includes the following subdivision steps:

[0026] Step 3-3-1: When the screw glue coating station is in the gluing state, first determine whether the hand touches the screw glue box area according to the detection result of step 2. When it is determined that the hand has touched the screw glue box area, proceed to the next step to determine whether the hand is in the state of touching the screw glue box area to pick up the glue stick or the state of touching the screw glue box area to put the glue stick back. When it is determined that the hand is neither in the state of touching the screw glue box area to pick up the glue stick nor in the state of touching the screw glue box area to put the glue stick back, determine whether the end time of the state of the hand touching the screw glue box area to pick up the glue stick is empty. If it is empty, it is considered that the hand is currently in the state of touching the screw glue box to pick up the glue stick, and the start time of the state of the hand touching the screw glue box to pick up the glue stick is recorded. If it is not empty, it is considered that the hand is currently in the state of touching the screw glue box area to put the glue stick back, and the start time of the state of the hand touching the screw glue box area to put the glue stick back is recorded.

[0027] When it is determined that the hand has not touched the screw glue box area, it is determined whether the hand is in the state of touching the screw glue box area to pick up the glue stick. If the hand is in the state of touching the screw glue box area to pick up the glue stick, the cumulative duration of the hand touching the screw glue box area to pick up the glue stick before the end of the state is recorded. When the cumulative duration exceeds the threshold, it is determined whether the glue stick is in the screw glue box area according to the detection result of the step 2. If the glue stick is determined to be in the screw glue box area, it is considered that the state of the hand touching the screw glue box area to pick up the glue stick is invalid; if the glue stick is determined to be in the screw glue box area, it is considered that the state of the hand touching the screw glue box area to pick up the glue stick is valid, and the end time of the state of the hand touching the screw glue box area to pick up the glue stick is recorded, otherwise, it is determined that the state of the hand touching the screw glue box area to pick up the glue stick is invalid, and the start time of the state of the hand touching the screw glue box area to pick up the glue stick is set to null; and

[0028] Then determine whether the hand is in the state of touching the screw glue box area to put the glue stick back. If the hand is in the state of touching the screw glue box area to put the glue stick back, then record the cumulative time before the end of the state of touching the screw glue box area to put the glue stick back. When the cumulative time exceeds the threshold, determine whether the glue stick is in the screw glue box area according to the detection result of step 2; if it is determined that the glue stick is not in the screw glue box area, it is considered that the state of touching the screw glue box area to put the glue stick back is invalid, and the start time of the state of touching the screw glue box area to put the glue stick back is set to null; if it is determined that the glue stick is in the screw glue box area, it is considered that the state of touching the screw glue box area to put the glue stick back is valid, and the end time of the state of touching the screw glue box area to put the glue stick back is recorded;

[0029] Step 3-3-2: judging whether the hand touches the screw area according to the detection result of step 2, and judging whether the hand is in the state of holding the screw when it is judged that the hand touches the screw area, and if not, setting the state of holding the screw to yes and setting the end time of the state of holding the screw to empty;

[0030] When it is determined that the hand does not touch the screw area, it is determined whether the hand is in the state of holding the screw. If so, the cumulative time before the end of the state of holding the screw is recorded. If the cumulative time exceeds the threshold, it is determined that the state of holding the screw is ended, and the end time of the state of picking up the screw by touching the screw area is recorded, and the cumulative time before the end of the state of holding the screw is set to 0;

[0031] Step 3-3-3: First, determine whether the end time of the hand holding the screw state is empty. If the end time of the hand holding the screw state is not empty, then determine whether the start time of the hand touching the screw glue box area to pick up the glue stick state is empty. If the start time of the hand touching the screw glue box area to pick up the glue stick state is not empty, then calculate the difference between the start time of the hand touching the screw glue box to pick up the glue stick state and the end time of the hand holding the screw state; if the difference exceeds the threshold, it is determined that there is currently a behavior of leaking glue on the screw, and an alarm is triggered; if the start time of the hand touching the screw glue box area to pick up the glue stick state is empty, then calculate the time difference between the time of the current frame of the real-time video and the end time of the hand touching the screw area to pick up the screw state. If the time difference exceeds the threshold, it is determined that there is currently a behavior of leaking glue on the screw; and

[0032] Step 3-3-4: First, determine whether the difference between the time of the current frame of the real-time video and the start time of the gluing state exceeds a threshold. If it exceeds the threshold, set the current state of the screw glue gluing station to the state to be glued, and initialize the relevant process variables; then, determine whether the end time of the hand touching the screw glue box area and putting back the glue stick state is empty. If it is empty, it is considered that the screw gluing process is completed, and at the same time, set the current state of the screw glue gluing station to the state to be glued, and initialize the relevant process variables.

[0033] According to one embodiment, after step 4 is completed, re-entering the next step cycle of the method includes: re-executing step 1, step 2, step 3 and step 4.

[0034] According to one embodiment, the deep learning algorithm in the method relies on a high performance computing server for execution.

[0035] According to one embodiment, the high-performance computing server includes an edge computing server, which is configured to simultaneously detect screw glue leakage behavior at multiple screw glue application stations.

[0036] According to one embodiment, the real-time video data is obtained through the RTSP video stream protocol of the camera configured at each screw glue application station.

[0037] By combining the method of the present invention with artificial intelligence algorithm judgment and recognition technology, it is possible to accurately and in real time determine whether a certain workstation has experienced screw glue leakage and issue a warning, thereby timely and effectively avoiding the occurrence of screw glue leakage, greatly reducing the failures, losses and safety hazards caused by screw glue leakage.

[0038] More embodiments of the present invention can also achieve other advantageous technical effects that are not listed one by one. These other technical effects may be partially described below and are predictable and understandable to those skilled in the art after reading the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The above features and advantages of these embodiments and other features and advantages and the manner in which they are achieved will become more apparent and embodiments of the present invention may be better understood by referring to the following description taken in conjunction with the accompanying drawings, in which:

[0040] Figure 1 1 is a schematic flow chart of the steps of a method for detecting and judging glue leakage of screw glue according to a first embodiment of the present invention.

[0041] Figure 2 yes Figure 1 Schematic diagram of the manually marked screw glue box area and screw area of ​​the method shown.

[0042] Figure 3 yes Figure 1 A schematic flow chart of the joint judgment of multiple frame recognition results combined with the relevant judgment module of the method shown.

[0043] Figure 4 yes Figure 1 A schematic flow chart of the judgment and processing of the method shown when the state is a non-glue state.

[0044] Figure 5 yes Figure 1 A schematic flow chart of the judgment and processing of the method shown when the state is a state to be glued.

[0045] Figure 6 yes Figure 1 A schematic diagram of the configuration of the glue-applying state logic judgment module involved in the method shown.

[0046] Figure 7 yes Figure 1 A schematic flow chart of the judgment and processing of the method shown when the state is the gluing state.

[0047] Figure 8 yes Figure 1 A schematic flow chart of the judgment and processing of the touch screw area logic judgment module involved in the method shown.

[0048] Fig. 9 yes Figure 1 A schematic flow chart of the judgment and processing of the screw glue leakage duration judgment module involved in the method shown.

[0049] Fig.10 yes Figure 1 A schematic flow chart of the judgment and processing of the gluing state end judgment module involved in the method shown.

[0050] Fig.11 Is configured for running, deploying Figure 1 A schematic diagram of an example configuration of the main relevant equipment and devices of the illustrated method. DETAILED DESCRIPTION

[0051] In the following description of the drawings and specific embodiments, the details of one or more embodiments of the present invention will be described. Other features, objects and advantages of the present invention can be clearly seen from these descriptions, drawings and claims.

[0052] It should be understood that the illustrated and described embodiments are not limited in application to the details of the construction and arrangement of the components set forth in the following description or illustrated in the accompanying drawings. The illustrated embodiments may be other embodiments and may be implemented or executed in various ways. Each example is provided by explaining the disclosed embodiments rather than limiting them. In fact, it will be apparent to those skilled in the art that various modifications and variations may be made to the embodiments of the present invention without departing from the scope or essence of the present invention. For example, a feature illustrated or described as part of one embodiment may be used together with another embodiment to still produce another embodiment. Therefore, the present invention discloses such modifications and variations that fall within the scope of the appended claims and their equivalent elements.

[0053] Likewise, it is understood that the phrases and terms used herein are for descriptive purposes and should not be considered restrictive. The use of "include," "comprise," or "have" and variations thereof herein is intended to be open-ended to include the items listed thereafter and their equivalents and additional items.

[0054] The present invention will be described in more detail below with reference to several specific embodiments of the present invention.

[0055] According to one or more embodiments of the present invention, the present invention is mainly intended to solve the problem of screw glue leakage that may occur when an operator is performing screw glue coating operations in actual application scenarios such as in factories, and proposes a method for detecting screw glue leakage behavior.

[0056] The embodiment of the present invention, which can be applied to the detection method of glue leakage of screws in factories and workshops, can be the following technical solution.

[0057] An example process of the method for detecting the leakage of screw glue of the present invention is as follows: Figure 1 As shown, Figure 1 It is a schematic flowchart of the steps of a method for detecting and judging the glue leakage behavior of screw glue based on artificial intelligence deep learning according to the first embodiment of the present invention.

[0058] An example of the technical solution of this embodiment may include the following method implementation steps: step 1, step 2, step 3 and step 4.

[0059] Step 1

[0060] Step 1 may include preliminary configuration and preprocessing of real-time video of the screw glue application station. The preliminary configuration of the equipment may include the basic configuration required for implementing the technical solution embodiment of the present invention, including a high-performance computing server, such as an edge computing server, a camera and an alarm device configured at the screw glue application station, and other supporting facilities, such as a power supply, optional network, communication equipment and accessories, etc.

[0061] The step 1 may further include the following several sub-steps 1-1, 1-2, 1-3, 1-4, and 1-5.

[0062] Step 1-1: Manually mark the screw glue box area and screw area in the real-time video frame, as follows: Figure 2 As shown, Figure 2 yes Figure 1 Schematic diagram of the manually marked screw glue box area and screw area of ​​the method shown.

[0063] Step 1-2: Use the camera to collect real-time video data of the workstation area.

[0064] Step 1-3: Cut frames of the collected video data.

[0065] Step 1-4: Calculate the frame difference between the current frame and the previous frame using the frame difference method. If the frame difference threshold is lower than the set value, no subsequent preprocessing is performed.

[0066] Step 1-5: Perform preprocessing operations on the frames that are determined in step 1-4 to require subsequent preprocessing.

[0067] Step 2

[0068] Step 2 is algorithm recognition based on a deep learning algorithm, such as a deep learning neural network model, for a single frame of real-time video. For example, algorithm recognition based on a deep learning algorithm is performed frame by frame on the preprocessed real-time video to identify the categories of three objects in the screw glue box, hand, and screw glue stick in the screw glue application station and their respective positions. This step 2 can also include the following several sub-steps 2-1, 2-2, and 2-3.

[0069] Step 2-1: Use the deep learning neural network model to extract features from the image / video data preprocessed in step 1.

[0070] Step 2-2: Use the features extracted in step 2-1 and put them into the feature enhancement module that combines the improved attention mechanism and enhanced convolutional attention module of the deep learning neural network model for feature enhancement. The improved attention mechanism can be expressed by the following formula 1, which can better retain the information of each channel and reduce computational overhead. This module reshapes some channels into batch dimensions and groups the channel dimensions into multiple sub-features so that the spatial semantic features are well distributed within each feature group. It also recalibrates the channel weights in each parallel branch by encoding global information, and further aggregates the output features of the two parallel branches to capture pixel-level pairwise relationships. Formula 1 is expressed as follows:

[0071]

[0072] Among them, H is the height of the input tensor of the deep learning neural network model, z is the improved attention value, W is the width of the input tensor of the deep learning neural network model, x is the self-attention function, i is the index of the input tensor width of the deep learning neural network model, and c is the channel index.

[0073] The enhanced convolutional attention module is shown in Formula 2. This module can simultaneously focus on the channel and spatial dimensions of the convolutional neural network, adaptively refine the input feature map, and significantly improve the classification and detection performance of the model. Formula 2 is expressed as follows:

[0074]

[0075] Among them, M is the enhanced convolution attention value, F is the input tensor of the deep learning neural network model, σ is the Sigmoid activation function, W1 is the value of the second dimension of the input tensor of the deep learning neural network model, W0 is the value of the first dimension of the input tensor of the deep learning neural network model, MaxPool is the maximum pooling function, AvgPool is the average pooling function, MLP is the multi-layer perceptron function, max is the maximum value function, avg is the average value function, and c is the channel index.

[0076] Step 2-3: The feature data after feature enhancement in step 2-2 is sent to the multi-layer perceptron to predict the target position and target category, including the following three target categories (hand, screw glue box and glue stick).

[0077] Step 3

[0078] Step 3 is a joint judgment of multiple frame recognition results. For example, a joint judgment is made on multiple frame recognition results of the algorithm recognition of the real-time video, so as to judge and process the respective situations of the screw glue gluing station being in the un-gluing state, the state to be glued and the state of being glued, including switching between the un-gluing state, the state to be glued and the state of being glued at the screw glue gluing station, wherein the initial state of the screw glue gluing station is set to the un-gluing state. This step 3 may also include the following several sub-steps 3-1, 3-2 and 3-3.

[0079] Step 3-1 is the judgment and processing step of the logic judgment module of the non-glue state, step 3-2 is the judgment and processing step of the logic judgment module of the glue-to-be-glue state, and step 3-3 is the judgment and processing step of the logic judgment module of the glue-being-glue state. Figure 3 As shown, the Figure 3 yes Figure 1 A schematic flow chart of the joint judgment of multiple frame recognition results combined with the relevant judgment module of the method shown.

[0080] Step 3-1: When the state is the non-gluing state, determine whether the screw glue box is in the screw glue box area through the detection result of the current frame in step 2. When it is detected that the screw glue box is in the screw glue box area, record the time the screw glue box is in the screw glue box area and set the time the screw glue box is not in the screw glue box area to 0. When it is detected that the screw glue box is not in the screw glue box area, record the time the screw glue box is not in the screw glue box area and set the time the screw glue box is in the screw glue box area to 0. Then judge the time the screw glue box is in the screw glue box area. When the time the screw glue box is in the screw glue box area exceeds the threshold, set the state to the state to be glued. The process is as follows: Figure 4 As shown, the Figure 4 yes Figure 1 A schematic flow chart of the judgment and processing of the method shown when the state is a non-glue state.

[0081] Step 3-2: Judgment and processing steps when the state is the state to be glued. First, judge whether the screw glue box is in the screw glue box area. When it is detected that the screw glue box is not in the screw glue box area, record the time the screw glue box is not in the screw glue box area. Conversely, when it is detected that the screw glue box is in the screw glue box area, set the time the screw glue box is not in the screw glue box area to 0. Then, judge whether the hand touches the screw box area according to the detection result of step 2. If the judgment result is that the hand touches the screw box area, set the state to the gluing state, and record the start time of the gluing state, and set whether the hand is in the screw holding state to yes. Then judge the time the screw glue box is not in the screw glue box area. If the time the screw glue box is not in the screw glue box area exceeds the threshold, and the state is not the gluing state, set the state to the non-gluing state. The flow chart is as follows Figure 5 As shown, Figure 5 yes Figure 1 A schematic flow chart of the judgment and processing of the method shown when the state is a state to be glued.

[0082] Step 3-3 is the judgment and processing step of the glue-applying state logic judgment module. The glue-applying state logic judgment module is further subdivided into four submodules: the screw glue area logic judgment module, the screw glue area logic judgment module, the screw glue leakage time judgment module, and the glue-applying state end judgment module. A schematic flow chart is as follows: Figure 6 As shown, Figure 6 yes Figure 1 A schematic diagram of the configuration of submodules of the glue-applying state logic judgment module involved in the method shown.

[0083] Step 3-3 may further include the following several sub-steps 3-3-1, 3-3-2, 3-3-3 and 3-3-4.

[0084] Step 3-3-1: When the glue is being applied, the logic judgment module of the screw glue area is first judged. Figure 7 As shown, Figure 7 yes Figure 1 A schematic flow chart of the judgment and processing of the method shown when the state is the gluing state.

[0085] According to the detection result of step 2, determine whether the hand touches the screw glue box area. When the judgment result is that the hand touches the screw glue box area, continue to the next step of judgment to determine whether the hand is in the state of touching the screw glue box area to pick up the glue stick or the state of touching the screw glue box area to put the glue stick back. When it is determined that the hand is neither in the state of touching the screw glue box area to pick up the glue stick nor in the state of touching the screw glue box area to put the glue stick back, determine whether the time when the hand touches the screw glue box area to pick up the glue stick ends is empty. If it is empty, it is considered that the current state is the hand touching the screw glue box to pick up the glue stick, and the start time of the hand touching the screw glue box to pick up the glue stick is recorded. If it is not empty, it is considered that the current state is the hand touching the screw glue box area to put the glue stick back, and the start time of the hand touching the screw glue box area to put the glue stick back is recorded.

[0086] When the judgment result is that the hand has not touched the screw glue box area, it is judged whether the hand is in the state of touching the screw glue box area to pick up the glue stick. If the hand is in the state of touching the screw glue box area to pick up the glue stick, the cumulative time when the hand touches the screw glue box area to pick up the glue stick is recorded. When the cumulative time exceeds the threshold, it is judged whether the glue stick is in the screw glue box area according to the detection result of step 2. If the glue stick is judged to be in the screw glue box area, the state of hand touching the screw glue box area to pick up the glue stick is considered invalid. If the glue stick is judged to be in the screw glue box area, the state of hand touching the screw glue box area to pick up the glue stick is considered valid, and the time when the hand touching the screw glue box area to pick up the glue stick is recorded. Otherwise, the state of hand touching the screw glue box area to pick up the glue stick is considered invalid, and the start time of the state of hand touching the screw glue box area to pick up the glue stick is set to null.

[0087] Then, determine whether the hand is in the state of touching the screw glue box area to put the glue stick back. If the hand is in the state of touching the screw glue box area to put the glue stick back, record the cumulative time when the hand touches the screw glue box area to put the glue stick back state ends. When the cumulative time exceeds the threshold, determine whether the glue stick is in the screw glue box area according to the detection result of step 2. If it is determined that the glue stick is not in the screw glue box area, it is considered that the hand touches the screw glue box area to put the glue stick back state is invalid, and the start time of the hand touches the screw glue box area to put the glue stick back state is set to empty. If it is determined that the glue stick is in the screw glue box area, it is considered that the hand touches the screw glue box area to put the glue stick back state is valid, and record the end time of the hand touches the screw glue box area to put the glue stick back state.

[0088] Step 3-3-2: The judgment process of the logic judgment module for the touched screw area is as follows: Figure 8 As shown, Figure 8 yes Figure 1A schematic flow chart of the judgment and processing of the logic judgment module for touching the screw area involved in the method shown. Judge whether the hand touches the screw area according to the detection result of step 2. When it is judged that the hand touches the screw area, judge whether the hand is in the state of holding the screw. If not, set the state of holding the screw to yes, and set the end time of the state of holding the screw to empty. When it is judged that the hand does not touch the screw area, judge whether it is in the state of holding the screw. If yes, record the cumulative time when the hand is in the state of holding the screw. When the cumulative time of the hand in the state of holding the screw exceeds the threshold, it is considered that the hand is in the state of holding the screw, and record the time when the hand touches the screw area to pick up the screw, and set the cumulative time when the hand is in the state of holding the screw to 0.

[0089] Step 3-3-3 is the judgment and processing of the screw glue leakage duration judgment module. The process is as follows Fig. 9 As shown, Fig. 9 yes Figure 1 A schematic flow chart of the judgment and processing of the screw glue leakage duration judgment module involved in the method shown.

[0090] First, determine whether the end time of the hand in the screw holding state is empty. If the end time of the hand in the screw holding state is not empty, then determine whether the start time of the hand in the touch screw glue box area to pick up the glue stick state is empty. If the start time of the hand in the touch screw glue box area to pick up the glue stick state is not empty, then calculate the time difference between the start time of the hand in the touch screw glue box to pick up the glue stick state and the end time of the hand in the screw holding state. If the difference exceeds the threshold, it is considered that the screw gluing is missed, and an alarm is triggered. If the start time of the hand in the touch screw glue box area to pick up the glue stick state is empty, then calculate the time difference between the current frame time and the end time of the hand touching the screw area to pick up the screw. If the difference exceeds the threshold, it is considered that the screw gluing is missed, and output the result of whether the screw is missed.

[0091] Step 3-3-4 is the judgment and processing of the glue application state end judgment module. This module is used to judge how to end the glue application state, so as to ensure the smooth operation of the process. The specific process is as follows: Fig.10 As shown, Fig.10 yes Figure 1 A schematic flow chart of the judgment and processing of the judgment module for the end of the gluing state involved in the method shown. First, it is judged whether the difference between the current frame time and the start time of the gluing state exceeds the threshold. If it exceeds the threshold, the current operation state is forced to be set to the state to be glued, and the variables needed in the process are initialized; then, it is judged whether the end time of the state of the hand touching the screw glue box area and putting the glue stick back is empty. If so, it is considered that the screw gluing process is over, the current operation state is set to the state to be glued, and the variables needed in the process are initialized.

[0092] Step 4

[0093] Step 4 is to perform corresponding processing according to whether there is any leakage of screw glue at the screw glue coating station output in step 3. If there is any leakage of screw glue at the station, it communicates with the alarm device of the station to make it alarm, then records the current frame time and enters the next cycle.

[0094] The above are all the process steps of an embodiment of a method for detecting glue leakage of screw glue in a factory.

[0095] At the same time, the present invention also provides a device and a medium for running and deploying the method, the specific structure is as follows Fig.11 As shown, Fig.11 Is configured for running, deploying Figure 1 According to one embodiment of the present invention, the inventors of the present invention deploy artificial intelligence deep learning algorithms, for example, using cameras with algorithms and recognition technologies based on real-time video single frames, in Fig.11 In the edge computing server shown in , the behavior of screw glue leakage at multiple workstations can be monitored based on the algorithm recognition of artificial intelligence technology. According to one example, each workstation can be provided with a camera and an alarm device, and the alarm device is not limited to an audible and visual alarm, a light alarm and / or an acoustic alarm. Among them, the edge computing server can obtain the real-time video data of each workstation through the rtsp video stream protocol of the camera configured at each workstation, and can analyze and determine whether there is a behavior of screw glue leakage at the workstation by performing the above method on the real-time video data of the workstation. When it is determined that a certain workstation has a situation of screw glue leakage, the edge computing server will notify the alarm device at the workstation to alarm in order to alert the operator.

[0096] Through the above-mentioned method of the present invention, including the above-mentioned exemplary judgment and identification technology combined with the algorithm, it is possible to accurately judge whether a certain workstation has a screw glue leakage situation, and issue a warning when this situation occurs, thereby effectively avoiding the occurrence of screw glue leakage in places such as factories and workshops, and reducing the failures, losses and safety hazards caused by screw glue leakage.

[0097] The foregoing description of several embodiments of the present invention is presented for illustrative purposes. The foregoing description is not intended to be exhaustive, nor is it intended to limit the present invention to the precise steps and / or forms disclosed, and it is apparent that many modifications and variations may be made in light of the above teachings. The scope of the present invention and all equivalents are intended to be defined by the appended claims.

Claims

1. A method for detecting screw glue leakage based on a deep learning algorithm, characterized in that: The method comprises the following steps: Step 1: Pre-process the real-time video of the screw glue application station; Step 2: performing algorithm recognition based on a deep learning algorithm frame by frame on the preprocessed real-time video to identify the categories and positions of three objects, namely, a screw glue box, a hand, and a screw glue stick, in the screw glue application station; Step 3: jointly judging the multiple frames of recognition results of the algorithm recognition of the real-time video, so as to judge and process the respective situations of the screw glue gluing station being in the non-gluing state, the state to be glued and the state of being glued, including switching between the non-gluing state, the state to be glued and the state of being glued of the screw glue gluing station, wherein the initial state of the screw glue gluing station is set to the non-gluing state; and Step 4: Perform corresponding processing according to whether there is any screw glue leakage at the screw glue coating station output in step 3, including: if there is any screw glue leakage at the screw glue coating station, instruct the alarm device of the screw glue coating station to alarm, and record the time of the current frame of the real-time video.

2. The method according to claim 1, characterized in that: The step 1 includes the following sub-steps: Step 1-1: manually marking the screw glue box area and the screw area in the real-time video frame; Step 1-2: using a camera to collect data of the real-time video of the screw glue application station area; Step 1-3: Cutting frames of the collected real-time video data; Step 1-4: Calculate the frame difference between the current frame and the previous frame of the real-time video using a frame difference method. If the threshold of the frame difference is lower than a set value, do not pre-process the current frame. and Step 1-5: If the threshold of the frame difference is higher than a set value, pre-processing operation is performed on the current frame.

3. The method according to claim 2, characterized in that The step 2 includes the following sub-steps: Step 2-1: using a deep learning neural network model, extracting features from the image / video data of the current frame that has been preprocessed in step 1; Step 2-2: The features extracted in step 2-1 are placed into a feature enhancement module that combines an improved attention mechanism and an enhanced convolutional attention module of the deep learning neural network model to perform feature enhancement; and Step 2-3: Send the feature data enhanced by the feature in step 2-2 to a multi-layer perceptron to predict the target position and target category, so as to identify the categories and positions of the three objects, namely, the screw glue box, the hand, and the screw glue stick, in the screw glue applying station.

4. The method according to claim 3, characterized in that The improved attention mechanism is expressed by the following formula 1: Wherein, H is the height of the input tensor of the deep learning neural network model, z is the improved attention value, W is the width of the input tensor of the deep learning neural network model, x is the self-attention function, i is the index of the input tensor width of the deep learning neural network model, and c is the channel index; And, the enhanced convolutional attention module is expressed by the following formula 2: Among them, M is the enhanced convolution attention value, F is the input tensor of the deep learning neural network model, σ is the Sigmoid activation function, W1 is the value of the second dimension of the input tensor of the deep learning neural network model, W0 is the value of the first dimension of the input tensor of the deep learning neural network model, MaxPool is the maximum pooling function, AvgPool is the average pooling function, MLP is the multi-layer perceptron function, max is the maximum value function, avg is the average value function, and c is the channel index.

5. The method according to claim 4, characterized in that The step 3 includes the following sub-steps: Step 3-1: When the screw glue gluing station is in a non-gluing state, determine whether the screw glue box is in the screw glue box area through the detection result of the current frame in step 2; when it is determined that the screw glue box is in the screw glue box area, record the time the screw glue box is in the screw glue box area, and set the time the screw glue box is not in the screw glue box area to 0; when it is determined that the screw glue box is not in the screw glue box area, record the time the screw glue box is not in the screw glue box area, and set the time the screw glue box is in the screw glue box area to 0; if it is determined that the time the screw glue box is in the screw glue box area exceeds a threshold, set the state of the screw glue gluing station to a waiting state; Step 3-2: When the screw glue coating station is in the state of waiting for gluing, first, determine whether the screw glue box is in the screw glue box area. When it is determined that the screw glue box is not in the screw glue box area, record the time period that the screw glue box is not in the screw glue box area; when it is determined that the screw glue box is in the screw glue box area, set the time period that the screw glue box is not in the screw glue box area to 0; then, determine whether the hand touches the screw box area according to the detection result of step 2. If it is determined that the hand touches the screw box area, set the state of the screw glue coating station to the gluing state, record the start time of the gluing state, and set whether the hand is in the screw holding state to yes; then, determine the time period that the screw glue box is not in the screw glue box area. If the time period exceeds the threshold and the screw glue coating station is not in the gluing state, set the state of the screw glue coating station to the non-gluing state; and Step 3-3: When the screw glue coating station is in the gluing state, judgment and processing are performed based on at least one of the following judgment module items: a logic judgment module for touching the screw glue area, a logic judgment module for touching the screw area, a screw glue missing coating time judgment module, and a gluing state end judgment module.

6. The method according to claim 5, characterized in that The step 3-3 includes the following subdivision steps: Step 3-3-1: When the screw glue coating station is in the gluing state, first determine whether the hand touches the screw glue box area according to the detection result of step 2. When it is determined that the hand has touched the screw glue box area, proceed to the next step to determine whether the hand is in the state of touching the screw glue box area to pick up the glue stick or the state of touching the screw glue box area to put the glue stick back. When it is determined that the hand is neither in the state of touching the screw glue box area to pick up the glue stick nor in the state of touching the screw glue box area to put the glue stick back, determine whether the end time of the state of the hand touching the screw glue box area to pick up the glue stick is empty. If it is empty, it is considered that the hand is currently in the state of touching the screw glue box to pick up the glue stick, and the start time of the state of the hand touching the screw glue box to pick up the glue stick is recorded. If it is not empty, it is considered that the hand is currently in the state of touching the screw glue box area to put the glue stick back, and the start time of the state of the hand touching the screw glue box area to put the glue stick back is recorded. When it is determined that the hand has not touched the screw glue box area, it is determined whether the hand is in the state of touching the screw glue box area to pick up the glue stick. If the hand is in the state of touching the screw glue box area to pick up the glue stick, the cumulative duration of the hand touching the screw glue box area to pick up the glue stick before the end of the state is recorded. When the cumulative duration exceeds the threshold, it is determined whether the glue stick is in the screw glue box area according to the detection result of step 2. If the glue stick is determined to be in the screw glue box area, it is considered that the state of the hand touching the screw glue box area to pick up the glue stick is invalid; if the glue stick is determined to be in the screw glue box area, it is considered that the state of the hand touching the screw glue box area to pick up the glue stick is valid, and the end time of the state of the hand touching the screw glue box area to pick up the glue stick is recorded, otherwise, it is determined that the state of the hand touching the screw glue box area to pick up the glue stick is invalid, and the start time of the state of the hand touching the screw glue box area to pick up the glue stick is set to empty; and Then determine whether the hand is in the state of touching the screw glue box area to put the glue stick back. If the hand is in the state of touching the screw glue box area to put the glue stick back, then record the cumulative time before the end of the state of touching the screw glue box area to put the glue stick back. When the cumulative time exceeds the threshold, determine whether the glue stick is in the screw glue box area according to the detection result of step 2; if it is determined that the glue stick is not in the screw glue box area, it is considered that the state of touching the screw glue box area to put the glue stick back is invalid, and the start time of the state of touching the screw glue box area to put the glue stick back is set to null; if it is determined that the glue stick is in the screw glue box area, it is considered that the state of touching the screw glue box area to put the glue stick back is valid, and the end time of the state of touching the screw glue box area to put the glue stick back is recorded; Step 3-3-2: judging whether the hand touches the screw area according to the detection result of step 2, and judging whether the hand is in the state of holding the screw when it is judged that the hand touches the screw area, and if not, setting the state of holding the screw to yes and setting the end time of the state of holding the screw to empty; When it is determined that the hand does not touch the screw area, it is determined whether the state is in the hand holding the screw state. If so, the cumulative time before the end of the hand holding the screw state is recorded. If the cumulative time exceeds a threshold, it is determined that the hand holding the screw state is ended, and the end time of the hand touching the screw area to pick up the screw state is recorded, and the cumulative time before the end of the hand holding the screw state is set to 0; and Step 3-3-3: First, determine whether the end time of the hand holding the screw state is empty. If the end time of the hand holding the screw state is not empty, then determine whether the start time of the hand touching the screw glue box area to pick up the glue stick state is empty. If the start time of the hand touching the screw glue box area to pick up the glue stick state is not empty, then calculate the difference between the start time of the hand touching the screw glue box to pick up the glue stick state and the end time of the hand holding the screw state; if the difference exceeds the threshold, it is determined that there is currently a behavior of leaking glue on the screw, and an alarm is triggered; if the start time of the hand touching the screw glue box area to pick up the glue stick state is empty, then calculate the time difference between the time of the current frame of the real-time video and the end time of the hand touching the screw area to pick up the screw state. If the time difference exceeds the threshold, it is determined that there is currently a behavior of leaking glue on the screw; and Step 3-3-4: First, determine whether the difference between the time of the current frame of the real-time video and the start time of the gluing state exceeds a threshold. If it exceeds the threshold, set the current state of the screw glue gluing station to the state to be glued, and initialize the relevant process variables; then, determine whether the end time of the hand touching the screw glue box area and putting back the glue stick state is empty. If it is empty, it is considered that the screw gluing process is completed, and at the same time, set the current state of the screw glue gluing station to the state to be glued, and initialize the relevant process variables.

7. The method according to any one of claims 1 to 6, characterized in that After step 4 is completed, re-entering the next step cycle of the method includes: re-executing step 1, step 2, step 3 and step 4.

8. The method according to any one of claims 1 to 7, characterized in that The deep learning algorithm in the method relies on high-performance computing servers for execution.

9. The method according to claim 8, characterized in that The high-performance computing server includes an edge computing server, which is configured to simultaneously detect screw glue leakage behavior at multiple screw glue application stations.

10. The method according to any one of claims 1 to 7, characterized in that The data of the real-time video is obtained through the rtsp video stream protocol of the camera configured at each screw glue application station.

Images