Method for judging abnormal actions in repeated mechanical actions
By utilizing feature extraction and frequency domain feature analysis in automated production, abnormal equipment actions can be identified in real time, solving the problem of difficulty in detecting occasional abnormal actions and achieving efficient and low-cost abnormal action detection.
Patent Information
- Application Number
- CN202511049598.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-29
- Publication Date
- 2025-11-11
AI Technical Summary
In automated production processes, occasional abnormal actions of equipment are difficult to detect in a timely manner, leading to fluctuations in product quality and equipment failures. Existing manual inspection methods are inefficient and costly.
By acquiring continuous frame images of the device during normal operation, a feature extraction model is used to convert them into feature vectors, a feature prediction model is trained, and the device is judged in real time whether abnormal actions occur. Frequency domain feature analysis is used to determine whether the device is abnormal.
It enables timely and accurate judgment of abnormal equipment actions, reduces the requirements for training samples, and reduces labor costs.
Smart Images
Figure CN120932030A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of automated production technology, and in particular to a method for judging abnormal actions in repetitive mechanical movements. Background Technology
[0002] In automated production processes, such as cigarette manufacturing, numerous pieces of equipment perform repetitive mechanical actions. Due to wear and tear on parts, vibration, slippage, or material quality issues, some equipment may exhibit occasional abnormal movements during hundreds of thousands of cycles. These occasional abnormal movements are difficult to observe by operators and maintenance personnel because they occur very briefly, are unpredictable, and occur in very low frequency. Consequently, maintenance personnel cannot promptly repair or maintain these abnormal movements. These occasional abnormal movements can cause fluctuations in product quality on the production line and may even lead to equipment malfunctions.
[0003] Currently, the main method for checking equipment for occasional abnormal movements is by manually monitoring it. This method is difficult to detect abnormal movements in a timely manner and is also costly in terms of manpower. Summary of the Invention
[0004] This invention provides a method for judging abnormal actions in repetitive mechanical movements, in order to solve the technical problem of how to detect abnormal actions in a timely manner.
[0005] To address the aforementioned technical problems, this invention provides a method for determining abnormal actions in repetitive mechanical movements, comprising the following steps:
[0006] S1. Acquire the first consecutive frame image corresponding to at least one action cycle of the device during normal operation;
[0007] S2. Using a feature extraction model, each frame in the first consecutive frame images is converted into a first feature vector in chronological order.
[0008] S3. Set the number of first feature vectors contained in each first input variable, and the time interval between each first input variable and its corresponding first output variable; sequentially select each first input variable and its corresponding first output variable from multiple first feature vectors in chronological order, and train the feature prediction model using multiple first input variables and multiple corresponding first output variables; wherein, each first input variable contains an equal number of first feature vectors and is more than one first feature vector, the first output variable contains only one first feature vector, the first input variables and the first output variables correspond one-to-one, and the time interval between each first input variable and its corresponding first output variable is equal;
[0009] S4. Obtain the second consecutive frame image corresponding to the mechanical actions of the device during operation;
[0010] S5. Using the feature extraction model, each frame of the second consecutive frame image is converted into a second feature vector in chronological order.
[0011] S6. In chronological order, select a second input variable from multiple second feature vectors and input the second input variable into the trained feature prediction model to obtain a predicted feature vector; wherein, the number of second feature vectors contained in each second input variable is equal to the number of first feature vectors contained in each first input variable;
[0012] S7. Based on the time when the predicted feature vector is located, obtain the actual feature vector corresponding to the predicted feature vector, and calculate the similarity between the predicted feature vector and the corresponding actual feature vector;
[0013] S8. Repeat steps S6 and S7 to obtain multiple similarities;
[0014] S9. Calculate the frequency domain features of multiple similarities and determine whether the frequency domain features meet the preset conditions. If they meet the conditions, it is determined that the device has not performed any abnormal actions; if they do not meet the conditions, it is determined that the device has performed any abnormal actions.
[0015] Preferably, step S1 includes the following steps: determining the action cycle required for the device to complete a complete mechanical action during normal operation; acquiring the video corresponding to at least one consecutive action cycle of the device during normal operation; and converting the video into a first consecutive frame image in chronological order.
[0016] Preferably, the first consecutive frame image and the second consecutive frame image are obtained by the same camera and using the same shooting parameters.
[0017] Preferably, the similarity is cosine similarity.
[0018] Preferably, the frequency domain features are calculated using a Fourier algorithm.
[0019] Preferably, step S9 includes the following steps: calculating the frequency domain features of multiple similarities using the Fourier algorithm, and determining whether the amplitude of the first-order feature is within a preset range. If it is, the device is determined not to have performed any abnormal actions; if not, the device is determined to have performed any abnormal actions.
[0020] This invention provides a method for identifying abnormal actions in repetitive mechanical movements. First, a first feature vector is obtained from a first series of consecutive frames corresponding to at least one action cycle during normal operation of the equipment. Then, a feature prediction model is trained using the first feature vector. During equipment operation, a second series of consecutive frames are acquired in real-time, and a second feature vector is obtained from these frames. This second feature vector is then input into the trained feature prediction model to obtain a predicted feature vector. Finally, the similarity between the predicted and measured feature vectors is converted into frequency domain features. By using the frequency domain features of the mechanical movements, it is possible to determine in a timely and accurate manner whether abnormal actions have occurred. Furthermore, this invention requires only a small amount of sample data from normal operation of the equipment, eliminating the need for large amounts of abnormal sample data or extensive annotation of abnormal action features, thus reducing the requirements for training samples. Attached Figure Description
[0021] Figure 1 This is a flowchart of a method for judging abnormal actions in repetitive mechanical actions according to an embodiment of the present invention.
[0022] Figure 2 This is a schematic diagram of the processing of a feature prediction model based on the first consecutive frame images of a device during normal operation, provided by an embodiment of the present invention.
[0023] Figure 3 This is a schematic diagram of the process from the second consecutive frame image of the device to be judged to the similarity calculation provided by an embodiment of the present invention.
[0024] Figure 4 This is a schematic diagram of the process from similarity sequence to frequency domain feature map provided by an embodiment of the present invention. Detailed Implementation
[0025] To make the objectives, advantages, and features of the present invention clearer, the method for judging abnormal actions in repetitive mechanical movements proposed by the present invention will be further described in detail below with reference to the accompanying drawings. It should be noted that the drawings are all in a very simplified form and use non-precise proportions, and are only used to facilitate and clarify the explanation of the embodiments of the present invention.
[0026] In the description of this invention, the terms "first," "second," and other qualifiers are added for convenience of description and reference, and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with qualifiers such as "first" and "second" may explicitly or implicitly include one or more of that feature.
[0027] like Figure 1 As shown, this embodiment provides a method for determining abnormal actions in repetitive mechanical movements, including the following steps:
[0028] S1. Obtain the first consecutive frame image corresponding to at least one action cycle of the device during normal operation. Preferably, step S1 includes the following steps: determining the action cycle required for the device to complete a full mechanical action during normal operation; obtaining the video corresponding to at least one consecutive action cycle of the device during normal operation; and converting the video into the first consecutive frame image in chronological order. For example, first, use a camera to shoot a normal video of the device pushing a packaging box into the packaging machine during normal operation. A complete action cycle is approximately 120 milliseconds, the shooting time can be about 3 minutes, the shooting frame rate can be 12 frames per second, and the resolution can be 80×60 pixels. After obtaining the video, the video can be decoded into consecutive frame images in JPG format or other formats. For ease of reference, the consecutive frame images here are referred to as the first consecutive frame images. In other embodiments, video can be shot simultaneously and then decoded into the first consecutive frame images.
[0029] S2. Using a feature extraction model, each frame in the first consecutive image frames is converted into a first feature vector in chronological order. Preferably, a pre-trained model such as the backbone network of YOLO-v9 (You Only Look Once, the ninth-generation object detection model), VGG (Visual Geometry Group), or ResNet (Residual Network) can be used as the feature extraction model. Pre-training means that the model has been trained using conventional data, and the pre-trained model can recognize conventional objects such as vehicles and people. The feature extraction model can extract features such as color, texture, and shape from the image. Figure 2 As shown, 9 represents the first consecutive frame image in JPG format obtained by normal video decoding, 10 represents the feature extraction model, and 11 represents the first feature vector. The first feature vectors are arranged in chronological order.
[0030] S3. Set the number of first feature vectors contained in each first input variable, and the time interval between each first input variable and its corresponding first output variable; sequentially select each first input variable and its corresponding first output variable from multiple first feature vectors in chronological order, and train the feature prediction model using multiple first input variables and multiple corresponding first output variables; wherein each first input variable contains an equal number of first feature vectors and is more than one first feature vector, the first output variable contains only one first feature vector, the first input variables and the first output variables are in one-to-one correspondence, and the time interval between each first input variable and its corresponding first output variable is equal. The number of first feature vectors contained in each first input variable and the time interval between each first input variable and its corresponding first output variable can be set based on experience and experimental results. For example, the number of first feature vectors contained in each first input variable can be set to 4, and the time interval between each first input variable and its corresponding first output variable can be set to 15 units of time, where each unit of time represents the time between two adjacent first feature vectors. The feature prediction model can be composed of a sequentially connected fully connected network, an LSTM (Long Short-Term Memory) network, and a fully connected network, or a single LSTM. When a feature prediction model consists of a fully connected network, an LSTM, and another fully connected network connected in sequence, the computational cost of the LSTM can be reduced, thus improving the computational speed of the feature prediction model. For example, a fully connected network with an input of 1800 and an output of 512 can be used first; then a two-layer LSTM with an input layer width of 512 and a hidden layer width of 512 can be used; finally, a fully connected network with an input of 512 and an output of 1800 can be used, thus ensuring that both the input and output of the feature prediction model are 1800. The units 512 and 1800 represent the total number of data points included in the vector. The first input variable can be a single first feature vector or a sequence of two or more first feature vectors. For example, if the first feature vector of the current input is at time t, the first feature vector at time t+15 can be used as the corresponding first output variable; similarly, the first feature vector at time t+1 corresponds to the first feature vector at time t+16; and the first feature vector at time t+n corresponds to the first feature vector at time t+15+n. The feature prediction model is trained until convergence. Figure 2As shown, 12 represents the first feature vector sequence of the input feature prediction model at time t (at this time, the first feature vector sequence contains 4 first feature vectors), 13 represents the feature prediction model to be trained, and 14 represents the first feature vector at time t+15, which is also the first output variable corresponding to 12, i.e., the target output. Assuming that 14 is the first first input variable, the times corresponding to the 4 first feature vectors it contains are t, t+1, t+2, and t+3 respectively. The first time t or the last time t+3 of the first input variable can be used as the time of the current first input variable. The next time, the 4 first feature vectors corresponding to times t+1, t+2, t+3, and t+4 can be used as the second first input variable, or t+2, t+3, t+4, and t+5 can be selected as the second first input variable in a skipping manner.
[0031] S4. Obtain the second consecutive frame images corresponding to the mechanical actions of the device during operation; this can be done by first capturing video and then converting the video into the second consecutive frame images, and the length of the video is not limited in this step; alternatively, the video can be captured and converted into images simultaneously. Preferably, the first and second consecutive frame images are obtained using the same camera and the same shooting parameters, so that the feature vectors corresponding to the same actions are the same, thereby improving the accuracy of determining whether abnormal actions have occurred.
[0032] S5. Using the feature extraction model, each frame in the second consecutive frame images is converted into a second feature vector in chronological order. For ease of reference, the feature vector with judgment is called the second feature vector. Each first feature vector and each second feature vector contain the same amount of data, for example, 1800 data points. Figure 3 As shown, 15 represents the second consecutive frame image after the video is decoded, and 16 represents the second feature vector to be judged.
[0033] S6. Following the chronological order, sequentially select a second input variable from multiple second feature vectors, and input the second input variable into the trained feature prediction model to obtain a predicted feature vector; wherein, the number of second feature vectors contained in each second input variable is equal to the number of first feature vectors contained in each first input variable. Figure 3 As shown, assuming the current input to the trained feature prediction model 18 is the second feature vector sequence 17, which includes four second feature vectors, the output of the trained feature prediction model 18 is the predicted feature vector 20. The second feature vector sequence 17 and the predicted feature vector 20 correspond to each other. Figure 3 The time interval between 17 and 20 is equal to Figure 2 The time interval between 12 and 14. (e.g.) Figure 3As shown, assuming the second feature vector sequence 17 corresponds to time t, the predicted feature vector 20 corresponds to time t+15.
[0034] S7. Based on the time of the predicted feature vector, obtain the measured feature vector corresponding to the predicted feature vector, and calculate the similarity between the predicted feature vector and the corresponding measured feature vector. The time interval between the second input variable and the predicted feature vector is equal to the time interval between the first input variable and the first output variable, for example, 15 units of time. Based on the time of the second input variable and the time interval between the second input variable and the predicted feature vector, the time of the predicted feature vector can be determined. Based on the time of the predicted feature vector, the measured feature vector corresponding to the measured second consecutive frame image at that time can be obtained. Preferably, the similarity is cosine similarity, bulldozer similarity, or Euclidean similarity. Figure 3 As shown, 20 represents the predicted feature vector at time t+15, 19 represents the measured feature vector at time t+15, 21 represents the similarity calculation algorithm, and 22 represents the similarity.
[0035] S8. Repeat steps S6 and S7 to obtain multiple similarities; each time steps S6 and S7 are executed, one similarity can be obtained; when repeating steps S6 and S7, the second input variable is updated in chronological order to obtain the predicted feature vectors corresponding to different second input variables.
[0036] S9. Calculate the frequency domain features of multiple similarities and determine whether the frequency domain features meet preset conditions. If they do, determine that the device has not performed any abnormal actions; otherwise, determine that the device has performed any abnormal actions. Preferably, the frequency domain features are calculated using a Fourier algorithm such as FFT (Fast Fourier Transform) or DFT (Discrete Fourier Transform). Specifically, whenever 64 similarities are obtained, the frequency domain features of multiple similarities are calculated using the FFT algorithm, and it is determined whether the first-order feature amplitude is within a preset range. If it is, the device has not performed any abnormal actions; otherwise, the device has performed any abnormal actions. By converting an appropriate number of similarities into frequency domain features, and judging the feature amplitude of a certain order, it is possible to quickly and accurately determine whether the device has performed any abnormal actions. For example, the first to 64th similarities are used as inputs to the FFT (the next inputs are the second to 65th similarities, and so on, determining subsequent inputs) to calculate frequency domain features. The first 32 frequency domain features are retained. Since the amplitudes after the sixth order are very small, only at least one of the feature amplitudes from the first to the fifth order needs to be considered. The sum of the feature amplitudes from the first to the fifth order is close to 1. During normal operation, the feature amplitudes of each order are within a preset range. If the amplitude of at least one order changes significantly, it indicates an abnormal operation of the equipment. For example, a preset range is set for the amplitude of the first order feature. This preset range can be obtained through experiments using sample data from normal operation. If the amplitude of the first order feature is within the preset range, it indicates that the equipment is not malfunctioning; if the amplitude of the first order feature is outside the preset range, it indicates that the equipment is malfunctioning. If the equipment malfunctions, the video of that period can be provided to maintenance personnel for observation and diagnosis, allowing for timely maintenance and repair. Figure 4 As shown, 23 represents the cosine similarity sequence data of the device's normal operation, 24 represents the cosine similarity sequence data of the device's abnormal operation, 25 represents the waveform corresponding to 23, 26 represents the waveform corresponding to 24, 27 represents the FFT algorithm, 28 represents the frequency domain feature corresponding to 23, and 29 represents the frequency domain feature corresponding to 24. Assuming the preset range of the first-order feature amplitude is 0.6–0.7, since… Figure 3 The first bar on the left, representing the first-order characteristic amplitude, is within the range of 0.6 to 0.7, therefore it can be determined that the equipment was operating normally during the time period corresponding to 28. Figure 4 As can be seen from 29, the first-order characteristic amplitude decreases and is no longer within the range of 0.6 to 0.7. When the first-order characteristic amplitude decreases, at least one other-order characteristic amplitude will increase. Therefore, it can be determined that the device has performed abnormal actions within the time period corresponding to 29.
[0037] This invention provides a method for identifying abnormal actions in repetitive mechanical movements. First, a first feature vector is obtained from a first series of consecutive frames corresponding to at least one action cycle during normal operation of the equipment. Then, a feature prediction model is trained using the first feature vector. During equipment operation, a second series of consecutive frames are acquired in real-time, and a second feature vector is obtained from these frames. This second feature vector is then input into the trained feature prediction model to obtain a predicted feature vector. Finally, the similarity between the predicted and measured feature vectors is converted into frequency domain features. By using the frequency domain features of the mechanical movements, it is possible to determine in a timely and accurate manner whether abnormal actions have occurred. Furthermore, this invention requires only a small amount of sample data from normal operation of the equipment, eliminating the need for large amounts of abnormal sample data or extensive annotation of abnormal action features, thus reducing the requirements for training samples.
[0038] The above description is only a description of preferred embodiments of the present invention and is not intended to limit the scope of the present invention in any way. Any changes or modifications made by those skilled in the art based on the above disclosure shall fall within the protection scope of the present invention.
Claims
1. A method for judging abnormal actions in repetitive mechanical movements, characterized in that, Includes the following steps: S1. Acquire the first consecutive frame image corresponding to at least one action cycle of the device during normal operation; S2. Using a feature extraction model, each frame in the first consecutive frame images is converted into a first feature vector in chronological order. S3. Set the number of first feature vectors contained in each first input variable, and the time interval between each first input variable and its corresponding first output variable; sequentially select each first input variable and its corresponding first output variable from multiple first feature vectors in chronological order, and train the feature prediction model using multiple first input variables and multiple corresponding first output variables; wherein, each first input variable contains an equal number of first feature vectors and is more than one first feature vector, the first output variable contains only one first feature vector, the first input variables and the first output variables correspond one-to-one, and the time interval between each first input variable and its corresponding first output variable is equal; S4. Obtain the second consecutive frame image corresponding to the mechanical action of the device during operation; S5. Using the feature extraction model, each frame of the second consecutive frame image is converted into a second feature vector in chronological order. S6. In chronological order, select a second input variable from multiple second feature vectors and input the second input variable into the trained feature prediction model to obtain a predicted feature vector; wherein, the number of second feature vectors contained in each second input variable is equal to the number of first feature vectors contained in each first input variable; S7. Based on the time when the predicted feature vector is located, obtain the actual feature vector corresponding to the predicted feature vector, and calculate the similarity between the predicted feature vector and the corresponding actual feature vector; S8. Repeat steps S6 and S7 to obtain multiple similarities; S9. Calculate the frequency domain features of multiple similarities and determine whether the frequency domain features meet the preset conditions. If they meet the conditions, it is determined that the device has not performed any abnormal actions; if they do not meet the conditions, it is determined that the device has performed any abnormal actions.
2. The method for determining abnormal actions in repetitive mechanical movements as described in claim 1, characterized in that, Step S1 includes the following steps: determining the action cycle required for the device to complete a full mechanical action during normal operation; acquiring the video corresponding to at least one consecutive action cycle of the device during normal operation; and converting the video into a first consecutive frame image in chronological order.
3. The method for determining abnormal actions in repetitive mechanical movements as described in claim 1, characterized in that, The first and second consecutive frame images were obtained using the same camera and the same shooting parameters.
4. The method for determining abnormal actions in repetitive mechanical movements as described in claim 1, characterized in that, The similarity is cosine similarity.
5. The method for determining abnormal actions in repetitive mechanical movements as described in claim 1, characterized in that, The frequency domain features are calculated using the Fourier algorithm.
6. The method for determining abnormal actions in repetitive mechanical movements as described in claim 5, characterized in that, Step S9 includes the following steps: calculating the frequency domain features of multiple similarities using the Fourier algorithm, and determining whether the amplitude of the first-order feature is within a preset range. If it is, the device is determined to have no abnormal operation; if not, the device is determined to have an abnormal operation.