A method and device for visual inspection and positioning of key workpiece installation processes

Abstract

The present application discloses a method and device for visual inspection and positioning of key workpiece installation processes, which belongs to the field of video behavior recognition. When performing workpiece installation process inspection and positioning, first collect a video image of the workpiece installation process, extract image features for rough positioning of the process to be inspected; based on the results of the rough positioning and the time required for the entire installation process of the workpiece, intercept the video of the process to be inspected in the workpiece installation; extract features from the intercepted workpiece installation process video to obtain the motion features of the workpiece installation process, and classify them, and judge whether the workpiece installation process is qualified based on the classification results. The device of the present application can implement the corresponding method steps of the present application. The present application can effectively detect and locate the workpiece installation process video, solving the problem that the prior art has poor detection effect on the workpiece installation process in the factory.

Description

Technical Field

[0001] The present invention belongs to the field of video behavior recognition, and in particular relates to a method and a device for visual detection and positioning of a key process of workpiece installation. Background Art

[0002] In the installation of key parts in the machinery industry with large production volumes and intensive technology, such as subways, high-speed railways, ships, etc., tiny differences in movements in many processes will have a huge impact on the quality of the entire production. The process is not only the basic link in the manufacturing process of product parts, but also the basic link in forming product quality. Therefore, process quality is the main object of enterprise on-site quality control and is also the key to the production site quality assurance system.

[0003] The installation process of some important parts is prone to fatigue due to the large workload, high repetitiveness, and strict requirements for action specifications, which can cause various human quality problems (such as loose screws, falling off, etc.) and whether the key process actions are completed accurately and orderly. These issues are particularly important. At present, for most factories, video surveillance is used to collect videos of the workpiece installation process, and then the video is sent back to the background server. Then, the installation process of the workpiece is visually inspected by manually viewing the video. However, relying solely on manual visual inspection for inspection is inefficient, has poor control effects, and cannot well guarantee the quality of the product process.

[0004] In order to detect the installation process more efficiently, the prior art has proposed a technology for process detection using deep learning. For example, Chinese patent application number 201810137318.2 discloses a process detection device based on deep learning and a process detection method thereof. However, such methods require a large amount of factory data support. However, for most factories, it is often difficult to meet the data needs. Therefore, such methods are still difficult to obtain effective results and cannot perform good detection on the workpiece installation process. Summary of the invention

[0005] Technical problem: In view of the problem that the existing technology has poor results in detecting and positioning the installation process of the workpiece, a method and device for visual detection and positioning of the key workpiece installation process is proposed, so that the workpiece installation process can be effectively detected and positioned.

[0006] Technical solution: On the one hand, the present application provides a method for visual inspection and positioning of key workpiece installation processes, including:

[0007] Collect video images of the workpiece installation process, extract necessary reference image features and perform positioning;

[0008] According to the image features of the necessary references located and the time required for the entire installation process of the workpiece, a video of the workpiece installation process is captured;

[0009] Extract features from the captured workpiece installation process video to obtain action features of the workpiece installation process;

[0010] The action features are classified, and whether the workpiece installation process is qualified is determined based on the classification results.

[0011] Furthermore, the method of extracting necessary reference image features and performing positioning includes:

[0012] Taking the necessary steps or necessary tools in the workpiece installation process as necessary references, extracting the image features of the necessary references through one or more of the color, contour and texture of the necessary references;

[0013] Frame and locate the image features that are necessary for reference.

[0014] Furthermore, the method for capturing the workpiece installation process video according to the located necessary reference image features and the time required for the entire workpiece installation process includes:

[0015] The video frame where the located necessary reference image feature is located is used as a starting key frame;

[0016] Determine the time required for the entire installation process of the workpiece;

[0017] Taking the starting key frame as a node, several minutes of long videos are captured forward and backward respectively, and the long videos cover the entire installation process of the workpiece.

[0018] Furthermore, the method of extracting features from the intercepted workpiece installation process video to obtain action features of key video sequences of the workpiece installation process includes:

[0019] The captured workpiece installation process video is used to obtain the action boundary information and action behavior characteristics using the background clipping method and the dense trajectory method respectively.

[0020] Performing feature fusion on the action boundary information and the action behavior features;

[0021] The fused features are sent to the feature extraction network for recognition to obtain the action features of the workpiece installation process.

[0022] Furthermore, the method of extracting features from the intercepted workpiece installation process video to obtain action features of key video sequences of the workpiece installation process also includes: training a feature extraction network; the training method includes

[0023] Collect video samples of the workpiece installation process;

[0024] Segment the video samples of interest according to the workpiece installation process;

[0025] The segmented video segments are labeled and used as training samples for training the feature extraction network;

[0026] The feature extraction network is trained using training samples.

[0027] Furthermore, the feature extraction network is a C3D network, and the pooling kernel size of the pooling layer Pool5 in the C3D network is 7×7 and the step size is 7.

[0028] Furthermore, the method for classifying action features includes:

[0029] The Softmax classifier is used to classify the action features and perform regression optimization.

[0030] Furthermore, the method for judging whether the operator installs the workpiece according to the process requirements according to the classification result includes:

[0031] Through the Softmax classifier, the score of each operation process is obtained;

[0032] Determine whether the process is qualified based on the score of each process and the score threshold of the corresponding process.

[0033] Furthermore, it also includes: after judging according to the classification result, if it is qualified, the video of the completed inspection is archived; if it is unqualified, an early warning prompt is issued, and a prompt is given to re-inspect the corresponding process.

[0034] On the other hand, the present application provides a device for visual inspection and positioning of key workpiece installation processes, comprising:

[0035] Video image acquisition module, used to acquire video images of the workpiece installation process;

[0036] A reference positioning module is used to extract the image features of the necessary reference and perform positioning;

[0037] A process video capture module is used to capture the workpiece installation process video based on the necessary reference image features located and the time required for the entire installation process of the workpiece;

[0038] A feature extraction module is used to extract features of the captured workpiece installation to obtain action features of key video sequences of the workpiece installation process;

[0039] The evaluation module is used to classify the action features and determine whether the workpiece installation process is qualified according to the classification results.

[0040] Beneficial effect: Compared with the existing technology, the technical solution of the present application first extracts and locates the necessary references before detecting the workpiece installation process, and intercepts the video of the workpiece installation process based on the necessary references, thereby improving the efficiency of the entire workpiece installation process detection. Then, the intercepted workpiece installation process video uses the background clipping method and the dense trajectory method to obtain the action boundary information and action behavior characteristics, and the action boundary information and action behavior characteristics are feature fused, and the fused features are sent to the feature extraction network for identification to obtain the action characteristics of the workpiece installation process. The feature extraction is completed by combining the traditional method with the neural network, with high accuracy and speed, so that the workpiece installation process can be effectively detected and located. Finally, the action features are classified, and the workpiece installation process is judged whether it is qualified according to the classification results, thereby judging whether the workpiece installation process is qualified and making corresponding operations. Therefore, the technical solution of the present application can effectively detect the workpiece installation process. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A flow chart of the method in the embodiments of the present application;

[0042] Figure 2 A schematic diagram of positioning by extracting a wrench in an embodiment of the present application;

[0043] Figure 3 A flowchart of feature extraction and classification in an embodiment of the present application;

[0044] Figure 4 This is a C3D network structure diagram used in the embodiments of the present application;

[0045] Figure 5 A flow chart of detecting a workpiece installation process in an embodiment of the present application;

[0046] Figure 6 This is a diagram illustrating an error in the use of a wrench detected in an embodiment of the present application. DETAILED DESCRIPTION

[0047] The present invention is further described below in conjunction with the embodiments and drawings of the specification. For a clearer description, the embodiments of the present application are described in conjunction with a specific application scenario. The scenario is the process of installing bolts on a certain equipment in a factory. When installing the bolts, there are a total of three key processes: rotating the wrench to fix the bolts, marking after the installation is completed, and painting the installed bolts. Figure 1 A flowchart showing a method of an embodiment of the present application is shown, Figure 1 , the method of the present application is described, and the method for visual inspection of the workpiece installation process includes:

[0048] Step S100: Capture video images of the workpiece installation process, extract necessary reference image features and perform positioning.

[0049] The factory is equipped with a video surveillance system, which collects video images of the workpiece installation process and then transmits them back to the computer terminal device. However, for the collected video images, a long part of the image does not contain operators installing the workpiece. If the collected video is directly tested, it will result in the testing of many video clips without action, which wastes a lot of time. Therefore, it is necessary to intercept the surveillance video and extract the video of the workpiece installation process to improve the detection efficiency. However, for a large number of surveillance videos, if they are intercepted manually, it will further increase the workload, so it is necessary to automatically intercept the surveillance video.

[0050] In order to quickly find the characteristics of the installation process from a long-term surveillance video, it is necessary to determine a necessary reference. In the embodiments of the present application, the so-called necessary reference refers to the steps that must be taken or the tools that must be used in the workpiece installation process, and then the image features of the necessary reference are extracted through one or more of the color, contour and texture of the necessary reference; and the image features of the necessary reference are framed and positioned. After locating the position of the necessary reference, the reference point for video capture can be found through the necessary reference, and the surveillance video can be quickly captured based on the reference point, thereby improving the efficiency of detection and positioning of the workpiece installation process.

[0051] In the application example of bolt installation, the wrench is an essential tool for bolt installation, so in this example, the wrench is used as a necessary reference. In the specific operation, the image feature of the wrench is extracted by the wrench color. In this scene, the YCbCr color transformation space method is used to extract the RGB value of the wrench color in the video data and then substitute it into the following transformation matrix:

[0052]

[0053] Among them, Y represents the brightness component of the color, CB and CR are the concentration offset components of blue and red respectively, and R, G, and B represent the three channels of RGB respectively. Traverse each pixel of the video image and perform matrix calculations on the three channels respectively to distinguish the colored parts and frame the color feature area, so as to locate and obtain the necessary reference, as shown in 2. Figure 2 It can be seen that the position of the wrench is located through the color characteristics of the wrench, so the video position where the wrench appears can be obtained.

[0054] In other application scenarios, the color of the necessary reference may be similar to the color of the background, or the size of the necessary reference may be small. In this case, image feature recognition can be performed by one or more of the necessary reference colors, contours, and textures. When recognizing by contour, contour recognition can be performed based on OpenCV; when recognizing by texture, statistical methods, structural methods, transformation-based methods, model-based methods, graph-based methods, learning-based methods, and entropy-based methods can be used. The above methods are all existing technologies and will not be described in detail here.

[0055] Step S200: capturing a workpiece installation process video based on the located necessary reference image features and the time required for the entire workpiece installation process.

[0056] This step mainly uses the necessary reference located in step S100. Specifically, the entire long video recorded by the video surveillance is read, and the start time and end time are set. The video frame where the image feature of the necessary reference is located is used as the starting key frame; then the time required for the entire installation process of the workpiece is determined, and the starting key frame is used as a node to intercept several minutes of long video forward and backward, respectively, and the long video covers the entire installation process of the workpiece. In fact, as long as the long video segment of the entire installation process is intercepted, the demand is met. However, because automatic interception is to be achieved, in order to ensure that the entire installation process is fully covered, assuming that the time required for the entire workpiece installation process is n minutes, it is only necessary to use the starting key frame as a node, intercept n minutes forward, and intercept n minutes backward, then it can be ensured that the video of the complete process is intercepted, and the intercepted video clips are saved in a folder waiting for detection.

[0057] Step S300: extracting features from the captured workpiece installation process video to obtain action features of the workpiece installation process.

[0058] In an embodiment of the present application, the captured workpiece installation process video is firstly used to obtain action boundary information and action behavior characteristics using the background clipping method and the dense trajectory method respectively, and then the action boundary information and action behavior characteristics are fused; finally, the fused features are sent to the feature extraction network for identification to obtain the action characteristics of the workpiece installation process.

[0059] More specifically, combined Figure 3As shown, in the embodiment of the present application, it can be specifically divided into two operation processes. One is to use the background clipping method to obtain the contour features of the foreground from the video of the RGB image of the workpiece installation process video, and then superimpose the contour features of all image frames to obtain the frame difference, and use the difference between the operator and the fixed background to obtain the action boundary information of the operator when performing the process installation, effectively distinguish the interference of the working environment, and facilitate the extraction of the action features of the operator. However, at this time, the static features in the video frame are extracted to obtain the action trajectory of the operator. Therefore, in the embodiment of the present application, another process uses the dense trajectory method to extract the action features of the entire process by tracking the motion trajectory of the worker's action in the working state using the optical flow image of the workpiece installation process video, and uses the optical flow field to obtain the motion trajectory of the dense sampling points of the video frame sequence, and the worker's motion trajectory is sampled in real time to obtain the action behavior characteristics of the operator.

[0060] It should be noted that, for the two processes, the first process may be performed first and then the second process, or the second process may be performed first and then the first process, or the two processes may be performed in parallel.

[0061] After extracting the action boundary information and action behavior features, in an embodiment of the present application, weighted linear fusion is used to obtain the feature descriptor of the video frame and sent to a feature extraction network to extract features, and then the action model of the workpiece installation process can be obtained.

[0062] In the embodiment of the present application, the feature extraction network adopts a C3D network, and the structure of the C3D network is as follows: Figure 4 As shown, the C3D network uses a fully convolutional 3D filter to extract the action features of the workpiece installation process. In the specific operation, a continuous video frame image of size 60*40 is used as input, 5 channel information is extracted for each frame, and a total of 8 convolutions and 4 pooling operations are performed, and output through two fully connected layers. In an embodiment of the present application, the pooling kernel size of the pooling layer Pool5 of the C3D network is set to 7×7, and the step size is set to 7 to cope with videos with too high resolution. Although the pixels are compressed after output, the video content is not damaged to ensure accuracy and readability. Each feature map in the convolution layer will be connected to multiple adjacent continuous frames in the previous layer to capture the behavior information of the operator. Finally, the network obtains the action features of the specific working status of the operator after two fully connected layers.

[0063] When using the C3D network for feature extraction, the network must be trained first. In the embodiment of the present application, the specific method for training the network model includes:

[0064] First, video samples of the workpiece installation process are collected to construct a sample data set. Then, according to the workpiece installation process, the video samples of interest are segmented. During segmentation, if the installation process of a workpiece includes m installation processes, each sample of interest can be segmented into segments. Then, the segmented video segments are annotated and used as training samples for training the feature extraction network; finally, the feature extraction network is trained using the training samples. It should be noted that during training, the training samples also first use the background clipping method and the dense trajectory method to obtain the action boundary information and action behavior features, and then the action boundary information and action behavior features are fused; finally, the fused features are sent to the feature extraction network, and all the training samples are used to complete the training of the feature extraction network.

[0065] It should be noted that in other embodiments of the present application, the feature extraction network may also adopt other convolutional neural network models, such as Resnet, DenseNet, CenterNet, etc.

[0066] In the embodiments of the present application, the background clipping method and the dense trajectory method are respectively used to obtain the action boundary information and the action behavior characteristics, which not only considers the static features in the video, but also considers the dynamic features at the same time, and uses the neural network model for feature extraction, thereby improving the detection accuracy of the workpiece installation process.

[0067] Step S400: classify the action features, and determine whether the workpiece installation process is qualified based on the classification results. Specifically, in an embodiment of the present application, the action features are passed through a Softmax classifier and then optimized through regression. When classifying in the Softmax classifier, IoU (Intersection over Union) is used to define the behavior category, and the score function is used to define the accuracy of the action completion. The candidate video segment is defined as an action when it has the highest IoU with the target behavior and the IoU is greater than 0.5, and is defined as others when it is less than 0.5. The ratio of positive and negative samples is 1:3. Under normal circumstances, the process represented by each action should be completed in sequence and the detection of each process should be carried out continuously. When optimizing through regression, the regression loss function uses a smooth L1 loss function to optimize the relative position offset between the candidate box and the ground truth.

[0068] While using the Softmax classifier for classification, the corresponding score of each operation process can be obtained, and the process can be judged whether it is qualified according to the score of each process and the score threshold of the corresponding process. Specifically, if the score of a process is greater than the score threshold corresponding to the process, then it means that the process is qualified. On the contrary, if the score of the process is less than or equal to the score threshold corresponding to the process, then it means that the process is unqualified. The reason for the failure may be that the worker's action when installing the workpiece does not meet the requirements, resulting in the workpiece not being installed in place, or it may be due to the wrong order of the installation process.

[0069] In other embodiments of the present application, after making a judgment based on the classification result, if the test result is qualified, it is recorded and stored; if the test result is unqualified, an early warning prompt is given to remind the operator to manually re-inspect the corresponding process of the workpiece installation process to ensure that the workpiece installation is qualified.

[0070] For example, in the embodiments of the present application, Figure 5 As shown, for the three processes of bolt installation, the process installation video is identified to complete the process classification. The entire classification and identification process includes judging whether there is an action of rotating a wrench to fix the bolt, whether there is an action of marking after installation, and whether there is an action of painting the installed bolt. After the classification is completed, the judgment is completed by comparing the score of each process with the score threshold of the corresponding process. If the score of the installation process one is greater than the score threshold T1, then the process is qualified, and it is recorded and stored. If the score of the installation process one is less than or equal to the score threshold T1, then an early warning is issued, indicating that the process one needs manual re-inspection during the installation of the workpiece; if the score of the installation process two is greater than the score threshold T2, then the process is qualified, and it is recorded and stored. If the score of the installation process one is less than or equal to the score threshold T2, then an early warning is issued, indicating that the process two needs manual re-inspection during the installation of the workpiece; if the score of the installation process three is greater than the score threshold T3, then the process is qualified, and it is recorded and stored. If the score of the installation process three is less than or equal to the score threshold T3, then an early warning is also issued, indicating that the process three needs manual re-inspection during the installation of the workpiece. Figure 6 A demonstration diagram of detecting an error in the use of a wrench is shown. When an error in the process is detected, an error warning will be issued, suggesting that the operator needs to recheck the process to ensure that the process is qualified.

[0071] Through the method proposed in the embodiment of the present application, the workpiece installation process in the factory can be quickly and accurately detected and positioned in the absence of a large amount of data, and there is no need for a large amount of manpower for visual inspection. Compared with the existing technology, the detection and positioning effects are better.

[0072] Furthermore, based on the proposed method, the present application also provides a device for visual inspection and positioning of key workpiece installation processes. In one embodiment of the present application, the device includes: a video image acquisition module, a reference positioning module, a process video capture module, a feature extraction module, and an evaluation module. Among them, the video image acquisition module is used to collect video images of the workpiece installation process; the reference positioning module is used to extract the image features of the necessary reference and perform positioning; the process video capture module is used to capture the workpiece installation process video according to the image features of the necessary reference located and the time required for the entire installation process of the workpiece; the feature extraction module is used to extract features of the intercepted workpiece installation and obtain the action features of the key video sequence of the workpiece installation process; the evaluation module is used to classify the action features and judge whether the workpiece installation process is qualified according to the classification results. The corresponding functions implemented by each module correspond to the contents of the above method, and will not be repeated here.

[0073] In other embodiments of the present application, the evaluation module includes a classification module and a judgment module, wherein the classification module uses a Softmax classifier to classify action features; and the judgment module judges whether the workpiece installation process is qualified based on the classification result.

[0074] In specific engineering applications, each module can be a computer software module or a hardware module that can execute computer instructions to implement corresponding functions. Using the device proposed in the embodiment of the present application, the workpiece installation process can be effectively detected and located quickly and effectively.

[0075] The above embodiments are only preferred implementation modes of the present invention. It should be pointed out that ordinary technicians in this technical field can make several improvements and equivalent substitutions without departing from the principles of the present invention. These technical solutions after improvements and equivalent substitutions to the claims of the present invention all fall within the protection scope of the present invention.

Claims

1. A method for visual inspection and positioning of key workpiece installation processes, characterized in that: include: Collect video images of the workpiece installation process, extract image features of necessary references and locate them; specifically, take necessary steps or necessary tools in the workpiece installation process as necessary references, extract image features of necessary references through one or more of the color, contour and texture of the necessary references; frame and locate the image features of the necessary references; According to the located necessary reference image features and the time required for the entire installation process of the workpiece, a video of the workpiece installation process is intercepted; specifically, the video frame where the located necessary reference image features are located is used as the starting key frame; the time required for the entire installation process of the workpiece is determined; with the starting key frame as a node, several minutes of long videos are intercepted forward and backward respectively, and the long video covers the entire installation process of the workpiece; The captured workpiece installation process video is respectively subjected to background clipping and dense trajectory methods to obtain action boundary information and action behavior characteristics; specifically, the RGB image of the workpiece installation process video is subjected to background clipping method to obtain the contour features of the foreground from the video, and then the contour features of all image frames are superimposed to obtain the frame difference, and the difference between the operator and the fixed background is used to obtain the action boundary information and the action trajectory of the operator during the process installation; the optical flow image of the workpiece installation process video is subjected to dense trajectory method to extract the action characteristics of the entire process by tracking the motion trajectory of the worker's action in the working state, and the motion trajectory of the dense sampling points of the video frame sequence is obtained by using the optical flow field, and the worker's motion trajectory is sampled in real time to obtain the action behavior characteristics of the operator; The action boundary information and action behavior features are sent to a feature extraction network for feature extraction fusion to obtain action features of the workpiece installation process; the method of extracting features from the intercepted workpiece installation process video to obtain action features of a key video sequence of the workpiece installation process also includes: training the feature extraction network; the training method includes: collecting video samples of the workpiece installation process; segmenting the video samples of interest according to the workpiece installation process; annotating the segmented video segments and using them as training samples for training the feature extraction network; and training the feature extraction network using the training samples; Classifying the action features, wherein the method for classifying the action features comprises: classifying the action features using a Softmax classifier, and performing regression optimization; when optimizing by regression, the regression loss function uses a smooth L1 loss function to optimize the relative position offset between the candidate box and the ground truth; Determine whether the workpiece installation process is qualified based on the classification results; specifically, obtain the score of each operation process through the Softmax classifier; determine whether the process is qualified based on the score of each process and the score threshold of the corresponding process; after judging based on the classification results, if qualified, the video of the completed inspection will be archived; if unqualified, an early warning will be issued, and a prompt will be given to re-inspect the corresponding process.

2. The method according to claim 1, characterized in that The feature extraction network is a C3D network, and the pooling kernel size of the pooling layer Pool5 in the C3D network is 7×7 and the step size is 7.

Images