Visual identification method and system for part assembly process

Abstract

The invention relates to the technical field of part assembly detection, in particular to a visual identification method and system for a part assembly process, and the method comprises the following steps: obtaining a video stream in real time, carrying out the decoding preprocessing, and generating continuous frame images; identifying a hand region and an operation button region in the frame image, and judging the process starting condition by calculating the intersection-union ratio of the hand region and the operation button region; performing part state acquisition on the frame image based on the classification model, identifying the part state through a multi-frame joint judgment method, comparing the part state with a preset part target state, and judging a process expected condition of each part state; when the process meets the expectation, a target detection algorithm is combined with a multi-frame joint technology to carry out remnant identification; and when the expected condition of the working procedure is not consistent or the remnants are identified, outputting an alarm. The method has the advantages of being low in use cost and high in universality and adaptability.

Description

Technical Field

[0001] The present application relates to the technical field of parts assembly detection, and in particular to a visual recognition method and system for parts assembly processes. Background Art

[0002] In the machinery industry, especially during the installation of critical parts such as automotive components, not only do multiple parts need to be installed, but operations such as button controls are also involved. Traditional quality control methods face many challenges, especially in monitoring process quality. While widely used, traditional manual visual inspection methods suffer from several significant issues: First, manual visual inspection is inefficient, especially during production, as operators often need to monitor each process for extended periods, which can lead to fatigue and compromise judgment and accuracy.

[0003] Secondly, due to the limitations of operators' experience and skill levels, oversights and misjudgments can easily occur, preventing quality issues from being discovered and corrected in a timely manner. Furthermore, manual visual inspections are highly subjective, and different inspectors may have different understandings of the quality standards for the same process, leading to inconsistent inspection results. Finally, the manual inspection process is cumbersome, not only consuming significant time and manpower, but also lacking real-time feedback, resulting in reduced production efficiency.

[0004] Therefore, intelligent monitoring and inspection methods are particularly urgent. Intelligent systems can promptly detect any deviations, issue warnings, and guide operators to make corrections. Unlike traditional manual visual inspections, AI systems are unaffected by fatigue, emotion, or experience. They can monitor every process 24 hours a day, efficiently, and effectively to ensure compliance with operational standards.

[0005] Artificial intelligence technology can also continuously optimize detection algorithms based on historical data, improving detection accuracy and comprehensiveness through continuous learning, thereby avoiding the subjective errors and omissions that occur in manual visual inspections. Furthermore, AI monitoring systems can provide real-time data feedback to production line managers, enabling them to quickly implement corrective measures and avoid batch quality issues. This significantly improves production efficiency and reduces rework and repair costs. Therefore, AI-based intelligent monitoring systems are not only an inevitable choice for improving quality control capabilities but also a key trend in the development of intelligent and sophisticated manufacturing in modern manufacturing.

[0006] Existing intelligent monitoring and inspection methods primarily capture video images of the workpiece installation process, then employ background subtraction and dense trajectory methods to obtain motion boundary information and behavioral features. These information and behavioral features are then fed into a feature extraction network for feature extraction and fusion, resulting in motion signatures for the workpiece installation process. These signatures are then classified, and the final classification results are used to determine if the workpiece installation process is qualified. The training and annotation costs of background subtraction and dense trajectory methods are inherently high, and the diverse nature of installation processes further increases these costs.

[0007] In addition, its feature expression relies on manual rules rather than data-driven. When the workpiece type or installation process changes, the existing features may not be directly reused, and the data needs to be re-labeled to adapt to the new scenario, which makes it difficult to promote. Summary of the Invention

[0008] In order to solve the above problems, the present application provides a visual recognition method and system for a parts assembly process, which has the advantages of low cost, versatility and strong adaptability.

[0009] On the one hand, the present application provides a visual recognition method for a parts assembly process, which adopts the following technical solution:

[0010] A visual recognition method for a parts assembly process comprises the following steps:

[0011] Acquire video stream in real time, perform decoding preprocessing, and generate continuous frame images;

[0012] Identify the hand area and the operation button area in the frame image, and determine the process start status by calculating the intersection-and-union ratio of the hand area and the operation button area;

[0013] Based on the classification model, the part status is acquired from the frame image, and the part status is identified through a multi-frame joint judgment method. It is then compared with the preset part target status to determine the expected process status of each part status.

[0014] When the process is as expected, the target detection algorithm combined with multi-frame joint technology is used to identify the remains;

[0015] When the process does not meet expectations or residues are identified, an alarm is output.

[0016] In one embodiment, the step of identifying the hand area and the operation button area in the frame image and determining the process start status by calculating the intersection-over-union ratio of the hand area and the operation button area specifically includes:

[0017] Use the deep learning target detection algorithm YOLOV5 to identify the areas of the worker's hands in the frame image and generate the hand areas of each hand;

[0018] Get the two preset operation button areas and calculate the intersection-and-union ratio between the two hand areas and the operation button area in each frame image one by one. The intersection-and-union ratio calculation formula is as follows:

[0019]

[0020] Where B hand Boxes representing hand detection, B buntton Boxes representing action buttons;

[0021] When both IoUs in the same frame image are greater than the threshold, the determination process starts.

[0022] In one embodiment, the step of acquiring the part status of the frame image based on the classification model specifically includes:

[0023] Get all parts placement areas in the frame image;

[0024] After cropping and scaling all part placement areas, the corresponding part area images are obtained;

[0025] Use the trained classification model to identify the status of all part area images and obtain the corresponding part status;

[0026] Add each part state to the corresponding state sequence S seq middle;

[0027] Among them, the classification model adopts the ShuffleNetV2 classification algorithm as the neural network, adds an ECA attention module after the conv1 layer of the neural network, and selects the cross entropy loss function as its loss function.

[0028] In one embodiment, the step of identifying the part status by the multi-frame joint judgment method specifically includes:

[0029] Determine the state sequence S seq Whether the state sequence length reaches the set frame number N;

[0030] If so, calculate the state sequence S seq The most frequent state label S mode ;

[0031] Judgment status label S mode Whether the frequency of occurrence exceeds the threshold θ;

[0032] If exceeded, update the part status to S mode ; If not, keep the current state unchanged.

[0033] In one embodiment, the step of comparing with the preset target state of the part to determine the expected process status of each part state specifically includes:

[0034] After updating, the part status S mode Input the preset part state model, based on the updated part state S mode Compare the current preset part target state with the preset part state model to determine the updated part state S mode Whether it meets the preset process requirements;

[0035] If it meets the requirements, it means that the current process meets expectations, then the next preset part target state is loaded into the preset part state model, and the next part state of the frame image is obtained based on the classification model.

[0036] In one embodiment: the part status includes part information, process number and status information, the status information includes normal status and abnormal status, the normal status includes existence, non-existence and assembly, and the abnormal status includes damage, obstruction and others;

[0037] If the status information of the updated part status is an abnormal status, it is directly determined that it does not meet the preset process requirements.

[0038] In one embodiment, the preset part status model is generated based on a standard operating procedure (SOP).

[0039] In one embodiment, when the process is as expected, the steps of using a target detection algorithm combined with a multi-frame joint technology to identify the remains specifically include:

[0040] When the process meets expectations, determine the corresponding updated part status S mode Whether it does not exist;

[0041] If it does not exist, the target detection algorithm YOLOV5 is used to identify all frame images before the next process meets expectations, and detect whether there are easy-to-miss parts outside the part placement area to obtain the missing status;

[0042] When the omission status of the consecutive M frame images is yes, it is determined that there are leftovers.

[0043] In one embodiment, when the process is as expected, the steps of using a target detection algorithm combined with a multi-frame joint technology to identify the remains specifically include:

[0044] When all processes meet expectations, the target detection algorithm YOLOV5 is used to identify all frame images to detect whether there are parts that are easy to miss outside the part placement area and obtain the missing status;

[0045] When the omission status of the consecutive M frame images is yes, it is determined that there are leftovers.

[0046] On the other hand, the present application provides a visual recognition system for a parts assembly process, which adopts the following technical solution:

[0047] A visual recognition system for a parts assembly process, comprising:

[0048] Video data decoding module: acquires video stream in real time and performs decoding preprocessing to generate continuous frame images;

[0049] Process start detection module: identifies the hand area and operation button area in the frame image, and determines the process start status by calculating the intersection and union ratio of the hand area and the operation button area;

[0050] Process production comparison module: Based on the classification model, the part status is acquired from the frame image, the part status is identified through a multi-frame joint judgment method, and compared with the preset part target status to determine the expected process status of each part status;

[0051] Parts Lost Detection Module: When the process is as expected, the target detection algorithm combined with multi-frame joint technology is used to identify leftover parts;

[0052] Central processing module: Outputs an alarm when the process does not meet expectations or when leftovers are identified.

[0053] In summary, this application has the following beneficial effects:

[0054] 1. This invention separates the assembly process from the operating process. It uses computer vision technology to locate the process area, extract features from the parts placement area, and use an efficient classification algorithm combined with multi-frame joint judgment to detect the part status. The expected status of the assembly process is then quickly determined based on the part status. The operating process is monitored by detecting the action of pressing the start switch with both hands through an object detection algorithm, enabling fast and efficient monitoring.

[0055] 2. The present invention only needs to train a large number of labeled images through the classification model to optimize the recognition ability of the part area, and combine the multi-frame joint judgment technology to improve the classification accuracy, reduce the requirements for model computing power and accuracy, so that the overall training cost is low, and has good scalability. It can be flexibly applied in different production lines and workpiece types, thus having strong promotion significance and a broader market application prospect. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 is a flow chart of the visual recognition method in this embodiment;

[0057] Figure 2This is a flow chart of the process start detection module in this embodiment;

[0058] Figure 3 This is a flow chart of the process production comparison module in this embodiment;

[0059] Figure 4 Schematic diagram of the optimized ShuffleNetV2 network structure in this embodiment. DETAILED DESCRIPTION

[0060] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. However, it should be understood by those skilled in the art that the present application can be implemented without these details. In some cases, in order to avoid unnecessary descriptions that make various aspects of the present application obscure, the well-known methods, processes, systems, components and / or circuits that have been described at a higher level will not be described in detail. It is obvious to those skilled in the art that various changes can be made to the embodiments disclosed in the present application, and the general principles defined in the present application can be applied to other embodiments and application scenarios without departing from the principles and scope of the present application. Therefore, the present application is not limited to the embodiments shown, but conforms to the broadest scope consistent with the scope claimed for protection in the present application.

[0061] It should be noted that the description of these embodiments is used to help understand the present invention, but does not constitute a limitation of the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0062] In the description of this application, "several" means one or more, "many" means more than two, "greater than," "less than," and "exceed" are understood to exclude the number itself, while "above," "below," and "within" are understood to include the number itself. The use of "first" and "second" in the description is solely for the purpose of distinguishing technical features and should not be construed as indicating or implying relative importance, implicitly specifying the number of the indicated technical features, or implicitly specifying the order of the indicated technical features.

[0063] In the description of this application, reference to the terms "one embodiment," "some embodiments," "illustrative embodiments," "examples," "specific examples," or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of this application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any combination in one or more embodiments or examples.

[0064] A visual recognition method for parts assembly process, such as Figure 1 As shown, the following steps are included:

[0065] S100: Acquire video stream in real time, perform decoding preprocessing, and generate continuous frame images.

[0066] In this step, the video stream can be provided by a camera or other equipment on the production line, wherein real-time acquisition can be achieved when a camera is used.

[0067] After obtaining the video stream, we use open-source video decoding libraries such as FFmpeg, OpenCV, and PyAV to decode the video stream and convert it into continuous frame images. We also pre-process each frame through methods such as denoising, grayscale conversion, and image scaling to improve the accuracy of subsequent recognition.

[0068] S200 , identifying the hand area and the operation button area in the frame image, and determining the process start status by calculating the intersection-and-union ratio of the hand area and the operation button area.

[0069] The flowchart of the process start detection module is as follows Figure 2 Specifically, the deep learning object detection algorithm YOLOV5 is used to identify the areas of the worker's hands in the frame image and generate hand regions for each hand. In this embodiment, all detection and recognition areas are in the form of a bounding box B(x, y, w, h), where (x, y) is the coordinate of the bounding box reference point, which can be any of the four vertices, and w and h represent the width and height of the bounding box, respectively.

[0070] In addition, it should be noted that during the operation, if one-handed judgment is used, false touches are likely to occur. Therefore, in order to improve the accuracy of subsequent judgments, it is necessary to identify both hands at the same time.

[0071] At the same time, two preset operation button areas are obtained. The operation button area is usually a fixed area, so it is preferred to use a preset method to reduce the recognition pressure of the algorithm. However, it should be noted that the operation button area can be the area where the actual operation button is located. However, if there is only one operation button, it can also be a set virtual position used only to avoid misjudgment. In addition, when there are two corresponding operation buttons on the workbench, the deep learning target detection algorithm YOLOV5 is also used to identify the operation button area in the frame image.

[0072] Then, the intersection-over-union ratio (IoU) between the two hand regions and the operation button region in each frame is calculated to determine whether the worker has pressed the button. The IoU calculation formula is as follows:

[0073]

[0074] Where Bhand represents the boxes for hand detection, and Bbuntton represents the boxes for operation buttons.

[0075] When both IoU ratios in the same frame are greater than the threshold, the button is considered pressed and the process is initiated. If only one of the IoU ratios is greater than the threshold, or both IoU ratios are less than the threshold, the process has not started and the system will continue to monitor the video frames.

[0076] In addition, it should be noted that the operation of pressing the operation button is not necessarily the first process in the entire parts assembly process, so the purpose of determining the process start is only to determine whether the process of pressing the operation button is carried out, and to be used as a start for subsequent visual recognition.

[0077] S300, acquiring the part status of the frame image based on the classification model, identifying the part status through a multi-frame joint judgment method, and comparing it with the preset part target status to determine the process expectation of each part status.

[0078] The flowchart of the process production comparison module is as follows Figure 3 As shown, the steps of acquiring the part status of the frame image based on the classification model specifically include:

[0079] S301: Acquire all parts placement areas in the frame image.

[0080] In this step, the parts placement area is preferably preset, that is, the placement positions of each part are pre-set on the workbench. This is especially true for applications with a large number of parts or where the differences between the parts are small, such as multi-gear assembly structures. For applications where the placement of parts is difficult to fix or where there are significant differences between the parts, frame image recognition can also be used to locate the placement areas of each part.

[0081] S302. After clipping and scaling all parts placement areas, corresponding parts area images are obtained:

[0082] R k =crop(B k ,I t )

[0083] R′ k =resize(R k ,S)

[0084] Among them, crop(B k ,I t ) means cutting out the part area from the image according to the bounding box, resize(R k,S) means scaling the cropped area to size S. After cropping and scaling, the background or other irrelevant objects are prevented from interfering with the recognition results.

[0085] S303: Use the trained classification model to perform state recognition on all part area images to obtain corresponding part states.

[0086] In this step, different classification models can be selected based on the requirements for part state recognition. For example, for subtle differences in part states, MobileNetV2 can be selected, as it has high-dimensional feature representation capabilities and ensures accurate state recognition. If the number of parts is large and technical resources are limited, ShuffleNet can be selected for its fast speed and low power consumption.

[0087] In one embodiment, the classification model selects ShuffleNetV2 as the baseline model for optimization to improve its recognition accuracy. The improved ShuffleNetV2 network structure diagram is shown in the attached figure. Figure 4 The ECA (Efficient Channel Attention) attention module is added after the conv1 layer of ShuffleNetV2, and the cross entropy loss function is selected as its loss function.

[0088] The ECA module adaptively assigns different weights to each channel, enhancing the expression of important features and suppressing irrelevant or noisy features. This mechanism can significantly improve the model's recognition ability in images with complex backgrounds. Especially when processing images with a large amount of noise, the ECA module can effectively extract useful features, thereby improving the model's classification accuracy.

[0089] The calculation process of ECA is as follows:

[0090] (1) Perform global average pooling on the input feature map X (of shape C×H×W, where C is the number of channels, H and W are the height and width of the feature map) in the channel dimension to obtain a vector of shape C×1.

[0091] (2) Apply a one-dimensional convolution operation to this vector. The size of the convolution kernel k is adaptive and is usually calculated by the following formula:

[0092]

[0093] Where ceil is the activation function, C is the number of channels, and int is the rounding operation. This convolution operation can be regarded as a fully connected layer to capture the correlation between channels.

[0094] (3) The output of the convolution is mapped to between 0 and 1 through the sigmoid function to obtain the attention weight of each channel. These weights are then multiplied with the original feature map in the channel dimension to obtain the adjusted feature map.

[0095] The cross entropy loss function is selected as the loss function to constrain the classification results. The loss function evaluates the performance of the model by measuring the difference between the predicted category distribution and the actual category distribution. The specific calculation formula is shown below:

[0096]

[0097] Where: N is the number of categories. i Is the true label, if the sample belongs to the i-th category, then y i =1, otherwise y i =0. p i is the probability that the model predicts that the sample belongs to the i-th class.

[0098] In this step, the part status includes part information, process number and status information. The status information includes normal status and abnormal status. The normal status includes existence, non-existence, assembly, etc., and the abnormal status includes damage, obstruction and others.

[0099] S304. Add each part state to the corresponding state sequence Sseq.

[0100] The steps of identifying the part status by the multi-frame joint judgment method specifically include:

[0101] S311 , determining whether the state sequence length of the state sequence Sseq reaches the set number of frames N.

[0102] S312: If yes, calculate the most frequent state label Smode in the state sequence Sseq.

[0103] S313: Determine whether the occurrence frequency of the state label Smode exceeds a threshold θ.

[0104] S314. If it exceeds, the state is considered stable and the part state is updated to Smode; if it does not exceed, the current state remains unchanged.

[0105] The steps for comparing the expected process status of each part state with the preset target state include:

[0106] S321. If the status information of the updated part status is abnormal, it is directly determined to be not in compliance with the preset process requirements.

[0107] S322. If the status information of the updated part status is normal, the updated part status Smode is input into the preset part status model. Based on the comparison between the updated part status Smode and the current preset part target status in the preset part status model, it is determined whether the updated part status Smode meets the preset process requirements.

[0108] The preset part status model is generated based on the standard operating procedure (SOP). In addition, during the generation process, the process steps related to button operations are removed.

[0109] S323. If it meets the requirements, it means that the current process meets expectations, then the next preset part target state is loaded into the preset part state model, and the next part state is acquired from the frame image based on the classification model.

[0110] S400: When the process is as expected, the target detection algorithm is combined with the multi-frame joint technology to identify the remains.

[0111] In this step, debris primarily refers to parts like gaskets and bolts that are easily dropped during the movement of the main part. This is done to prevent incomplete installation due to the parts falling out when being removed from the parts area for assembly. The debris identification step can be performed at different points depending on the situation. For timely identification, debris identification can be performed each time a process is determined to be meeting expectations, or it can be performed directly after all processes have met expectations.

[0112] In one embodiment, the residue identification is performed each time the process is determined to be in compliance with expectations. The specific steps of the residue identification method include:

[0113] S401. When the process meets expectations, determine whether the corresponding updated part state Smode does not exist.

[0114] Relic identification is mainly performed when the part is removed from the parts area for assembly, that is, when the part status changes from the placed status to the disappeared status, which corresponds to the part non-existent status.

[0115] S402. If it does not exist, the target detection algorithm YOLOV5 is used to identify all frame images before the next process meets expectations, and detect whether there are easy-to-miss parts outside the part placement area to obtain the missing status.

[0116] On the one hand, this setting can minimize the workload of residual object identification, as it only needs to identify all frames before the next process meets expectations. On the other hand, it can be performed after each process is determined to meet expectations, which can promptly detect problems and issue alarms, allowing operators to make timely adjustments to the assembly.

[0117] S403: When the missing status of the consecutive M frame images is yes, it is determined that there are leftovers.

[0118] In this step, when an object is detected as left behind, the corresponding frame image is output, indicating the specific location and time of the object being left behind. Multi-frame joint technology improves recognition accuracy, preventing false alarms in situations where an object is left behind but the operator promptly discovers and corrects the situation.

[0119] In another embodiment, the residue identification is performed after all processes are in compliance with expectations. The specific steps of the residue identification method include:

[0120] S411. When all processes meet expectations, the target detection algorithm YOLOV5 is used to identify all frame images, detect whether there are parts that are easy to miss outside the part placement area, and obtain the missing status.

[0121] S421: When the missing status of the consecutive M frame images is yes, it is determined that there are leftovers.

[0122] S500: When the process does not meet the expected situation or residual objects are identified, an alarm is output.

[0123] In one embodiment, the present application provides a visual recognition system for a parts assembly process, comprising:

[0124] Video data decoding module: acquires video stream in real time and performs decoding preprocessing to generate continuous frame images;

[0125] Process start detection module: identifies the hand area and operation button area in the frame image, and determines the process start status by calculating the intersection and union ratio of the hand area and the operation button area;

[0126] Process production comparison module: Based on the classification model, the part status is acquired from the frame image, the part status is identified through a multi-frame joint judgment method, and compared with the preset part target status to determine the expected process status of each part status;

[0127] Parts Lost Detection Module: When the process is as expected, the target detection algorithm combined with multi-frame joint technology is used to identify leftover parts;

[0128] Central processing module: Outputs an alarm when the process does not meet expectations or when leftovers are identified.

[0129] The above description is merely a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Although the present invention has been disclosed as a preferred embodiment as above, it is not intended to limit the present invention. Any person skilled in the art can make some changes or modifications to equivalent embodiments using the technical contents disclosed above without departing from the scope of the technical solution of the present invention. However, any brief modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. A visual recognition method for a parts assembly process, characterized in that: The following steps are involved: Acquire video stream in real time, perform decoding preprocessing, and generate continuous frame images; Identify the hand area and the operation button area in the frame image, and determine the process start status by calculating the intersection-and-union ratio of the hand area and the operation button area; Based on the classification model, the part status is acquired from the frame image, and the part status is identified through a multi-frame joint judgment method. It is then compared with the preset part target status to determine the expected process status of each part status. When the process is as expected, the target detection algorithm combined with multi-frame joint technology is used to identify the remains; When the process does not meet expectations or residues are identified, an alarm is output.

2. The visual recognition method for parts assembly process according to claim 1, characterized in that: The step of identifying the hand area and the operation button area in the frame image and determining the process start status by calculating the intersection-and-union ratio of the hand area and the operation button area specifically includes: Use the deep learning target detection algorithm YOLOV5 to identify the areas of the worker's hands in the frame image and generate the hand areas of each hand; Get the two preset operation button areas and calculate the intersection-and-union ratio between the two hand areas and the operation button area in each frame image one by one. The intersection-and-union ratio calculation formula is as follows: Where B hand Boxes representing hand detection, B buntton Boxes representing action buttons; When both IoUs in the same frame image are greater than the threshold, the determination process starts.

3. The visual recognition method for parts assembly process according to claim 1, characterized in that: The step of acquiring the part status of the frame image based on the classification model specifically includes: Get all parts placement areas in the frame image; After cropping and scaling all part placement areas, the corresponding part area images are obtained; Use the trained classification model to identify the status of all part area images and obtain the corresponding part status; Add each part state to the corresponding state sequence S seq middle; Among them, the classification model adopts the ShuffleNetV2 classification algorithm as the neural network, adds an ECA attention module after the conv1 layer of the neural network, and selects the cross entropy loss function as its loss function.

4. The visual recognition method for parts assembly process according to claim 3, characterized in that: The step of identifying the part status by the multi-frame joint judgment method specifically includes: Determine the state sequence S seq Whether the state sequence length reaches the set frame number N; If so, calculate the state sequence S seq The most frequent state label S mode ; Judgment status label S mode Whether the frequency of occurrence exceeds the threshold θ; If exceeded, update the part status to S mode ; If not, keep the current state unchanged.

5. The visual recognition method for parts assembly process according to claim 4, characterized in that: The step of comparing with the preset target state of the part to determine the expected process status of each part state specifically includes: After updating, the part status S mode Input the preset part state model, based on the updated part state S mode Compare the current preset part target state with the preset part state model to determine the updated part state S mode Whether it meets the preset process requirements; If it meets the requirements, it means that the current process meets expectations, then the next preset part target state is loaded into the preset part state model, and the next part state of the frame image is obtained based on the classification model.

6. The visual recognition method for parts assembly process according to claim 5, characterized in that: The part status includes part information, process number and status information, and the status information includes normal status and abnormal status. The normal status includes existence, non-existence and assembly, and the abnormal status includes damage, obstruction and others. If the status information of the updated part status is an abnormal status, it is directly determined that it does not meet the preset process requirements.

7. The visual recognition method for parts assembly process according to claim 5 or 6, characterized in that: The preset part status model is generated based on the standard operating procedure (SOP).

8. The visual recognition method for parts assembly process according to claim 1, characterized in that: When the process is as expected, the steps of using the target detection algorithm combined with the multi-frame joint technology to identify the remains specifically include: When the process meets expectations, determine the corresponding updated part status S mode Whether it does not exist; If it does not exist, the target detection algorithm YOLOV5 is used to identify all frame images before the next process meets expectations, and detect whether there are easy-to-miss parts outside the part placement area to obtain the missing status; When the omission status of the consecutive M frame images is yes, it is determined that there are leftovers.

9. The visual recognition method for parts assembly process according to claim 1, characterized in that: When the process is as expected, the steps of using the target detection algorithm combined with the multi-frame joint technology to identify the remains specifically include: When all processes meet expectations, the target detection algorithm YOLOV5 is used to identify all frame images to detect whether there are parts that are easy to miss outside the part placement area and obtain the missing status; When the omission status of the consecutive M frame images is yes, it is determined that there are leftovers.

10. A visual recognition system for parts assembly process, characterized in that: include: Video data decoding module: acquires video stream in real time and performs decoding preprocessing to generate continuous frame images; Process start detection module: identifies the hand area and operation button area in the frame image, and determines the process start status by calculating the intersection and union ratio of the hand area and the operation button area; Process production comparison module: Based on the classification model, the part status is acquired from the frame image, the part status is identified through a multi-frame joint judgment method, and compared with the preset part target status to determine the expected process status of each part status; Parts Lost Detection Module: When the process is as expected, the target detection algorithm combined with multi-frame joint technology is used to identify leftover parts; Central processing module: Outputs an alarm when the process does not meet expectations or when leftovers are identified.

Images