A screw nut assembly state recognition method and system based on machine vision

Abstract

The application provides a screw nut assembly state recognition method and system based on machine vision, belonging to the technical field of machine vision recognition. The method comprises: adjusting the detection parameter and confidence threshold of the visual recognition model based on the model of the workpiece to be assembled, obtaining the target detection parameter and target confidence threshold; collecting the original image of the screw nut assembly area and preprocessing, extracting the assembly body area image, inputting the visual recognition model, and obtaining the preliminary assembly state recognition result and confidence score; if the confidence score is greater than or equal to the target confidence threshold and is not a pre-defined fuzzy state, the final assembly state result is obtained; if the confidence score is less than the target confidence threshold or the preliminary assembly state result is a fuzzy state, time sequence auxiliary data is obtained, and auxiliary judgment information is generated; the preliminary assembly state recognition result, the confidence score and the auxiliary judgment information are input into the decision model to obtain the final assembly state result, and then alarm information and control instructions are generated.

Description

Technical Field

[0001] This application relates to the field of machine vision recognition technology, and in particular to a method and system for recognizing the assembly status of screws and nuts based on machine vision. Background Technology

[0002] Current methods for identifying the assembly status of screws and nuts rely on a single machine vision inspection and recognition approach. This method determines the status by analyzing the appearance features of the assembled parts through image acquisition. However, it is susceptible to environmental factors such as lighting and lens distortion. Furthermore, relying solely on static image features results in insufficient accuracy in determining critically ambiguous assembly states, leading to low confidence levels. Additionally, the visual model parameters and confidence thresholds of existing methods are fixed values, failing to adapt to the dimensional tolerances of different product models, resulting in poor adaptability. Existing auxiliary inspection data is not effectively integrated with visual features, failing to compensate for the limitations of single-vision inspection and recognition. This leads to unreliable inspection results, including false positives and false negatives, and ultimately fails to meet the automated identification needs of industry.

[0003] Therefore, there is an urgent need for a machine vision-based method and system for recognizing the assembly status of screws and nuts. Summary of the Invention

[0004] To address the aforementioned technical problems, this application provides a machine vision-based method and system for recognizing the assembly status of screws and nuts.

[0005] A first aspect of this application provides a machine vision-based method for recognizing the assembly state of a screw and nut, comprising: S101: Based on the product model information of the workpiece to be assembled, adjust the detection parameters of the preset visual recognition model and the preset confidence threshold to obtain the target detection parameters and the target confidence threshold. S102: Acquire the original image of the screw and nut assembly area to be inspected and preprocess it to locate and extract the assembly area image; S103: Input the image of the assembly area into a preset visual recognition model for analysis to obtain preliminary assembly status recognition results and corresponding confidence scores; S104: If the confidence score is greater than or equal to the target confidence threshold and the preliminary assembly state identification result is not a predefined fuzzy state, then the preliminary assembly state identification result shall be taken as the final assembly state result. S105: If the confidence score is less than the target confidence threshold or the preliminary assembly state identification result is a predefined fuzzy state, then obtain the time-series auxiliary data associated with the current assembly process, and generate assembly process auxiliary judgment information based on the time-series auxiliary data. S106: Input the preliminary assembly state identification result, the confidence score and the assembly process auxiliary judgment information into a preset multi-feature decision model to generate the final assembly state result; S107: Based on the final assembly status result, generate corresponding alarm information and control instructions.

[0006] A second aspect of this application provides a machine vision-based screw and nut assembly status recognition system, comprising: The data adjustment module 21 is used to adjust the detection parameters and preset confidence threshold of the preset visual recognition model based on the product model information of the workpiece to be assembled, so as to obtain the target detection parameters and the target confidence threshold. The acquisition and extraction module 22 is used to acquire the original image of the screw and nut assembly area to be inspected and perform preprocessing, and to locate and extract the image of the assembly area; The preliminary identification module 23 is used to input the image of the assembly area into a preset visual recognition model for analysis, and obtain the preliminary assembly state identification result and the corresponding confidence score. The dual judgment module 24 is used to take the preliminary assembly state recognition result as the final assembly state result if the confidence score is greater than or equal to the target confidence threshold and the preliminary assembly state recognition result is not a predefined fuzzy state. The auxiliary judgment module 25 is used to obtain time-series auxiliary data associated with the current assembly process if the confidence score is less than the target confidence threshold or the preliminary assembly state recognition result is a predefined fuzzy state, and to generate assembly process auxiliary judgment information based on the time-series auxiliary data. The fusion judgment module 26 is used to input the preliminary assembly state identification result, the confidence score and the assembly process auxiliary judgment information into a preset multi-feature decision model to generate the final assembly state result; The alarm control module 27 is used to generate corresponding alarm information and control instructions based on the final assembly status result.

[0007] A third aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the steps of the above-described machine vision-based screw and nut assembly state recognition method.

[0008] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described machine vision-based screw and nut assembly state recognition method.

[0009] The beneficial effects of the machine vision-based screw and nut assembly state recognition method and system provided in this application are as follows: This application achieves screw and nut assembly state recognition based on machine vision. First, the visual model detection parameters and confidence threshold are adaptively adjusted according to the workpiece model to meet the recognition requirements of different models, thereby improving the adaptability of detection and recognition. Images are acquired by dual cameras and preprocessed to extract the assembly area. A semantic segmentation model is used to achieve accurate preliminary judgment of the assembly state. The reliability of the preliminary judgment result is ensured by dual screening of confidence and fuzzy state. For results with doubts, torque-angle and audio time-series auxiliary data are integrated to generate auxiliary judgment information. Then, the final result is obtained through multi-feature decision model fusion analysis, which improves the accuracy and precision of assembly state recognition, reduces false judgments and missed judgments, and finally generates alarms and control commands, thereby improving the intelligence level and efficiency of assembly detection and recognition. Attached Figure Description

[0010] Figure 1 A flowchart illustrating a machine vision-based screw and nut assembly state recognition method provided in an embodiment of this application; Figure 2 A structural block diagram of a machine vision-based screw and nut assembly status recognition system provided in an embodiment of this application; Figure 3 This is a schematic block diagram of an electronic device provided in an embodiment of this application.

[0011] Explanation of reference numerals in the attached figures: 20. Machine vision-based screw and nut assembly status recognition system; 21. Data adjustment module; 22. Data acquisition and extraction module; 23. Preliminary recognition module; 24. Dual judgment module; 25. Auxiliary judgment module; 26. Fusion judgment module; 27. Alarm control module; 300. Electronic device; 301. Processor; 302. Input device; 303. Output device; 304. Memory; 305. Communication bus. Detailed Implementation

[0012] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0013] To make the purpose, technical solution, and advantages of this application clearer, the following will be described in conjunction with the appendix. Figure 1-3 The following is an explanation using specific examples.

[0014] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating a machine vision-based screw and nut assembly state recognition method according to an embodiment of this application. The method includes: S101: Based on the product model information of the workpiece to be assembled, adjust the detection parameters of the preset visual recognition model and the preset confidence threshold to obtain the target detection parameters and the target confidence threshold.

[0015] In this embodiment, the product model information of the workpiece to be assembled is a set of specification parameters characterizing the screw and nut assembly to be assembled, including screw diameter, pitch, nut thickness, assembly standard tolerance range, and appropriate assembly station number. This information serves as the basis for adjusting parameters and confidence thresholds; different workpiece models correspond to different inspection standards. The preset visual recognition model is a pre-trained algorithm model used to identify the assembly state of the screw and nut. In this embodiment, an improved semantic segmentation neural network model, namely the improved U-Net, is used. It possesses image segmentation, contour extraction, and gap calculation functions, serving as a tool for preliminary identification of the assembly state. The detection parameters directly affect the recognition accuracy.

[0016] Specifically, the improved U-Net semantic segmentation model adapted for screw and nut assembly detection in this application is based on the classic U-Net encoder-decoder symmetric architecture, incorporating a channel attention module and a sub-pixel edge enhancement layer. The overall structure consists of six layers: input layer, encoder module, bottleneck layer, decoder module, attention fusion layer, and output layer. The encoder module includes four downsampling units, each consisting of two 3×3 convolutional layers + BN layer + ReLU activation function, downsampling is achieved using MaxPool2d, progressively extracting shallow texture features from the assembly image to deep semantic features. The bottleneck layer consists of two sets of 3×3 convolutions + Dropout, analyzing the distinguishing features of the screw, nut, and assembly reference surface. The decoder module... For each of the four upsampling units, the feature map size is restored by transposed convolution, and a skip connection is made with the feature map of the same scale as the encoder to supplement edge detail features. After the channel attention module is embedded in each decoder unit, the feature channel weights are adaptively allocated to enhance the feature signals of the detection areas such as the lower edge of the nut and the upper edge of the assembly reference surface. The subpixel edge enhancement layer is placed before the output layer, and the feature map is refined at the pixel level by 1×1 convolution to improve the edge accuracy of the segmentation mask. The output layer is a 1×1 convolutional layer, which outputs pixel-level three-class segmentation results according to the Softmax activation function, namely the nut region, the assembly reference surface region, and the background / screw region. The residual connections between each layer alleviate the gradient vanishing problem in deep networks and ensure the integrity of feature transfer.

[0017] In this embodiment, the detection parameters are adjustable parameters in the visual recognition model that match the workpiece specifications. In this embodiment, they are dimensional tolerance parameters, including the standard gap threshold for screw and nut assembly, the grayscale gradient threshold for contour extraction, and the Gaussian surface parameters for sub-pixel fitting. Adjusting the detection parameters allows the visual recognition model to adapt to the dimensional characteristics of different workpiece models, avoiding recognition deviations due to specification differences. The preset reliability threshold is the default reliability judgment benchmark when the visual recognition model outputs preliminary results. It is a value between 0 and 1, used to screen the reliability of preliminary recognition results, for example, the default is 0.85. The target detection parameters are visual recognition model detection parameters that are precisely matched with the current workpiece to be assembled after adjustment based on product model information. They are the optimal operating parameters of the visual recognition model for the current workpiece, ensuring the accuracy of assembly status recognition for this type of workpiece.

[0018] In this embodiment, the target confidence threshold is a confidence judgment benchmark that is adapted to the current workpiece assembly accuracy requirements after adjusting the tolerance range of the product model. The narrower the tolerance range, the higher the assembly accuracy requirements, and the higher the target confidence threshold, which can improve the rigor of high-precision workpiece identification.

[0019] S102: Acquire the original image of the screw and nut assembly area to be inspected and preprocess it to locate and extract the assembly area image.

[0020] In this embodiment, the screw and nut assembly area to be detected is the physical spatial location of the screw and nut after the engagement operation, located in the fixture positioning area of ​​the assembly station. This is the target area for visual inspection, determining the camera's field of view and the focusing position for image acquisition. The original image is image data directly captured by the visual acquisition device without any algorithmic processing. It includes original defects such as illumination interference, optical distortion, and background noise. It serves as the input data source for subsequent image preprocessing, and its quality directly affects the final recognition accuracy. The visual acquisition device consists of the main camera and the auxiliary camera. Preprocessing involves a series of algorithmic optimization operations performed on the original image. Its purpose is to eliminate interference and enhance target features, laying the foundation for the localization extraction in this step and the subsequent state recognition. Its function is to improve the distinguishability between the assembly area and the background in the image. The assembly area image is a partial sub-image cropped from the preprocessed image, containing only the screw-nut assembly. Irrelevant backgrounds such as the station fixture and workbench are removed. This image is the data input to the visual recognition model and is directly used for assembly state analysis.

[0021] S103: Input the assembly area image into the preset visual recognition model for analysis to obtain the preliminary assembly status recognition results and the corresponding confidence score.

[0022] In this embodiment, the assembly area image is a partial sub-image of the screw-nut assembly after preprocessing and positioning and cropping. It includes two types: main view and auxiliary view. The main view is the planar view of the assembly contact surface, and the auxiliary view is the side profile view. It is the input data of the preset visual recognition model, and its quality directly determines the accuracy of the preset visual recognition model analysis.

[0023] In this embodiment, the preliminary assembly status recognition result is the assembly quality category output by the preset visual recognition model after analyzing the image of the assembly area. Predefined categories include: qualified, unqualified, and ambiguous. Qualified indicates that the engagement gap is within tolerance; unqualified indicates looseness, offset, or incomplete engagement; and ambiguous indicates that the gap is at the tolerance boundary or that local features are incomplete, making a clear judgment impossible. This is an intermediate result to be verified and requires further screening or fusion of auxiliary information based on the confidence score. The confidence score is the reliability assessment value of the preliminary assembly status recognition result by the preset visual recognition model, ranging from 0 to 1. A higher score indicates stronger reliability and is positively correlated with the feature matching degree and segmentation accuracy output by the visual recognition model. It serves as the basis for determining the reliability of the preliminary result and needs to be compared with a determined target confidence threshold to determine the subsequent processing path.

[0024] S104: If the confidence score is greater than or equal to the target confidence threshold and the preliminary assembly state identification result is not a predefined fuzzy state, then the preliminary assembly state identification result shall be taken as the final assembly state result.

[0025] In this embodiment, the target confidence threshold is obtained by adjusting the tolerance range of the workpiece model to be assembled. It is a confidence judgment benchmark that adapts to the current workpiece assembly accuracy requirements and is the judgment threshold of this embodiment. The preliminary assembly status recognition result is generated by the visual recognition model analysis and includes three predefined results: qualified, unqualified, and ambiguous. These are intermediate states to be verified and are the judgment objects of this embodiment. They must simultaneously meet the requirements of being in an unambiguous state and having a qualified confidence level in order to be used as the final result.

[0026] In this embodiment, the predefined fuzzy state refers to a special scenario where the visual recognition model cannot clearly determine the assembly state. Specifically, this includes situations where the engagement gap is at the tolerance boundary or the edge contour integrity is less than 95%, resulting in a fuzzy state. If such a state exists, an auxiliary judgment process is required, and the result is not directly output. The engagement gap at the tolerance boundary is defined as 0.02±0.001mm or 0.05±0.001mm. The final assembly state result, verified through this embodiment, is the final assembly quality conclusion that can be used for subsequent alarms and controls. It is consistent with the preliminary result and directly connects to the subsequent alarm and control module to guide the assembly station operation.

[0027] S105: If the confidence score is less than the target confidence threshold or the preliminary assembly status identification result is a predefined fuzzy state, then obtain the time-series auxiliary data associated with the current assembly process, and generate assembly process auxiliary judgment information based on the time-series auxiliary data.

[0028] In this embodiment, the temporal auxiliary data is non-image data that is collected synchronously with the current assembly process and changes over time. It can supplement the limitations of visual recognition, represent the characteristics of the assembly process, and serves as the input data for this embodiment. It is used to compensate for insufficient visual image information and improve the accuracy of status determination. The assembly process auxiliary judgment information is a supplementary conclusion generated based on the extracted features and analysis of the temporal auxiliary data. It is used to help confirm the assembly status, such as: the torque curve is normal, and the assembly is not loose; the audio is abnormal, and there is a risk of incomplete engagement.

[0029] S106: Input the preliminary assembly status identification results, confidence scores, and assembly process auxiliary judgment information into the preset multi-feature decision model to generate the final assembly status results.

[0030] In this embodiment, the preliminary assembly state recognition results include three categories: qualified, unqualified, and ambiguous. This is one of the basic input features of this embodiment and requires further verification based on auxiliary information. If the state is ambiguous, auxiliary judgment information is needed to clarify the conclusion. The confidence score is a reliability assessment value of the preliminary result, denoted as a 0-1 range, representing the credibility of the visual model analysis results. It serves as a reliability weight feature input to the preset multi-feature decision model, and the score affects the priority of the preliminary result during multi-feature fusion. The assembly process auxiliary judgment information is a supplementary conclusion generated based on time-series auxiliary data. It is a supplementary feature to compensate for the limitations of visual recognition, used to verify or correct the preliminary results and solve problems such as insufficient confidence and ambiguous states. The time-series auxiliary data includes torque-angle curves and audio signals. The preset multi-feature decision model is a pre-trained ensemble learning model that can fuse visual features and time-series auxiliary features to output a uniquely determined assembly state. It possesses fault tolerance and high-precision judgment capabilities and is the processing unit of this embodiment, responsible for summarizing multi-dimensional information and outputting the final result that can be used for control. The visual features include the preliminary result and confidence score, while the time-series auxiliary features include auxiliary judgment information.

[0031] Specifically, the multi-feature decision model in this application adopts a lightweight three-layer architecture of feature fusion layer, feature enhancement layer, and decision output layer. It is suitable for multi-source data fusion scenarios in screw and nut assembly inspection, with no redundant layers and a balance between inference speed and judgment accuracy. The feature fusion layer is a fully connected layer with an input dimension of 20. It is responsible for linearly fusing the initial assembly state encoding features, confidence scores, and multimodal feature vectors. It assigns adaptive weights to features from different sources through a weight matrix to enhance the correction effect of auxiliary judgment information on ambiguous states. The feature enhancement layer embeds a gated recurrent unit and a dropout layer with a dropout rate of 0.2. The feature enhancement layer embeds a gated recurrent unit to capture the temporal correlation and logical dependency between features, such as the consistency between confidence and auxiliary judgment information. The dropout layer suppresses overfitting and retains core decision features. The decision output layer is a fully connected layer with a Softmax activation function. It outputs the probability distribution of three final assembly states, including qualified, unqualified, and requiring manual review. Features are passed between layers through the ReLU activation function, and residual connections are used to ensure the integrity of feature fusion. The overall architecture is adapted to industrial decision-making needs.

[0032] In this embodiment, the final assembly status result is a unique and definite assembly quality conclusion generated after multi-feature fusion judgment, that is, only qualified or unqualified without ambiguity, which clarifies whether the assembly meets the specifications and guides the actual operation of the workstation.

[0033] S107: Based on the final assembly status result, generate corresponding alarm information and control instructions.

[0034] In this embodiment, the final assembly status result is uniquely output by the multi-feature decision model, and the determined assembly quality conclusion directly determines the type of alarm information and control instructions. The final assembly status result includes two categories: qualified and unqualified. Alarm information is generated based on the final assembly status result and serves as quality feedback information to prompt on-site personnel or the system. It is clear in content and associated with specific reasons, conveying the assembly quality status to on-site maintenance personnel and system monitoring terminals. When unqualified, the root cause of the problem is simultaneously marked. This alarm information includes prompt and warning types. Control instructions are generated based on the final assembly status result and can directly drive the assembly station equipment to perform actions. They possess real-time performance and executable capability, interfacing with the station's mechanical structure, conveying system, etc., and are directed towards the industrial equipment end to achieve automated closed-loop control of the assembly process, ensuring the flow of qualified workpieces and the handling of unqualified workpieces.

[0035] As can be seen from the above, this application achieves the identification of the assembly status of screws and nuts based on machine vision. First, it adaptively adjusts the detection parameters and confidence threshold of the vision model by workpiece model to adapt to the identification needs of different models and improve the adaptability of detection and identification. It acquires images by dual cameras and extracts the assembly area through preprocessing. It achieves accurate preliminary determination of the assembly status through semantic segmentation model. The reliability of the preliminary determination result is ensured by dual screening of confidence and fuzzy state. For the results with doubts, it integrates torque-angle and audio time series auxiliary data to generate auxiliary judgment information. Then, it obtains the final result through multi-feature decision model fusion analysis, which improves the accuracy and precision of assembly status identification, reduces false judgments and missed judgments, and finally generates alarms and control commands, thereby improving the intelligence level and efficiency of assembly detection and identification.

[0036] In one embodiment of this application, based on the product model information of the workpiece to be assembled, the detection parameters of a preset visual recognition model and a preset confidence threshold are adjusted to obtain target detection parameters and a target confidence threshold, including: Query the preset model parameter mapping table, which records the standard assembly dimensions and tolerance ranges corresponding to different product models; Based on the standard assembly dimensions and tolerance ranges obtained from the query, the dimensional tolerance parameters used for visual recognition model analysis are calculated and determined as target detection parameters. The target confidence threshold is obtained by adjusting the preset confidence threshold based on the tolerance range.

[0037] In this embodiment, the model parameter mapping table is a pre-built and stored structured parameter table that associates each screw and nut product model with its corresponding assembly parameters. It serves as the fundamental data source for parameter adjustment and the starting point for this embodiment. The standard assembly dimensions and tolerance range of the current workpiece are obtained by querying this table. The standard assembly dimensions are the design assembly reference dimensions for the screw and nut of the corresponding product model, such as standard engagement clearance, screw fitting diameter, and nut assembly mating surface dimensions. These dimensions provide a dimensional judgment benchmark for visual recognition, are the basis for calculating dimensional tolerance parameters, and determine the range of acceptable dimensions for the visual model.

[0038] In this embodiment, the tolerance range refers to the allowable upper and lower deviation range of the standard assembly dimensions. It is an indicator of assembly accuracy, such as a screw-in gap of 0.02-0.05mm and a tolerance range of ±0.015mm. It also affects the target detection parameters and the target confidence threshold. The higher the accuracy requirement, the smaller the tolerance range, and the more stringent the parameter adjustments. The dimensional tolerance parameter is an adjustable dimensional parameter adapted to the analysis logic of the visual recognition model. It is derived from the standard assembly dimensions and the tolerance range and is a core parameter for the visual model to determine whether the assembly dimensions are qualified. As a target detection parameter, it is directly written into the visual model to guide the model's contour extraction, gap calculation, and other analysis operations.

[0039] In this embodiment, the target detection parameter is the detection parameter of the visual recognition model adjusted for the current product model. In this embodiment, it is the size tolerance parameter, which is the optimal size determination parameter for the current workpiece, so that the visual recognition model can adapt to the size characteristics of different models. The preset confidence threshold is the default confidence determination benchmark of the visual recognition model, which is in the range of 0-1. For example, the default confidence threshold of 0.85 is the basis for adjustment. The target value is obtained by adaptability correction based on the tolerance range of the current workpiece.

[0040] As can be seen from the above, this embodiment adjusts the target detection parameters and target confidence threshold of the visual recognition model by using the workpiece product model information. A preset model parameter mapping table can quickly match the standard assembly dimensions and tolerance ranges corresponding to different models, achieving accurate parameter retrieval. The dimensional tolerance parameters calculated based on the matched dimensions and tolerances allow the detection standards of the visual recognition model to be adapted to the actual assembly requirements of the workpiece, avoiding recognition deviations for different models with fixed parameters. Adjusting the preset confidence threshold according to the tolerance range ensures that the threshold setting matches the workpiece assembly accuracy requirements, improving the accuracy of the visual recognition model in determining the assembly status of different workpiece models.

[0041] In one embodiment of this application, adjusting a preset confidence threshold based on a tolerance range to obtain a target confidence threshold includes: Calculate the difference between the upper and lower limits of the tolerance range, and use it as the tolerance bandwidth; Based on the preset confidence level adjustment rules, the first confidence level adjustment amount is determined according to the tolerance bandwidth; wherein the tolerance bandwidth is negatively correlated with the first confidence level adjustment amount, and the smaller the tolerance bandwidth, the larger the first confidence level adjustment amount; The target confidence threshold is obtained by adjusting the preset confidence threshold based on the first confidence adjustment amount.

[0042] In this embodiment, the tolerance bandwidth is the difference between the upper and lower limits of the tolerance range, reflecting the breadth of the tolerance range and directly indicating the assembly precision requirements. A smaller bandwidth indicates higher precision requirements. It serves as an intermediate parameter connecting the tolerance range and the confidence adjustment amount, and is an input item for the confidence adjustment rule. The preset confidence adjustment rule is a rule pre-defined based on industrial measured data and visual recognition model training results. It is used to calculate the first confidence adjustment amount based on the tolerance bandwidth, including calculation formulas, coefficients, and value constraints. It is the basis for the adjustment process and defines the negative correlation between the tolerance bandwidth and the first confidence adjustment amount.

[0043] In this embodiment, the first confidence adjustment amount is a value calculated based on the tolerance bandwidth and confidence adjustment rules, used to correct the preset confidence threshold. It can be positive or negative and is the specific quantitative value for adjusting the preset confidence threshold. Changes in the tolerance bandwidth are converted into adjustments to the confidence threshold through this value. The preset confidence threshold is the default confidence judgment benchmark of the visual recognition model, i.e., the 0-1 range, for example, 0.85. It is calculated together with the first confidence adjustment amount to obtain the target value adapted to the current product model. The target confidence threshold, after correction by the first confidence adjustment amount, is the final confidence judgment benchmark adapted to the assembly accuracy requirements of the current product model. It is used for reliability screening of the subsequent preliminary recognition results and directly connects to the subsequent assembly status recognition judgment process.

[0044] Furthermore, the target confidence threshold The following formula can be used to calculate and comprehensively represent tolerance requirements, real-time operating conditions, and historical quality feedback: ; in, To pre-set the reliability threshold; This is an adjustment based on the tolerance bandwidth; the smaller the tolerance bandwidth, The larger the positive value; The adjustment amount is estimated by filtering sensor data based on real-time assembly conditions, such as environmental vibration and lighting stability. The confidence threshold is adjusted based on recent continuous assembly pass rates. A higher pass rate allows for a slight decrease in the threshold to improve efficiency, and vice versa. This model enables the confidence threshold to adapt adaptively, optimizing system efficiency while maintaining accuracy.

[0045] As can be seen from the above, this embodiment adjusts the preset confidence threshold through tolerance bandwidth. First, the difference between the upper and lower tolerance limits is calculated to obtain the tolerance bandwidth, which serves as the basis for confidence adjustment, providing a clear standard for threshold adjustment and avoiding subjective bias. A negative correlation rule is established between tolerance bandwidth and confidence adjustment amount, allowing workpieces with higher assembly precision requirements and smaller tolerance bandwidth to be matched with larger adjustment amounts and more stringent target confidence thresholds. This achieves precise adaptation between the threshold and the workpiece assembly precision requirements, improving the rationality and accuracy of judging the assembly status of workpieces with different precision requirements, effectively reducing the probability of misjudgment, and providing a judgment benchmark that conforms to actual process requirements for subsequent assembly status identification.

[0046] In one embodiment of this application, acquiring and preprocessing the original image of the screw and nut assembly area to be detected, and locating and extracting the assembly area image, includes: The system synchronously acquires original images using a main camera positioned above the assembly station and an auxiliary camera positioned to the side. The original images include images from the main camera and images from the auxiliary camera. The main camera is used to acquire a planar view of the assembly contact surface, and the auxiliary camera is used to acquire a side profile view. The original image is processed by illumination non-uniformity correction, noise filtering and optical distortion correction to obtain the processed main camera image and auxiliary camera image; Based on a preset template matching algorithm, the region where the screw and nut assembly is located is located from the processed main camera image to obtain the position of the assembly from the main viewpoint. Based on the position of the assembly from the main viewpoint and the preset dual-camera position calibration relationship, the corresponding assembly area in the auxiliary camera image is determined; Sub-images containing the assembly region are cropped from the main camera image and the auxiliary camera image respectively, and used as the assembly region image.

[0047] In this embodiment, the main camera and the auxiliary camera are industrial area array cameras. The main camera is positioned directly above the assembly station, and the auxiliary camera is positioned to the side of the station. They work together to acquire images, providing multi-view original images for visual recognition, compensating for blind spots and feature loss issues in single-view applications. These are the hardware components for image acquisition in this embodiment, determining the viewpoint and feature integrity of the original images. The original images are unprocessed images directly captured by the main and auxiliary cameras, including images from the main camera and the auxiliary camera. The main camera image is a planar view, and the auxiliary camera image is a side-view contour view. These images contain defects such as uneven lighting, noise, and optical distortion, and are the initial processing objects in this embodiment, serving as the basic data source for subsequent preprocessing and region extraction.

[0048] In this embodiment, uneven illumination correction eliminates brightness deviations; noise filtering removes electronic noise and interference points from the image while preserving edge contour information; distortion correction eliminates image distortion caused by the lens, improving image quality and providing clear features for subsequent template matching and region localization. The preset template matching algorithm is a pre-calibrated image matching algorithm. This embodiment uses normalized cross-correlation (NCC) and uses the standard assembly template image of the screw and nut as a reference to match the target region in the preprocessed image, achieving accurate positioning of the assembly from the main viewpoint, which is a prerequisite for subsequent auxiliary viewpoint region mapping.

[0049] Specifically, in this embodiment, the NCC template matching algorithm adapted for screw and nut assembly detection is a two-dimensional sliding window matching hierarchical architecture for industrial vision. It has no deep network layers and consists of three layers: a template preprocessing layer, a sliding window calculation layer, and a matching result filtering layer. These layers are unidirectionally connected according to the data processing flow, adapting to the needs of industrial positioning. The template preprocessing layer is responsible for performing grayscale normalization and mean elimination preprocessing on the standard screw and nut assembly template image, eliminating the influence of the template's own grayscale deviation on the matching. The sliding window calculation layer is the core layer of the algorithm. It traverses the preprocessed image of the main camera to be matched through the sliding window, completing the calculation of the grayscale covariance and variance between the template and the window region, as well as solving the NCC correlation coefficient for each window. The matching result filtering layer is responsible for threshold filtering and extreme value extraction of the NCC correlation coefficient of all windows, locking in the optimal matching region. The layers are connected through pixel-level grayscale data transfer, without complex feature mapping, which improves the real-time inference efficiency of the algorithm on the industrial control terminal.

[0050] In this embodiment, the position of the assembly in the main view is the region where the screw and nut assembly is located in the main camera image after template matching, represented by the bounding box pixel coordinates, and serves as the mapping reference for the assembly region in the auxiliary camera image. The dual-camera position calibration relationship is the spatial position association relationship between the main and auxiliary cameras obtained in advance through camera calibration, expressed in the form of a homography matrix, to achieve pixel coordinate transformation across cameras and coordinate mapping between the main and auxiliary camera images, thereby improving the accuracy of the assembly region positioning in the auxiliary view. The assembly region image is a partial sub-image containing only the screw and nut assembly, cropped from the preprocessed main and auxiliary camera images, after removing irrelevant backgrounds such as fixtures and worktables, and directly input into the adjusted visual recognition model for state analysis.

[0051] As can be seen from the above, this embodiment simultaneously acquires images from multiple perspectives using dual cameras. The main camera captures the planar features of the assembly contact surface, while the auxiliary camera acquires the side-view contour features, compensating for the lack of information from a single perspective and providing comprehensive image data for subsequent recognition. After preprocessing with illumination correction, noise filtering, and distortion correction, image defects are effectively eliminated, restoring the true shape and size of the assembly and improving image quality. A template matching algorithm accurately locates the position of the assembly from the main perspective, and the auxiliary perspective region is derived based on the dual-camera calibration relationship, achieving accurate extraction of the assembly region from multiple perspectives. After cropping, only the core assembly region is retained, and irrelevant background interference is eliminated, reducing the computational load of subsequent models and providing high-quality, targeted input data for the visual recognition model, laying the foundation for accurate detection and recognition.

[0052] In one embodiment of this application, determining the corresponding assembly region in the auxiliary camera image based on the main view assembly position and a preset dual-camera position calibration relationship includes: Obtain the bounding box coordinates of the assembly position from the main viewpoint. The bounding box coordinates are pixel coordinates in the main camera image coordinate system. Based on the homography matrix in the preset dual-camera position calibration relationship, the coordinates of the four vertices of the bounding box are transformed from the main camera image coordinate system to the auxiliary camera image coordinate system. Based on the transformed coordinates of the four vertices, the corresponding minimum bounding rectangle region is determined in the auxiliary camera image, which serves as the corresponding assembly region in the auxiliary camera image.

[0053] In this embodiment, the bounding box coordinates of the main view assembly position are the pixel coordinates of the rectangular area surrounding the screw and nut assembly located in the main camera image through template matching. These coordinates are represented by the top-left / bottom-right vertices or all four vertices and belong to the main camera image coordinate system. They are the original input coordinates for the coordinate transformation in this embodiment and form the basis for the auxiliary view assembly region positioning. Both the main camera image coordinate system and the auxiliary camera image coordinate system are two-dimensional pixel coordinate systems with the top-left corner of the camera-captured image as the origin, the horizontal x-axis to the right, and the vertical y-axis downwards. They are independent pixel reference systems, and the unit is pixels.

[0054] In this embodiment, the dual-camera position calibration relationship is the association between the spatial positions and imaging parameters of the primary and auxiliary cameras, obtained beforehand through camera calibration. It serves as the basis for cross-camera coordinate mapping and provides mathematical support for coordinate transformation, represented by a homography matrix. The homography matrix is ​​a 3×3 floating-point matrix characterizing the projection transformation relationship between two planes. It embodies the dual-camera position calibration relationship and enables precise transformation of a point / line / plane in one pixel coordinate system to another. It is the mathematical tool for coordinate transformation in this embodiment, directly completing the vertex coordinate transformation between the primary and auxiliary camera coordinate systems.

[0055] In this embodiment, the transformed coordinates of the four vertices are the new pixel coordinates obtained in the auxiliary camera image coordinate system after the homography matrix transformation of the four vertices of the main view bounding box. These coordinates serve as the positioning reference points for the auxiliary view assembly region and are the intermediate coordinates connecting the main and auxiliary views, providing a basis for the subsequent calculation of the minimum bounding rectangle. The minimum bounding rectangle region is the smallest rectangular region generated in the auxiliary camera image that can completely enclose all vertices, with the transformed four vertices as references. Its bounding box is the precise outline of the auxiliary view assembly region and is directly used as the clipping basis for the assembly region in the auxiliary camera image.

[0056] As can be seen from the above, this embodiment achieves precise coordinate transformation between the main and auxiliary camera image coordinate systems based on the pixel coordinates of the assembly bounding box from the main viewpoint and the homography matrix calibrated by the dual cameras. This provides rigorous mathematical support for cross-view assembly region localization, ensuring accuracy and consistency in positioning. By taking the minimum bounding rectangle of the transformed vertex coordinates, the core region of the assembly can be accurately defined in the auxiliary camera image, adapting to the positional offset after coordinate transformation and avoiding missed or over-selected regions. This achieves precise mapping of the assembly region from the main viewpoint to the auxiliary viewpoint, improving the matching and completeness of the dual-view assembly region extraction. This provides accurate regional basis for subsequent multi-view image analysis, enhancing the overall detection accuracy.

[0057] In one embodiment of this application, the preset visual recognition model is a preset semantic segmentation neural network model; the assembly region image is input into the preset visual recognition model for analysis to obtain preliminary assembly state recognition results and corresponding confidence scores, including: The assembly region image is input into a preset semantic segmentation neural network model to obtain a segmentation mask image. The segmentation mask image includes the nut region, screw region and preset assembly reference surface region in the assembly region image. Based on the segmentation mask image, the sub-pixel level lower edge contour of the nut region and the sub-pixel level upper edge contour of the assembly reference surface region are extracted. Calculate the average pixel distance between the lower edge contour and the upper edge contour; Based on a pre-calibrated pixel-to-physical size conversion coefficient, the average pixel distance is converted into an actual physical gap value, which is used as the spin gap; Based on the engagement gap and preset state determination rules, preliminary assembly state identification results and confidence scores are generated.

[0058] In this embodiment, the semantic segmentation neural network model is a deep learning model for pixel-by-pixel image classification. It can label each pixel in the image as a corresponding target category. It is the visual recognition model in this embodiment, pre-trained with screw and nut assembly samples, to achieve pixel-level segmentation of the assembly region image, providing accurate region division basis for subsequent contour extraction and gap calculation. The segmentation mask image is the output of the semantic segmentation model. It is a pixel-level labeled image with the same size as the input assembly region image. Each pixel corresponds to a unique category label, such as nut / screw / assembly reference surface / background. Different regions are distinguished by binarization / color annotation. It is an intermediate product of this embodiment and is the direct basis for extracting sub-pixel edge contours, eliminating background interference.

[0059] In this embodiment, the assembly reference surface area is a preset reference plane area used to determine the assembly gap, such as the mating surface of the screw installation or the reference end face of the nut engagement. It is one of the labeling categories of the segmentation mask image and serves as a reference reference for calculating the engagement gap. The distance between its edge and the edge of the nut is the determination dimension. Subpixel-level edge contours are regional edge contours with a precision greater than that of pixels, for example, reaching 0.1-0.5 pixels. This breaks through the physical resolution limitations of camera pixels, achieves accurate edge positioning, and can significantly improve the contour extraction accuracy, ensuring that the physical dimensional error of the subsequent gap calculation is within the industrially permissible range.

[0060] In this embodiment, the average pixel distance is the average of the pixel distances of all sampling points along the vertical gap direction between the lower edge of the nut sub-pixel and the upper edge of the assembly reference surface sub-pixel. It is the intermediate value for calculating the engagement gap and serves as a bridge connecting the pixel dimension and the physical dimension. The pixel-to-physical size conversion coefficient is a pre-calibrated conversion ratio between pixels and actual physical dimensions, in mm / pixel. It is determined by camera resolution, installation distance, and lens parameters, and requires separate calibration for different workstations / models. It is used to convert pixel distance to actual physical engagement gap and is a parameter for visual inspection calibration. The engagement gap is the actual physical gap value between the nut and the assembly reference surface, in mm. It is an indicator for judging the assembly state of the screw and nut and is directly substituted into the state judgment rules to generate preliminary assembly state results. The preset state judgment rules are assembly state judgment criteria based on the standard assembly dimensions and tolerance range of the product model, with the engagement gap as the core. They include judgment intervals for qualified / unqualified / ambiguous states, define the correspondence between the engagement gap and the assembly state, and serve as the rule basis for generating preliminary results and confidence levels.

[0061] As a possible embodiment, to improve the accuracy of gap measurement, the engagement gap is calculated by merging the front view and the side view. The formula for calculating the actual engagement gap G is as follows: ; ; in, and The average pixel distances calculated for the main and secondary views are respectively; and This refers to the pixel-to-physical size conversion factor for the corresponding view; This indicates the clarity score of the outline in the main view. The outline sharpness score of the auxiliary view for The fusion weight is determined by the contour sharpness score C in each view analysis area. The higher the sharpness, the greater the weight. The C value can be obtained by calculating the mean of the image gradient magnitude. This fusion method effectively reduces the measurement error of a single view.

[0062] As can be seen from the above, this embodiment achieves pixel-level precise segmentation of the assembly region by employing a semantic segmentation neural network model, clearly distinguishing the nut, screw, and assembly reference surface areas, laying a precise regional foundation for subsequent contour extraction. By extracting sub-pixel-level edge contours, the pixel resolution limitation is overcome, improving contour positioning accuracy. Based on the average pixel distance calculation and pixel-to-physical size conversion, the engagement gap is accurately quantified. Using the engagement gap as the judgment index, preliminary results and confidence scores are generated through preset rules, providing a clear quantitative basis for assembly state judgment, improving the accuracy and reliability of preliminary identification. At the same time, the confidence score provides a quantitative standard for subsequent result screening, effectively reducing invalid judgments and laying a solid foundation for accurate visual analysis of assembly state recognition.

[0063] In one embodiment of this application, based on a segmentation mask image, the sub-pixel-level lower edge contour of the nut region and the sub-pixel-level upper edge contour of the assembly reference surface region are extracted, including: Based on the segmentation mask image, binary masks for the nut region and the assembly reference surface region are generated respectively; The binarized mask is dilated to obtain the region to be analyzed. In the assembly region image, obtain the grayscale image block corresponding to the region to be analyzed; Calculate the grayscale gradient of a grayscale image patch in the direction perpendicular to the edge; Gaussian surface fitting is performed on the gray-level gradient distribution to determine the sub-pixel positions of the gradient extrema; Connect subpixel location points to generate subpixel-level lower and upper edge contours.

[0064] In this embodiment, the binarization mask is an image obtained by binarizing a single target region based on a segmented mask image. The single target region is the nut / assembly reference surface, with a target region pixel value of 255 and a background value of 0, retaining only the outline and region information of the single target. The dilation operation is a fundamental operation in morphological image processing. It expands the target region of the binarized mask pixel-level using preset structuring elements, filling in small holes, connecting discontinuous edges, and expanding the target region's range. This operation generates the region to be analyzed, ensuring that the extracted edge regions include complete grayscale feature information. The region to be analyzed is the target region obtained after the binarized mask undergoes dilation. It represents the effective grayscale analysis range for subsequent edge extraction, covering the actual edges of the target region and a small number of surrounding grayscale transition pixels. This limits the cropping range of the grayscale image block, avoids interference from irrelevant background pixels, and improves edge extraction accuracy.

[0065] In this embodiment, the grayscale image block is a grayscale image region that perfectly matches the position and size of the region to be analyzed, cropped from the original assembly region image. The original grayscale value distribution within this region is retained, and the grayscale data source extracted from the edges is preserved. The original grayscale value distribution is 0-255. All gradient calculations and Gaussian fitting are based on the original grayscale features of this region. The grayscale gradient perpendicular to the edge direction is the rate of change of grayscale values ​​calculated along a direction at 90° to the target edge. It is an indicator characterizing edge features; that is, significant extreme values ​​of the grayscale gradient will appear at the edge. Gaussian surface fitting is an algorithm based on the least squares method that fits discrete data points of the grayscale gradient distribution into a continuous Gaussian surface function. This can achieve sub-pixel-level extreme point localization, breaking through the limitations of camera pixel physical resolution and improving edge localization accuracy from the pixel level to the sub-pixel level, for example, 0.1-0.5 pixels.

[0066] In this embodiment, the sub-pixel positions of gradient extrema are floating-point pixel coordinates corresponding to the extrema in the gray-level gradient distribution curve found after Gaussian surface fitting. These coordinates represent the precise positions of the target edges and are the basic units constituting the sub-pixel-level edge contours. The floating-point nature of the coordinates enables high-precision edge positioning. The sub-pixel-level lower and upper edge contours are smooth, continuous curve contours formed by sequentially connecting all the continuous sub-pixel positions of gradient extrema. They correspond to the lower edge of the nut area and the upper edge of the assembly reference surface area, respectively, directly providing a high-precision edge coordinate basis for subsequent engagement clearance calculations.

[0067] As can be seen from the above, this embodiment achieves precise separation of the nut and the assembly reference surface area through a binarization mask. The area to be analyzed defined by the expansion operation can completely cover the edge grayscale transition features, providing a precise analysis range for contour extraction. By extracting the corresponding grayscale image block and calculating the grayscale gradient along the vertical edge direction, the grayscale change features of the edge can be accurately captured. The sub-pixel positions of the gradient extrema are determined by Gaussian surface fitting, breaking through the pixel-level resolution limitation and achieving high-precision edge positioning. The continuous contour generated by connecting the sub-pixel position points improves the accuracy and completeness of edge extraction, providing a reliable contour coordinate basis for the subsequent accurate calculation of the engagement gap, and effectively reducing the error in gap calculation.

[0068] In one embodiment of this application, a dilation operation is performed on the binarized mask to obtain the region to be analyzed, including: Based on the preset edge analysis width, determine the size of the dilation structuring element and the number of dilation iterations; The binarized mask of the nut region is expanded using a defined expansion structuring element to obtain the expanded nut mask region. The binarized mask of the assembly reference surface region is expanded using a defined expansion structural element to obtain the expanded reference surface mask region. The expanded nut mask area and the expanded reference plane mask area are taken as the regions to be analyzed.

[0069] In this embodiment, the preset edge analysis width is pre-calibrated based on camera resolution and edge grayscale transition characteristics. It covers the edge analysis region in pixels, ensuring that subsequent grayscale gradient calculations capture complete edge features. This width serves as the basis for the dilation operation, directly determining the size of the dilation structuring element and the number of iterations, thus avoiding insufficient analysis range or excessive dilation. The dilation structuring element is a template pixel block used to expand the target region in morphological dilation operations. Commonly, it is rectangular or circular, with dimensions in pixels. It determines the range of a single dilation and is the core tool of the morphological dilation operation. Its size matches the edge analysis width, ensuring that the expanded target region precisely covers the edge grayscale transition area.

[0070] In this embodiment, the number of dilation iterations refers to the number of times the dilation operation is repeated. A single dilation expands according to the structural element size, while multiple iterations allow for precise range control, avoiding edge shifts caused by single dilations. This, in conjunction with the structural element size, achieves the preset edge analysis width, balancing the analysis range and edge positioning accuracy. The dilated nut mask region and reference surface mask region are binarized images of the target area after the dilation operation on the binary mask of the nut or assembly reference surface. The pixel value of the target area is 255, filling in small holes and expanding the perimeter of the edges, corresponding to the effective analysis range of the two types of targets respectively. Together, they form the basis for subsequent grayscale image block cropping. The area to be analyzed is the set composed of the dilated nut mask region and the dilated reference surface mask region. This is the only effective range for subsequent grayscale image block cropping and edge gradient analysis, clearly defining the precise area for edge extraction and eliminating irrelevant background interference.

[0071] As can be seen from the above, this embodiment accurately determines the size and iteration number of the expansion structural element based on the preset edge analysis width, avoiding analysis deviations caused by excessive or insufficient expansion. A unified expansion parameter is used to expand the binary mask of the nut and the assembly reference surface separately, ensuring the consistency and standardization of the analysis range for both types of regions. This fills in small holes in the mask, connects discontinuous edges, and completely covers the edge grayscale transition area, preserving complete feature information for subsequent grayscale gradient calculation and sub-pixel fitting. The defined analysis area can accurately define the effective range of edge extraction, eliminate irrelevant background interference, improve the efficiency and accuracy of subsequent sub-pixel contour extraction, and lay a solid foundation for accurate calculation of the engagement gap.

[0072] In one embodiment of this application, if the confidence score is less than the target confidence threshold or the preliminary assembly state identification result is a predefined fuzzy state, then time-series auxiliary data associated with the current assembly process is acquired, and based on the time-series auxiliary data, assembly process auxiliary judgment information is generated, including: Acquire torque-angle curve data and audio signal data for the current assembly process; Preprocessing and feature extraction were performed on the torque-angle curve data and audio signal data respectively to obtain torque feature vector and acoustic feature vector; The torque feature vector and the acoustic feature vector are fused to obtain the multimodal feature vector; The multimodal feature vectors are input into a preset multimodal classification model to obtain the assembly process quality assessment results, which serve as auxiliary judgment information for the assembly process.

[0073] In this embodiment, the timing auxiliary data is non-image-based dynamic data that is collected synchronously with the screw and nut assembly process and changes over time. It represents the real-time physical characteristics of the assembly process, overcoming the limitations of visual recognition which only works with static images. This data serves as the input data for this embodiment and includes two types: torque-angle curves and audio signals. The torque-angle curve data is timing data collected synchronously by the torque and angle sensors during assembly, showing the torque value changing with the screw rotation angle. The unit is N·m / °. It represents the force-displacement matching relationship during engagement, characterizing the mechanical characteristics of the assembly and serving as the mechanical basis for determining whether it is loose or not fully engaged. The audio signal data is timing data of mechanical vibration audio collected by the microphone during the screw and nut engagement, measured in dB / Hz. It includes characteristic acoustic information of engagement, meshing, and friction, characterizing the vibrational acoustic features of the assembly, assisting in verifying the judgment results of the mechanical data, and improving reliability.

[0074] In this embodiment, feature extraction involves extracting quantified values ​​that characterize the data features from the preprocessed time-series data, removing redundant and noisy data, and transforming the original time-series data into fixed-dimensional feature vectors. This achieves dimensionality reduction and feature condensation of the time-series data, providing a foundation for subsequent feature fusion and model inference. The torque feature vector and acoustic feature vector are fixed-dimensional sets of values ​​extracted from the torque-angle curve and audio signal data, respectively, and represent the corresponding time-series data features, such as torque peak value and spectral centroid.

[0075] In this embodiment, the multimodal feature vector is a combined feature vector obtained by concatenating / fusing the torque feature vector and the acoustic feature vector according to preset rules. It summarizes the assembly process features of two different dimensions, namely mechanical and acoustic features, and serves as the input data for the multimodal classification model. The preset multimodal classification model is a classification model pre-trained based on a massive number of torque and audio samples of screw and nut assembly. It can analyze the multimodal feature vector and output the quality assessment results of the assembly process.

[0076] Specifically, the multimodal classification model adopts a three-level architecture: a dual-branch feature encoding layer, a cross-modal fusion layer, and a classification output layer, to meet the differentiated encoding requirements of torque temporal features and audio spectral features. The torque branch encoding layer consists of two 1D convolutional layers (Conv1d) and one bidirectional LSTM layer. Conv1d extracts local mechanical features of the torque-angle curve, such as torque peaks and angle abrupt change points, while the bidirectional LSTM captures the temporal dependencies of the engagement process. The audio branch encoding layer consists of three 1D convolutional layers and one GRU layer. Conv1d extracts frequency domain features of the audio signal, such as meshing frequency band energy and spectral centroid. The ReLU layer enhances the temporal correlation of acoustic features; the cross-modal fusion layer uses an attention mechanism to assign weights to the features output by the two branches, strengthening features strongly correlated with assembly quality, such as the torque steady-state segment and the meshing core frequency band, and maps the fused features into a fixed-dimensional vector through a fully connected layer; the classification output layer is a fully connected layer + Softmax activation function, which outputs three classification results of assembly quality, including normal, torque abnormal, and meshing abnormal. Features are passed between each layer through the ReLU activation function, and a Dropout layer is embedded to suppress overfitting with a dropout rate of 0.2. The overall architecture is adapted to the real-time inference needs of industry.

[0077] In this embodiment, the assembly process quality assessment result is a quantitative and qualitative assessment conclusion of the current assembly process output by the multimodal classification model, which can clearly identify whether there are abnormalities in the assembly process, such as insufficient torque or poor engagement.

[0078] As can be seen from the above, this embodiment introduces two types of time-series auxiliary data—torque-angle curves and audio signals—to supplement the features of the assembly process from mechanical and acoustic dimensions, compensating for the information limitations of single static visual inspection in scenarios where visual recognition judgment is questionable. By preprocessing and extracting features from the two types of data separately and fusing them into a multimodal feature vector, assembly information from different dimensions is summarized, and then analyzed by a multimodal classification model to obtain the quality assessment result. The generated auxiliary judgment information is highly targeted and reliable. This method allows assembly state judgment to solve the problems of critical ambiguity and insufficient confidence in judgment by using static visual features and dynamic process features, reducing the probability of false positives and false negatives, and providing support for accurate decision-making on the final assembly state.

[0079] Corresponding to the machine vision-based screw and nut assembly status recognition method in the above embodiment, Figure 2 This is a structural block diagram of a machine vision-based screw and nut assembly status recognition system provided in one embodiment of this application. For ease of explanation, only the parts relevant to the embodiment of this application are shown. References Figure 2The machine vision-based screw and nut assembly status recognition system 20 includes: a data adjustment module 21, an acquisition and extraction module 22, a preliminary recognition module 23, a dual judgment module 24, an auxiliary judgment module 25, a fusion judgment module 26, and an alarm control module 27.

[0080] Among them, the data adjustment module 21 is used to adjust the detection parameters of the preset visual recognition model and the preset confidence threshold based on the product model information of the workpiece to be assembled, so as to obtain the target detection parameters and the target confidence threshold. The acquisition and extraction module 22 is used to acquire the original image of the screw and nut assembly area to be inspected and perform preprocessing, and to locate and extract the image of the assembly area; The preliminary identification module 23 is used to input the assembly area image into a preset visual recognition model for analysis, and obtain the preliminary assembly state identification result and the corresponding confidence score. The dual judgment module 24 is used to take the preliminary assembly state recognition result as the final assembly state result if the confidence score is greater than or equal to the target confidence threshold and the preliminary assembly state recognition result is not a predefined fuzzy state. The auxiliary judgment module 25 is used to obtain the time-series auxiliary data associated with the current assembly process if the confidence score is less than the target confidence threshold or the preliminary assembly state recognition result is a predefined fuzzy state, and to generate assembly process auxiliary judgment information based on the time-series auxiliary data. The fusion judgment module 26 is used to input the preliminary assembly state recognition result, confidence score and assembly process auxiliary judgment information into a preset multi-feature decision model to generate the final assembly state result. The alarm control module 27 is used to generate corresponding alarm information and control commands based on the final assembly status results.

[0081] See Figure 3 , Figure 3 This is a schematic block diagram of an electronic device provided according to an embodiment of this application. Figure 3 The electronic device 300 in this embodiment may include one or more processors 301, one or more input devices 302, one or more output devices 303, and one or more memories 304. The processors 301, input devices 302, output devices 303, and memories 304 communicate with each other via a communication bus 305. The memories 304 store computer programs, including program instructions. The processors 301 execute the program instructions stored in the memories 304. Specifically, the processors 301 are configured to invoke the program instructions to perform the functions of the modules in the aforementioned device embodiments, for example... Figure 2The functions of the data adjustment module 21, acquisition and extraction module 22, preliminary identification module 23, dual judgment module 24, auxiliary judgment module 25, fusion judgment module 26, and alarm control module 27 are shown.

[0082] It should be understood that, in the embodiments of this application, the processor 301 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0083] Input device 302 may include a touchpad, a fingerprint sensor (for collecting the user's fingerprint information and fingerprint orientation information), a microphone, etc., and output device 303 may include a display (LCD, etc.), a speaker, etc.

[0084] The memory 304 may include read-only memory and random access memory, and provides instructions and data to the processor 301. A portion of the memory 304 may also include non-volatile random access memory. For example, the memory 304 may also store device type information.

[0085] In specific implementations, the processor 301, input device 302, and output device 303 described in the embodiments of this application can execute the implementation methods described in any embodiment of the machine vision-based screw and nut assembly state recognition method provided in the embodiments of this application, or they can execute the implementation methods of the electronic devices described in the embodiments of this application, which will not be repeated here.

[0086] In another embodiment of this application, a computer-readable storage medium is provided. This computer-readable storage medium stores a computer program, which includes program instructions. When executed by a processor, the program instructions implement all or part of the processes in the methods described above. Alternatively, the computer program can instruct related hardware to complete the process. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include any entity or device capable of carrying computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0087] The computer-readable storage medium can be an internal storage unit of the electronic device in any of the foregoing embodiments, such as a hard disk or memory of the electronic device. The computer-readable storage medium can also be an external storage device of the electronic device, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the electronic device. Furthermore, the computer-readable storage medium can include both internal and external storage units of the electronic device. The computer-readable storage medium is used to store computer programs and other programs and data required by the electronic device. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0088] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0089] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the electronic devices and units described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0090] In the several embodiments provided in this application, it should be understood that the disclosed electronic devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces or units, or it may be an electrical, mechanical, or other form of connection.

[0091] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of this application, depending on actual needs.

[0092] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0093] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for recognizing the assembly state of a screw and nut based on machine vision, characterized in that, include: S101: Based on the product model information of the workpiece to be assembled, adjust the detection parameters of the preset visual recognition model and the preset confidence threshold to obtain the target detection parameters and the target confidence threshold. S102: Acquire the original image of the screw and nut assembly area to be inspected and preprocess it to locate and extract the assembly area image; S103: Input the image of the assembly area into a preset visual recognition model for analysis to obtain preliminary assembly status recognition results and corresponding confidence scores; S104: If the confidence score is greater than or equal to the target confidence threshold and the preliminary assembly state identification result is not a predefined fuzzy state, then the preliminary assembly state identification result shall be taken as the final assembly state result. S105: If the confidence score is less than the target confidence threshold or the preliminary assembly state identification result is a predefined fuzzy state, then obtain the time-series auxiliary data associated with the current assembly process, and generate assembly process auxiliary judgment information based on the time-series auxiliary data. S106: Input the preliminary assembly state identification result, the confidence score and the assembly process auxiliary judgment information into a preset multi-feature decision model to generate the final assembly state result; S107: Based on the final assembly status result, generate corresponding alarm information and control instructions.

2. The method for identifying the assembly state of a screw and nut based on machine vision according to claim 1, characterized in that, The process involves adjusting the detection parameters of a preset visual recognition model and a preset confidence threshold based on the product model information of the workpiece to be assembled, to obtain target detection parameters and target confidence thresholds, including: Query the preset model parameter mapping table, which records the standard assembly dimensions and tolerance ranges corresponding to different product models; Based on the standard assembly dimensions and tolerance ranges obtained from the query, the dimensional tolerance parameters used for visual recognition model analysis are calculated and determined as the target detection parameters; The target confidence threshold is obtained by adjusting the preset confidence threshold based on the tolerance range.

3. The method for identifying the assembly state of a screw and nut based on machine vision according to claim 2, characterized in that, The step of adjusting the preset confidence threshold based on the tolerance range to obtain the target confidence threshold includes: Calculate the difference between the upper and lower limits of the tolerance range, and use it as the tolerance bandwidth; Based on a preset confidence adjustment rule, a first confidence adjustment amount is determined according to the tolerance bandwidth; wherein the tolerance bandwidth is negatively correlated with the first confidence adjustment amount, and the smaller the tolerance bandwidth, the larger the first confidence adjustment amount; The preset confidence threshold is adjusted based on the first confidence adjustment amount to obtain the target confidence threshold.

4. The method for identifying the assembly state of a screw and nut based on machine vision according to claim 1, characterized in that, The process of acquiring and preprocessing the original image of the screw and nut assembly area to be inspected, and locating and extracting the assembly area image includes: The original images are acquired synchronously by a main camera positioned above the assembly station and an auxiliary camera positioned to the side. The original images include images from the main camera and images from the auxiliary camera. The main camera is used to acquire a planar view of the assembly contact surface, and the auxiliary camera is used to acquire a side profile view. The original image is processed by illumination non-uniformity correction, noise filtering and optical distortion correction to obtain the processed main camera image and auxiliary camera image; Based on a preset template matching algorithm, the region where the screw and nut assembly is located is located from the processed main camera image to obtain the position of the assembly from the main viewpoint. Based on the position of the assembly in the main view and the preset dual-camera position calibration relationship, the corresponding assembly area in the auxiliary camera image is determined; Sub-images including the assembly region are cropped from the main camera image and the auxiliary camera image, respectively, to serve as the assembly region image.

5. The method for identifying the assembly state of a screw and nut based on machine vision according to claim 4, characterized in that, The step of determining the corresponding assembly region in the auxiliary camera image based on the main view assembly position and the preset dual-camera position calibration relationship includes: Obtain the bounding box coordinates of the position of the main view assembly, wherein the bounding box coordinates are pixel coordinates in the main camera image coordinate system; Based on the homography matrix in the preset dual-camera position calibration relationship, the coordinates of the four vertices of the bounding box coordinates are transformed from the main camera image coordinate system to the auxiliary camera image coordinate system. Based on the transformed coordinates of the four vertices, the corresponding minimum bounding rectangle region is determined in the auxiliary camera image, which serves as the corresponding assembly region in the auxiliary camera image.

6. The method for identifying the assembly state of a screw and nut based on machine vision according to claim 1, characterized in that, The preset visual recognition model is a preset semantic segmentation neural network model; The step of inputting the image of the assembly area into a preset visual recognition model for analysis to obtain preliminary assembly state recognition results and corresponding confidence scores includes: The assembly region image is input into a preset semantic segmentation neural network model to obtain a segmentation mask image, which includes the nut region, screw region and preset assembly reference surface region in the assembly region image; Based on the segmentation mask image, the subpixel-level lower edge contour of the nut region and the subpixel-level upper edge contour of the assembly reference surface region are extracted. Calculate the average pixel distance between the lower edge contour and the upper edge contour; Based on a pre-calibrated pixel-to-physical size conversion coefficient, the average pixel distance is converted into an actual physical gap value, which is used as the spin gap; Based on the engagement gap and the preset state determination rules, the preliminary assembly state identification result and the confidence score are generated.

7. The method for recognizing the assembly state of a screw and nut based on machine vision according to claim 6, characterized in that, The step of extracting the subpixel-level lower edge contour of the nut region and the subpixel-level upper edge contour of the assembly reference surface region based on the segmentation mask image includes: Based on the segmentation mask image, binary masks for the nut region and the assembly reference surface region are generated respectively; The binarized mask is dilated to obtain the region to be analyzed. In the assembly region image, obtain the grayscale image block corresponding to the region to be analyzed; Calculate the grayscale gradient of the grayscale image block in the direction perpendicular to the edge; Gaussian surface fitting is performed on the gray-level gradient distribution to determine the sub-pixel positions of the gradient extrema. Connect the subpixel position points to generate the subpixel-level lower edge contour and upper edge contour.

8. The method for recognizing the assembly state of a screw and nut based on machine vision according to claim 7, characterized in that, The process of dilating the binarized mask to obtain the region to be analyzed includes: Based on the preset edge analysis width, determine the size of the dilation structuring element and the number of dilation iterations; The binarized mask of the nut region is expanded using the determined expansion structural element to obtain the expanded nut mask region. The binarized mask of the assembly reference surface region is expanded using the determined expansion structural element to obtain the expanded reference surface mask region. The expanded nut mask area and the expanded reference surface mask area are taken as the region to be analyzed.

9. The method for identifying the assembly state of a screw and nut based on machine vision according to claim 1, characterized in that, If the confidence score is less than the target confidence threshold or the preliminary assembly state identification result is a predefined fuzzy state, then time-series auxiliary data associated with the current assembly process is acquired, and based on the time-series auxiliary data, assembly process auxiliary judgment information is generated, including: Acquire torque-angle curve data and audio signal data for the current assembly process; The torque-angle curve data and audio signal data are preprocessed and feature extracted respectively to obtain torque feature vector and acoustic feature vector; The torque feature vector and the acoustic feature vector are fused to obtain a multimodal feature vector; The multimodal feature vectors are input into a preset multimodal classification model to obtain the assembly process quality assessment results, which serve as auxiliary judgment information for the assembly process.

10. A machine vision-based screw and nut assembly status recognition system, characterized in that, include: The data adjustment module (21) is used to adjust the detection parameters of the preset visual recognition model and the preset confidence threshold based on the product model information of the workpiece to be assembled, so as to obtain the target detection parameters and the target confidence threshold. The acquisition and extraction module (22) is used to acquire the original image of the screw and nut assembly area to be detected and perform preprocessing, and to locate and extract the image of the assembly area; The preliminary identification module (23) is used to input the image of the assembly area into a preset visual recognition model for analysis, and obtain the preliminary assembly state identification result and the corresponding confidence score. The dual judgment module (24) is used to take the preliminary assembly state recognition result as the final assembly state result if the confidence score is greater than or equal to the target confidence threshold and the preliminary assembly state recognition result is not a predefined fuzzy state. The auxiliary judgment module (25) is used to obtain time-series auxiliary data associated with the current assembly process if the confidence score is less than the target confidence threshold or the preliminary assembly state recognition result is a predefined fuzzy state, and generate assembly process auxiliary judgment information based on the time-series auxiliary data. The fusion judgment module (26) is used to input the preliminary assembly state identification result, the confidence score and the assembly process auxiliary judgment information into a preset multi-feature decision model to generate the final assembly state result; The alarm control module (27) is used to generate corresponding alarm information and control instructions based on the final assembly status result.

Images