An automated assembly anomaly detection method and system based on machine vision

Abstract

The application discloses an automatic assembly anomaly detection method and system based on machine vision, which uses the network structure pre-trained by the DINO model to extract features from the collected screw safety rope images, extracts image features through self-supervised learning and knowledge distillation to generate a core set, uses a greedy selection algorithm to sample the core set, filters and aggregates the core set to complete the extraction of the safety rope image, clusters the filtered feature set through the Kmeans algorithm, and then uses a fully connected Gaussian conditional random field, a threshold segmentation algorithm and an area filtering algorithm to complete the extraction of the safety rope image, uses the SAM large model and the obtained safety rope image to input a prior bounding box, completes the extraction of the screw image, and judges the installation of the screw safety rope according to the extracted safety rope image and screw image, so that the assembly anomaly detection is realized. The application ensures the accurate segmentation and discrimination of the screw safety rope image and improves the efficiency and reliability of the automatic detection.

Description

Technical Field

[0001] This invention belongs to the field of machine vision-based automated assembly anomaly detection technology, specifically relating to a machine vision-based automated assembly anomaly detection method and system. Background Technology

[0002] Bolt safety ropes are crucial in industry, serving as devices to prevent bolts from loosening and falling off. Typically made of high-strength metal wire or synthetic materials, one end is secured to a bolt, and the other end to another bolt, ensuring that the bolt remains in place even under vibration or load variations. Proper installation of bolt safety ropes is essential to ensure their maximum effectiveness and prevent mechanical failures and safety accidents caused by bolt loosening, especially in high-vibration and high-impact environments such as aviation, aerospace, railway, and heavy machinery. Using bolt safety ropes can significantly improve equipment reliability and lifespan, ensuring production safety and efficiency.

[0003] Automated inspection of screw safety ropes faces numerous challenges in industrial applications, primarily stemming from the complexity of their operating environment and the objects they are used with. Firstly, the background in industrial environments is typically complex and varied, including the intersection of various mechanical equipment and components, making accurate identification of the screw safety rope from the background particularly difficult. Furthermore, due to manual intervention during installation, the appearance of the screw safety rope can exhibit significant inconsistencies, such as varying tension and installation angles, increasing the design complexity of the inspection system. Simultaneously, the diverse standards for defining normal and abnormal states of screw safety ropes also present a challenge. For example, some screw safety ropes may become loose or damaged due to improper installation or prolonged use; the criteria for defining these states are not fixed and must be adjusted according to specific application scenarios and safety requirements. In addition, the distribution of screw safety ropes on equipment can be very complex, including different types of connection points and path configurations, requiring the inspection system to possess a high degree of flexibility and adaptability. Moreover, in some large industrial enterprises or projects, multiple insurance institutions may be involved, and these institutions have stringent audits and requirements regarding equipment safety. Effective screw safety rope testing requires not only meeting the manufacturer's internal standards but also complying with the safety regulations of all relevant insurance agencies. This necessitates a testing system capable of identifying and adapting to diverse standards and requirements. For example... Figure 1 As shown, the left side illustrates the correct way to install a bolt safety rope. If a bolt becomes loose due to external force, the tension inside the safety rope will restrict the bolt's movement and apply a tightening force, preventing the bolt from completely loosening or falling off. Figure 9 As shown in (a), the safety rope applies a clockwise tightening force to the bolts on both sides to prevent them from loosening. Figure 9In (b), the safety rope applies opposite tightening forces to the bolts on both sides, failing to achieve the goal of tightening both bolts, thus constituting an incorrect bolt safety rope installation method. Automated inspection of screw safety ropes using machine vision technology can significantly improve inspection accuracy and efficiency, reduce human error, and overcome environmental complexity and operational diversity through advanced algorithms and learning models. This enables precise assessment and management of the screw safety rope's condition, thereby providing a more robust guarantee for industrial safety.

[0004] The commonly used machine vision inspection method in industry is supervised learning, which requires manual analysis and annotation of images and a large number of positive and negative samples. However, the bolt safety rope industrial production line targeted by this invention usually has enough normal samples and very few defective samples. In addition, the production line has certain requirements for inspection efficiency and inspection equipment, requiring low memory usage while maximizing efficiency.

[0005] Unsupervised learning in deep learning can meet the requirements of production line inspection while achieving high detection accuracy. Unsupervised learning only requires model training on normal samples on the production line, with very low requirements for real defect samples, and has strong environmental adaptability, effectively solving problems such as false alarms in actual inspection. However, existing unsupervised detection algorithms mainly target low-level structural anomalies, such as dents and scratches, but they encounter certain challenges when dealing with logical problems such as incorrect screw installation orientation. Moreover, in actual production lines, the main problem with screws and safety ropes is incorrect installation, while surface defects such as dents and scratches are not the focus of inspection. Therefore, in order to effectively detect the spatial information between screws and safety ropes... Summary of the Invention

[0006] Traditional methods require a large number of positive and negative samples and involve tedious manual annotation. Furthermore, the inconsistencies in bolt safety rope installation and complex backgrounds increase the difficulty of detection. The technical problem this invention aims to solve is to address the shortcomings of the existing technology by providing an automated assembly anomaly detection method and system based on machine vision. This method employs unsupervised learning methods from deep learning, using only normal samples for model training to improve detection accuracy and efficiency. It adapts to complex industrial environments, effectively identifies logical problems such as installation errors, reduces false alarms, and achieves accurate assessment of the condition of bolt safety ropes, ensuring the safety and reliability of industrial equipment. This invention addresses the technical problem of automated detection of bolt safety ropes in industrial environments.

[0007] The present invention adopts the following technical solution:

[0008] An automated assembly anomaly detection method based on machine vision includes the following steps:

[0009] S1. The network structure pre-trained by the DINO model is used to extract features from the collected screw and safety rope images. Image features are extracted through self-supervised learning and knowledge distillation to generate a core set.

[0010] S2. Use a greedy selection algorithm to sample the core set, filter and aggregate the core set to complete the extraction of the safety rope image;

[0011] S3. Cluster the feature set filtered in step S2 using the Kmeans algorithm, and then use the fully connected Gaussian conditional random field, threshold segmentation and area filtering algorithm to extract the safety rope image.

[0012] S4. Using the SAM large model and the safety rope image obtained in step S3, input the prior prompt box to complete the extraction of the screw image.

[0013] S5. Based on the safety rope and screw images extracted in steps S3 and S4, use position judgment knowledge to determine the installation of the screw and safety rope, thereby realizing assembly anomaly detection.

[0014] Preferably, step S1 specifically includes:

[0015] A set of images of normal screws and safety ropes (x) were collected; following the strategy of the DINO network structure model, the x-value of each image was calculated using the pre-trained DINOVisualTransformer-S / 8 model. i The original feature map f i The original feature map f i eigenvector v i Stored in memory library M i middle.

[0016] Preferably, in step S2, sampling the core set using a greedy selection algorithm specifically involves:

[0017]

[0018] M C ←M C ∪{m i}

[0019] Where M is the set of total image features before core set sampling, M C Let n be the sampled core set, where n is M. C One of the samples, m is MM C One of the samples, m i This refers to the new core set samples found after each algorithm iteration.

[0020] Preferably, step S3 specifically includes:

[0021] The obtained core sets are concatenated to construct a memory M. K-means is then used to cluster the feature vectors in memory M, obtaining K clusters and their central features f. Kmeans ;

[0022] Computational central feature f Kmeans Compared with the original feature map f i The cosine similarity between the images is calculated, and then the similarity map is interpolated to the original image size to obtain a preliminary segmentation map.

[0023] Post-processing was performed using a fully connected Gaussian conditional random field, and the safety rope segmentation image S was obtained by image threshold binarization and area threshold filtering.

[0024] Preferably, the CRF is implemented using multiple iterations of DenseCRF, and the specific paired potential function is as follows:

[0025]

[0026] Among them, f i and f j It is a pixel node, p i and p j That's their corresponding position, I i and I j They are their corresponding colors, w (1) w (2) θ α θ β θ γ This is a hyperparameter.

[0027] Preferably, in step S4, the SAM large model includes:

[0028] The image encoder, consisting of multiple convolutional and pooling layers, is used to extract image features by using a visual transformer pre-trained with MAE and minimizing it.

[0029] The prompt encoder uses the coordinates of the prior bounding box as prompt input;

[0030] The lightweight mask decoder predicts the segmentation mask by combining information from the image encoder and the cue encoder, restores the feature map to the original image size, and generates the segmentation result.

[0031] Preferably, in step S5, if the centers of the two screws are located on both sides of the smallest circumscribed rectangle of the safety rope, the configuration is considered correct; if this condition is not met, it is determined to be incorrect.

[0032] Preferably, the screw image extraction process is as follows:

[0033] Based on the image segmentation and feature extraction results, obtain the specific segmentation images of the screw and safety rope;

[0034] Image preprocessing is performed on the segmented images of the screw and safety rope to identify the contours of the screw and fuse, and to calculate the geometric center (x) of the minimum bounding rectangle of the screw and safety rope contours. c ,y c );

[0035] Compare the positional relationship between the geometric centers of the two screws and the geometric center of the safety rope; if the product of x and y is positive, the safety rope is correctly installed; if the product of x and y is negative, the safety rope is incorrectly installed; if either x or y is 0, it is a normal installation.

[0036] Based on the judgment results, mark the workpiece as correct or incorrect. If it is determined to be an incorrect configuration, mark the assembly and generate an anomaly report to prompt the operator to make adjustments.

[0037] Preferably, in steps S3 and S4, an anomaly detection mechanism is introduced during feature extraction and analysis. An alarm is triggered when data deviation or feature set anomalies are detected. The specific anomaly detection conditions are as follows:

[0038] Significant deviations in the feature clustering results indicate assembly errors;

[0039] The core feature set failed to pass the greedy algorithm screening, indicating potential image damage or occlusion;

[0040] Positional relationship analysis shows that the relative positions of the screw and the safety rope do not conform to the design specifications.

[0041] Secondly, embodiments of the present invention provide an automated assembly anomaly detection system based on machine vision, comprising:

[0042] The acquisition module uses a pre-trained network structure of the DINO model to extract features from the acquired screw and safety rope images. It extracts image features through self-supervised learning and knowledge distillation to generate a core set.

[0043] The filtering module uses a greedy selection algorithm to sample and filter the core set.

[0044] The first extraction module uses the Kmeans algorithm to cluster the filtered feature set, and then uses a fully connected Gaussian conditional random field, threshold segmentation and area filtering algorithm to extract the safety rope image.

[0045] The second extraction module uses the SAM large model and the obtained safety rope image as input to the prior prompt box to complete the extraction of the screw image;

[0046] The detection module uses positional analysis to determine the installation of the safety rope and screw based on images of the safety rope and screw, thus enabling assembly anomaly detection.

[0047] Thirdly, a computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of the aforementioned machine vision-based automated assembly anomaly detection method.

[0048] Fourthly, embodiments of the present invention provide a computer-readable storage medium including a computer program, which, when executed by a processor, implements the steps of the above-described automated assembly anomaly detection method based on machine vision.

[0049] Compared with the prior art, the present invention has at least the following beneficial effects:

[0050] An automated assembly anomaly detection method based on machine vision is disclosed. According to claim 1, the entire detection method aims to achieve automated assembly anomaly detection based on machine vision. The advantage of this method lies in its ability to extract image features and generate a core set through a pre-trained DINO model network structure, combined with self-supervised learning and knowledge distillation. This enables accurate extraction and analysis of screw and safety rope images. Such a design effectively detects and judges assembly anomalies, improving assembly accuracy and efficiency.

[0051] Furthermore, step S1 is designed to extract features from the acquired screw and safety rope images using a pre-trained DINO model network structure. The purpose of this step is to extract image features through self-supervised learning and knowledge distillation to generate a core set. This feature extraction process effectively identifies and extracts key features from the screw and safety rope images, providing a reliable foundation for subsequent safety rope image extraction and analysis. This method improves the accuracy and efficiency of image processing, thereby achieving the goal of automated assembly anomaly detection.

[0052] Furthermore, step S2 is designed to sample the core set using a greedy selection algorithm, filtering and clustering the core set to extract the safety rope image. The purpose of this step is to effectively sample the core set using a greedy selection algorithm to filter and cluster it, thereby achieving accurate extraction of the safety rope image. The greedy selection algorithm can choose the best option at each step according to specific criteria, gradually constructing the optimal solution. This method effectively improves the efficiency of core set filtering and clustering, thus improving the accuracy and efficiency of safety rope image extraction.

[0053] Furthermore, step S3 aims to cluster the feature set filtered in step S2 using the K-means algorithm, and then extract the safety rope image using a fully connected Gaussian conditional random field, threshold segmentation, and area filtering algorithm. The purpose of this step is to use the K-means algorithm to cluster the feature set, achieving effective classification and grouping of features, thereby improving the extraction of the safety rope image. The application of the fully connected Gaussian conditional random field, threshold segmentation, and area filtering algorithm can further optimize and improve the image segmentation process, enhancing the accuracy and stability of the extraction results. Through the comprehensive application of these algorithms, accurate extraction of the safety rope image can be effectively achieved, providing a reliable foundation for subsequent assembly anomaly detection.

[0054] Furthermore, step S4 is designed to utilize the SAM large model, including the image encoder, cue encoder, and lightweight mask decoder, to achieve image feature extraction and segmentation result prediction. The purpose of this step is to achieve efficient feature extraction and accurate segmentation of the image through the collaborative work of the various components in the SAM large model, thereby improving the accuracy and stability of assembly anomaly detection. The image encoder extracts image features through convolutional and pooling layers, the cue encoder provides cue information through prior bounding box coordinates, and the lightweight mask decoder combines this information to predict the segmentation result. Through the comprehensive application of the SAM large model, effective segmentation and anomaly detection of assembly images can be achieved, improving the system's detection accuracy and efficiency.

[0055] Furthermore, step S5 aims to determine the correctness of the assembly by judging the relationship between the screw position and the minimum circumscribed rectangle of the safety rope. The purpose of this step is to verify the correctness of the assembly through positional relationship judgment, ensuring that the screw position meets the requirements of the safety rope. If the screw position meets the conditions, it is considered correctly configured; otherwise, it is considered incorrect. The principle behind this step is to achieve rapid detection and judgment of the assembly process through simple positional relationship judgment, improving assembly accuracy and efficiency.

[0056] Furthermore, the purpose of introducing an anomaly detection mechanism during feature extraction and analysis is to promptly trigger alarms when data deviations or feature set anomalies are detected, prompting operators to make adjustments. This mechanism improves the system's automated monitoring efficiency and ensures assembly quality. The principle analysis involves monitoring anomalies in feature clustering results, core feature set selection, and positional relationship analysis to determine if assembly errors exist. When anomalies are detected, the system can issue timely alarms, enabling operators to take prompt action and ensure the accuracy and stability of the assembly process.

[0057] It is understood that the beneficial effects of the second and third aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here.

[0058] In summary, this invention introduces an anomaly detection mechanism during feature extraction and analysis, enabling timely monitoring and alerts for data deviations or feature set anomalies, thereby improving assembly quality and automated monitoring efficiency. Furthermore, the system can accurately segment and distinguish screw and safety rope images, enhancing the efficiency and reliability of automated detection and effectively improving the accuracy and stability of the assembly process.

[0059] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0060] Figure 1 This is a flowchart of the present invention;

[0061] Figure 2 Here is a diagram of the DINO network model structure;

[0062] Figure 3 The images show a comparison of safety rope segmentation diagrams with and without DenseCRF treatment, where (a) is a safety rope segmentation diagram without DenseCRF treatment and (b) is a safety rope segmentation diagram with DenseCRF treatment.

[0063] Figure 4 Images were extracted for the safety rope, where (a) is the original image of the screw safety rope and (b) is the image of the screw safety rope with extracted features.

[0064] Figure 5 The SAM model structure;

[0065] Figure 6 The images show the results of screw segmentation, where (a) is the segmentation of the left screw and (b) is the segmentation of the right screw.

[0066] Figure 7 Flowchart for assembly anomaly detection;

[0067] Figure 8 The diagrams are for judging whether the safety rope is correctly installed and the center coordinates are correct. (a) is a diagram for judging the center coordinates of the safety rope that is correctly installed, and (b) is a diagram for judging the center coordinates of the safety rope that is incorrectly installed.

[0068] Figure 9 The diagrams show the correct and incorrect installation of the screw safety rope, where (a) shows the correct installation and (b) shows the incorrect installation.

[0069] Figure 10 A schematic diagram of a computer device provided in an embodiment of the present invention;

[0070] Figure 11 This is a block diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0071] This invention provides an automated assembly anomaly detection method based on machine vision. Utilizing a pre-trained DINO model network structure, it extracts features from acquired screw and safety rope images. This includes scaling the images and inputting them into the DINO model, extracting image features through self-supervised learning and knowledge distillation to generate a core set. Next, the core set is filtered and clustered to complete the extraction of the safety rope images. A greedy selection algorithm is used to sample the core set, reducing data inference time and storage space, thus improving algorithm efficiency and performance. Then, the filtered feature set is clustered using the K-means algorithm, and the safety rope images are extracted using a fully connected Gaussian conditional random field, threshold segmentation, and area filtering algorithm. Subsequently, using the SAM large model technique and the extracted safety rope images, a priori prompt box is input to complete the extraction of screw images. Finally, based on the extracted images, positional judgment knowledge is used to determine the correct installation of the screw and safety rope. Specifically, the geometric center of the minimum bounding rectangle of each screw image is calculated, and these center points are compared with the minimum bounding rectangle of the safety rope image. This study establishes a standard: if the centers of the two screws are located on opposite sides of the smallest bounding rectangle of the safety rope, the configuration is considered correct; otherwise, it is deemed incorrect. This scheme ensures accurate segmentation and discrimination of screw and safety rope images, improving the efficiency and reliability of automated detection.

[0072] This invention employs an unsupervised segmentation model to segment images into multiple components, which are then filtered and extracted according to predetermined rules. Unsupervised learning models often generate large feature sets during feature extraction because they lack explicit label guidance during training, thus tending to capture as much information as possible from the data to avoid missing any potentially important details. While this method can reveal the underlying structure of the data to some extent, it can also lead to a significant increase in feature dimensionality, thereby increasing computational load and storage requirements. In this context, large-scale model algorithms combining deep learning and feature selection techniques are particularly crucial. These algorithms, by designing more efficient network architectures and employing advanced data representation methods, not only maintain or improve performance but also effectively reduce the number of feature sets. For example, some algorithms utilize the hierarchical structure of deep learning to automatically filter and optimize features, retaining only the key features that contribute most to the prediction task. This not only improves the model's operating efficiency but also significantly reduces its size, making it more suitable for applications requiring rapid response, such as real-time data processing and applications on mobile devices. Advances in this type of technology are of great significance in improving the efficiency and operability of models.

[0073] This invention discloses an automated assembly anomaly detection method based on machine vision, aiming to improve the accuracy of feature recognition and anomaly judgment in screw and safety rope images. It employs two lightweight large models for processing and includes the following steps:

[0074] S1. By using a pre-trained data network, preliminary segmentation processing is performed on the screw safety rope images captured on the assembly line;

[0075] This invention uses the first module of the DINO large model to perform feature extraction.

[0076] S2. Apply a greedy selection algorithm to assist in the selection of the core feature set, and further cluster the features using the K-means clustering algorithm;

[0077] S3. In order to improve the detail and accuracy of feature extraction, the SAM large model was introduced to supplement and improve the image feature recognition process, so as to achieve independent extraction of screw and safety rope features.

[0078] S4. By analyzing the positional relationship of the extracted features, determine whether the installation of the screw safety rope meets the preset standards.

[0079] Furthermore, this invention introduces an anomaly detection mechanism during feature extraction and analysis. If data deviation or feature set anomalies are detected at any step, the system will trigger an alarm. Anomaly detection conditions include, but are not limited to:

[0080] 1) Significant biases in the feature clustering results indicate possible assembly errors;

[0081] 2) The core feature set failed to pass the greedy algorithm screening, indicating potential image damage or occlusion;

[0082] 3) Positional relationship analysis shows that the relative positions of the screw and the safety rope do not conform to the design specifications.

[0083] The accurate detection of these anomalies provides an important guarantee for ensuring assembly quality, thereby significantly improving the efficiency of automated monitoring on the production line.

[0084] Those skilled in the art will understand that various aspects of the present invention can be implemented as systems, methods, or program products. Therefore, various aspects of the present invention can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, collectively referred to herein as a "circuit," "module," or "platform."

[0085] In another embodiment of the present invention, an automated assembly anomaly detection system based on machine vision is provided. This system can be used to implement the above-mentioned automated assembly anomaly detection method based on machine vision. Specifically, the automated assembly anomaly detection system based on machine vision includes a data acquisition module, a screening module, a first extraction module, a second extraction module, and a detection module.

[0086] The acquisition module uses a pre-trained DINO model network structure to extract features from the acquired screw and safety rope images. It extracts image features through self-supervised learning and knowledge distillation to generate a core set.

[0087] The filtering module uses a greedy selection algorithm to sample and filter the core set.

[0088] The first extraction module uses the Kmeans algorithm to cluster the filtered feature set, and then uses a fully connected Gaussian conditional random field, threshold segmentation and area filtering algorithm to extract the safety rope image.

[0089] The second extraction module uses the SAM large model and the obtained safety rope image as input to the prior prompt box to complete the extraction of the screw image;

[0090] The detection module uses positional judgment knowledge to determine the installation of the screw and safety rope based on the images of the safety rope and screw, thereby realizing the detection of assembly abnormalities.

[0091] In another embodiment of the present invention, a terminal device is provided, comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions to achieve a corresponding method flow or corresponding function. The processor described in this embodiment of the present invention can be used for the operation of an automated assembly anomaly detection method based on machine vision, including:

[0092] Feature extraction is performed on the acquired screw and safety rope images using a pre-trained DINO model network structure. Image features are extracted through self-supervised learning and knowledge distillation to generate a core set. A greedy selection algorithm is used to sample the core set, and the core set is then filtered and clustered to complete the extraction of the safety rope images. The filtered feature set is clustered using the K-means algorithm, and then the fully connected Gaussian conditional random field, threshold segmentation, and area filtering algorithms are used to complete the extraction of the safety rope images. Using the SAM large model and the obtained safety rope images as input for a priori prompt box, the screw images are extracted. Based on the extracted safety rope and screw images, position judgment knowledge is used to determine the installation of the screw and safety rope, realizing assembly anomaly detection.

[0093] In another embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory), which is a memory device in a terminal device for storing programs and data. It is understood that the computer-readable storage medium here can include both built-in storage media in the terminal device and extended storage media supported by the terminal device; it can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by a processor, which can be one or more computer programs (including program code). It should be noted that more specific examples (a non-exhaustive list) of the computer-readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0094] Computer-readable storage media also include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable storage medium can also be any readable medium other than a readable storage medium that can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium can be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0095] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0096] One or more instructions stored in a computer-readable storage medium can be loaded and executed by a processor to implement the corresponding steps of the machine vision-based automated assembly anomaly detection method in the above embodiments; one or more instructions in the computer-readable storage medium are loaded and executed by the processor to perform the following steps:

[0097] Feature extraction is performed on the acquired screw and safety rope images using a pre-trained DINO model network structure. Image features are extracted through self-supervised learning and knowledge distillation to generate a core set. A greedy selection algorithm is used to sample the core set, and the core set is then filtered and clustered to complete the extraction of the safety rope images. The filtered feature set is clustered using the K-means algorithm, and then the fully connected Gaussian conditional random field, threshold segmentation, and area filtering algorithms are used to complete the extraction of the safety rope images. Using the SAM large model and the obtained safety rope images as input for a priori prompt box, the screw images are extracted. Based on the extracted safety rope and screw images, position judgment knowledge is used to determine the installation of the screw and safety rope, realizing assembly anomaly detection.

[0098] Please see Figure 10 The terminal device is a computer device. In this embodiment, the computer device 60 includes a processor 61, a memory 62, and a computer program 63 stored in the memory 62 and executable on the processor 61. When executed by the processor 61, the computer program 63 implements the fluid composition calculation method in the reservoir stimulation wellbore of this embodiment. To avoid repetition, these details are not elaborated here. Alternatively, when executed by the processor 61, the computer program 63 implements the functions of each model / unit in the fluid composition calculation system in the reservoir stimulation wellbore of this embodiment. To avoid repetition, these details are not elaborated here.

[0099] Computer device 60 can be a desktop computer, laptop, handheld computer, cloud server, or other computing device. Computer device 60 may include, but is not limited to, a processor 61 and a memory 62. Those skilled in the art will understand that... Figure 10 This is merely an example of computer device 60 and does not constitute a limitation on computer device 60. It may include more or fewer components than shown, or combine certain components, or different components. For example, computer device may also include input / output devices, network access devices, buses, etc.

[0100] The processor 61 may be a central processing unit (CPU), or other general-purpose processors, CPUs, graphics processing units (GPUs), digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, quantum computing-based data processing logic units, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0101] The memory 62 can be an internal storage unit of the computer device 60, such as a hard disk or RAM of the computer device 60. The memory 62 can also be an external storage device of the computer device 60, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc., equipped on the computer device 60.

[0102] Furthermore, the memory 62 may include both internal storage units of the computer device 60 and external storage devices. The memory 62 is used to store computer programs and other programs and data required by the computer device. The memory 62 can also be used to temporarily store data that has been output or will be output.

[0103] Any references to memory, databases, or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0104] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0105] Please see Figure 11 The terminal device 600 is an electronic device, which takes the form of a general-purpose computing device. The components of the electronic device may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different platform components (including storage unit 620 and processing unit 610), a display unit 640, etc.

[0106] The storage unit stores program code, which can be executed by the processing unit 610 to perform the steps described in the method section of this specification according to various exemplary embodiments of the present invention. For example, the processing unit 610 can perform actions such as... Figure 1 The steps are shown in the figure.

[0107] Storage unit 620 may include a readable medium in the form of a volatile storage unit, such as random access memory (RAM) 6201 and / or cache memory 6202, and may further include a read-only memory (ROM) 6203.

[0108] Storage unit 620 may also include a program / utility 6204 having a set (at least one) program module 6205, such program module 6205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0109] Bus 630 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the multiple bus structures.

[0110] Electronic device 600 can also communicate with one or more external devices 700 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 600, and / or with any device that enables electronic device 600 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 650. Furthermore, electronic device 600 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 660. Network adapter 660 can communicate with other modules of electronic device 600 via bus 630. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms.

[0111] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0112] A method for clustering and segmenting safety ropes using the K-means algorithm based on DINO (Self-distillationWithNoLabels) pre-trained features is described below:

[0113] First, a set of normal screw and safety rope images χ={x1,x2,...x i};

[0114] Then, following the strategy in the DINO network, the first block of the pre-trained DINOVisualTransformer-S / 8 model is used to compute each image. The intermediate features are represented as Where (H,W) and (I,J) represent spatial dimensions, and C and D represent channel dimensions;

[0115] For the original feature map f of each image i The present invention uses its feature vector Stored in memory library M i Then, a greedy selection algorithm is used to sample its core set;

[0116] After sampling, this invention further concatenates these core sets to construct a memory bank M, and then performs K-means clustering on these feature vectors in M ​​to obtain K clusters and their central features.

[0117] Then calculate the central feature f. Kmeans Compared with the original feature map f i The cosine similarity between the images is calculated, and then the similarity map is interpolated to the original image size to obtain a preliminary segmentation map.

[0118] Then, Benfenming uses a fully connected Gaussian Conditional Random Field (CRF) for post-processing, and obtains the safety rope segmentation map through image thresholding binarization and area thresholding.

[0119] VisionTransformer (ViT) is a deep learning model for computer vision tasks that introduces a self-attention mechanism into the vision domain. The core idea of ​​ViT is to segment the input image into fixed-size image patches and flatten the pixel values ​​of each image patch into a vector form as the input sequence.

[0120] Then, these sequences are processed by stacking Transformer blocks, thereby achieving the extraction and processing of image features. In the ViT model, a "block" refers to a basic component in the Transformer architecture, consisting of multiple layers, including multi-head self-attention layers, feedforward fully connected layers, residual connections, and layer normalization. These blocks are responsible for feature transformation and representation learning of the input sequences. The first block in ViT plays a special role. During the pre-training phase, by converting each image block of the image into a sequence and inputting it into the first block of the model, the global features and structural information of the image can be captured. This is because the first block usually contains a large receptive field and strong parameterization capabilities, enabling it to effectively learn global features in the image.

[0121] Therefore, the first block of the ViT model can effectively extract image features while preserving global information. DINO is an image feature extraction method based on self-supervised learning, which uses the information of the image itself to learn the feature representation of the image without requiring labeled data. By combining DINO and ViT, the ViT model is trained using self-supervised learning, thereby learning richer and more general feature representations on large-scale unlabeled image data. Introducing DINO into the ViT model makes ViT learn image feature representations more efficiently and effectively during training. By pre-training the DINOViT model, the parameters of the ViT model are learned using large-scale unlabeled image data, resulting in a more generalizable image feature extractor.

[0122] Therefore, this invention uses DINO to pre-train features on screw and safety rope images and utilizes the first block of the DINOViT model to extract image features. Furthermore, since this invention primarily targets industrial images, which exhibit a higher level of distribution consistency compared to natural images, and industrial environments have high requirements for detection efficiency, the model has been simplified to some extent. In this invention, image segmentation of screws and safety ropes is a key task. Therefore, the model should focus on specific industrial applications rather than overgeneralization. For example, in self-supervised DINO training, color enhancement is often used to improve the model's adaptability to various color changes. While this has its advantages in broad natural environments, as diverse colors often point to the same semantic category, ignoring subtle color changes in the industrial detection of screws and safety ropes can mask crucial anomalous signals. Given this scenario difference, this invention employs a DINO-based feature clustering and precise post-processing strategy to construct a highly targeted and efficient segmentation model.

[0123] Please see Figure 2 In the DINO network structure model, the input image x is a captured image of the screw safety rope, and g θsFor student networks, g θt For the teacher network, both networks will output by p s and p t The probability distribution in the K-dimensional dimension is represented by the probability P, which is obtained by normalizing the output of network g using the softmax function. The specific formula is as follows:

[0124]

[0125] Where, τ s >0 is a temperature parameter that controls the sharpness of the output distribution; a similar formula also applies to conditions with temperature τ. t P t Given a fixed teacher network g θt By minimizing the network parameters θ of the student s The relevant cross-entropy loss is used to learn to match these distributions.

[0126] Student Network θ s The parameters are:

[0127]

[0128] Where H(a,b)=-alogb.

[0129] Equation 2 is applied to the self-supervised learning of screw and safety rope images, as follows:

[0130] First, a multi-crop strategy is used to construct different distorted views or cropped images of the image, i.e., to generate a set of different perspectives V from a given image of a screw and safety rope.

[0131] Two global perspectives in this set and And several smaller local views at resolution f; where the global view is passed only through the teacher, and the cropped image is passed through the student; minimizing the loss, which is general and applicable to any number of views, even only two, to meet the needs of industrial production lines with few defective samples. However, following the standard multi-crop setup, a resolution of 224 is used. 2 Two global attempts were made to cover a large area (greater than 50%) of the original image, and several at a resolution of 96. 2 A local view, covering only a small region (less than 50%) of the original image. Both networks share the same architecture g but have different parameter sets θ. s and θ t This invention uses Equation 3, stochastic gradient descent minimization, to learn the parameter θ. s .

[0132] Loss function Ls Represented as:

[0133]

[0134] This invention constructs a core set from the collected features. The goal of selecting the core set is to find a subset S∈A from the existing screw and safety rope images, such that a smaller, less time-consuming solution is found on the subset S, and this solution has the highest similarity to the solution in the whole set A. This reduces the overall algorithm consumption and improves efficiency, meeting the efficiency requirements in actual production, while also providing robustness and generalization ability of the model.

[0135] The total dataset is M, and the core set covers approximately M. C Specifically, it is expressed as follows:

[0136]

[0137] To further reduce the time consumption of core set selection, this invention employs an iterative greedy core set selection algorithm. This algorithm reduces the dimensionality of m∈M through random linear projection. In each iteration, for each sample point outside the core set, the algorithm searches for the nearest core set and calculates the distance between them. This distance is used as the distance between the point and the core set. This strategy aims to preserve the local structure of the original dataset to the greatest extent possible, while ensuring that the feature distribution is similar to the original dataset. Then, the sample furthest from the core set is selected as part of the core samples to guarantee the diversity and representativeness of the core set. The entire process is specifically implemented through Equation 5:

[0138]

[0139] Where M is the set of total image features before core set sampling, M C Let n be the sampled core set, where n is M. C One of the samples, m is MM C One of the samples, m i This refers to the new core set samples found after each algorithm iteration.

[0140] In the implementation of the iterative greedy core set selection algorithm, the memory bank M and the core set length L are used as inputs. The core set length L is determined by the sampling rate r, i.e.

[0141] Through experiments, the same image of a screw and safety rope was divided into segments. From left to right, the features of each part of the safety rope image were clustered after selecting the core set according to different sampling rates. These segments were the shaft, screw, safety rope, and central platform region. When r = 0.1, the model did not segment the safety rope and screw thoroughly enough. However, when r = 0.01, the safety rope was segmented more effectively. After repeated experiments, r = 0.01 was found to be the best result.

[0142] After selecting the core set, the selected core sets are concatenated to construct a memory M′. Then, KMeans is executed to cluster the feature vectors in the memory M′, thereby obtaining K clusters and their central features f. Kmeans The KMeans algorithm is widely used in various scenarios, including image recognition, data mining, and machine learning, with its main advantages being simplicity and efficiency. This algorithm optimizes cluster quality through an iterative process, ultimately minimizing the distance between each data point and its corresponding centroid within each cluster. It effectively handles large datasets, is easy to implement, and converges quickly without requiring excessive hardware or long convergence times. In the detection of screws and safety ropes, KMeans can help identify different operating patterns or potential anomalies from large-scale detection data. By analyzing the centroid features of each cluster, specific data points that may require further investigation can be quickly located, facilitating subsequent extraction of target features. Furthermore, as an unsupervised learning method, it can organize and classify data without relying on predefined category labels. This is particularly important for objects like screws and safety ropes, as sufficient labeled samples are often difficult to obtain in practical applications. Cluster analysis reveals the inherent structure and patterns in the data, providing a basis for further analysis and decision-making.

[0143] In this invention, the KMeans clustering algorithm only requires adjusting the parameter k, the number of clusters. Experiments showed that for the screw safety rope sample, k=4 yielded the best overall results, effectively segmenting the safety rope image and providing a reliable basis for determining the correctness of the safety rope.

[0144] In the DINO network feature extraction process, the present invention causes a certain loss of image features, resulting in blurred boundaries of the classification targets. In order to optimize the coarse and uncertain labels in the classification image, correct the fragmented misclassified regions, and obtain more detailed segmentation boundaries, the present invention uses a fully connected Gaussian conditional random field (DenseCRF) to optimize the initial segmentation map of the screw safety rope.

[0145] Fully connected Gaussian Conditional Random Fields (DenseCRF) are a variant of dense conditional random fields, specifically designed for dense prediction tasks such as image segmentation. Compared to traditional CRFs, DenseCRFs place greater emphasis on modeling the spatial relationships between pixels to better capture the dependencies between neighboring pixels in an image.

[0146] Specifically, in a fully connected Gaussian conditional random field, "fully connected" means that there are connections between every pixel, that is, every pixel is associated with all other pixels in the image.

[0147] The term "Gaussian" refers to using a Gaussian kernel function to model the relationships between these pixels. In a fully connected Gaussian conditional random field, a pixel's label prediction is influenced not only by its own features but also by the features and labels of its surrounding pixels. This model can leverage the spatial locality information between pixels in an image, improving pixel-level label prediction accuracy by considering the correlations between neighboring pixels.

[0148] DenseCRF typically defines the relationship between pixel labels using an energy function. This energy function consists of two parts: a data term and a smoothing term.

[0149] The data term measures how well the predicted labels fit the observed data, while the smoothing term measures the consistency between adjacent pixel labels. In a fully connected Gaussian conditional random field, the smoothing term is typically represented by a Gaussian potential, which measures the similarity between pixels based on their distance, thus taking spatial relationships into account.

[0150] This invention uses DenseCRF for multiple iterations to implement CRF, and its specific pairwise potential function is shown in Equation 6:

[0151]

[0152] Among them, f i and f j It is a pixel node, p i and p j That's their corresponding position, I i and I j They are their corresponding colors, where w (1) w (2) θ α θ β θ γ These are the hyperparameters used in this formula.

[0153] Through experimentation, when the number of iterations was 2, w (1) =4, w (2) =3, θ α =67, θβ =3, θ γ When = 1, the experimental results are the best, such as Figure 3 As shown, after DenseCRF processing, the edges of the safety rope's segmentation diagram are clearer and more obvious.

[0154] Following DenseCRF processing, this invention employs a threshold binarization method to further clearly distinguish the safety rope from the background in the image. This step converts the image to a black and white format, where pixel values ​​within a preset threshold range [35, 255] are set to white (representing the safety rope), while other values ​​are converted to black (representing the background). This threshold range was selected based on experimental data to minimize misclassification and improve the accuracy of target detection.

[0155] Furthermore, to eliminate noise or non-target elements in the image, this invention introduces an area threshold filtering step. In this step, only connected regions with an area within the range of [1000, 3000] pixel squares are retained. This range is derived based on the analysis of multiple sample images to ensure that the retained regions are most likely to represent the actual safety rope. This method effectively eliminates excessively large or small image regions, improving the accuracy and reliability of the final image segmentation.

[0156] The image processing method of this invention combines DenseCRF optimization with precise post-processing steps, including threshold binarization and area thresholding, significantly improving the recognition accuracy and efficiency of safety rope images. The final safety rope extraction result is as follows: Figure 4 As shown.

[0157] The post-processing method for screw safety ropes based on the SegmentAnything (SAM) large model includes the following specific steps:

[0158] The above steps yielded an image of the screw and safety rope. However, to determine if the safety rope is correct, an image of the screw is still needed. The above method often mixes the screw image with the background image, making it impossible to accurately extract the screw image.

[0159] To address this issue, this invention utilizes the large model SAM. Given the known safety rope image, a priori bounding box is generated based on its minimum bounding rectangle and used as input to extract and visualize screw features, thereby enabling subsequent determination of whether the screw and safety rope images are correct.

[0160] SAM is a general instance segmentation model based on Transformer, designed to generate accurate instance segmentation masks for any input image and any location prompt. The core idea of ​​SAM is to utilize the self-attention mechanism of Transformer to capture the relationships between different objects in an image, thereby achieving more accurate instance segmentation.

[0161] Please see Figure 5 The architecture of SAM consists of three components that work together to return a valid segmentation mask.

[0162] The first is the image encoder, which uses a visual transformer pre-trained with MAE and minimized, consisting of multiple convolutional and pooling layers to extract image features. Each convolutional layer includes operations such as convolutional kernels, activation functions, and batch normalization to extract features and reduce dimensionality of the input image. The pooling layers are used to downsample the feature maps to reduce computation and memory consumption.

[0163] The second is the prompt encoder, in which the coordinates of the prior bounding box are used as the prompt input in this invention;

[0164] The third is a lightweight mask decoder, which predicts the segmentation mask by combining the information provided by the first two information sources. This part consists of multiple deconvolutional layers and upsampling layers, which are used to restore the feature map to the original image size and generate the segmentation result.

[0165] By using weights pre-trained on the SA-1B dataset, this invention directly applies the large SAM model to the task of segmenting screw and safety rope images. Using the input prior bounding box as a cue, it can directly segment two screw parts that are difficult to segment effectively, as shown in the results. Figure 6 As shown.

[0166] Threshold binarization segmentation is used to segment an image into representative binary images based on pixel values, thereby clearly distinguishing the target object from the background and effectively segmenting the safety rope image and screw image.

[0167] After successful segmentation, this invention applies the Minimum Bounding Rectangle (MBR) algorithm to each target object. The MBR algorithm calculates the smallest rectangular region that can cover each segmented object. This step is crucial for determining the geometric position and size of the objects. The minimum bounding rectangle of each object is mapped back into the original image for further spatial analysis.

[0168] Spatial location analysis is primarily based on the center position of the screw images. By calculating the geometric center of the minimum bounding rectangle of each screw image and comparing these center points with the minimum bounding rectangle of the safety rope image, this invention establishes a standard:

[0169] If the centers of the two screws are located on opposite sides of the smallest circumscribed rectangle of the safety rope, the configuration is considered correct; otherwise, it is considered incorrect.

[0170] This standard is based on the practical application needs of safety ropes and screws in vision systems. For example, in safety inspections or assembly work, ensuring the correct position of screws is crucial for overall safety.

[0171] Please see Figure 7 The specific implementation process is as follows:

[0172] 1) Based on the previous image segmentation and feature extraction results, obtain the specific segmentation images of the screw and safety rope;

[0173] 2) Perform image preprocessing operations, including noise reduction and contrast enhancement, in order to more accurately identify the outlines of screws and fuses;

[0174] 3) Identify the outlines of the screws and safety rope, and calculate their minimum bounding rectangle;

[0175] 4) Calculate the geometric center (x) of the minimum circumscribed rectangle for each screw and safety rope according to Formula 7. c ,y c );

[0176]

[0177] Where x1 and y1 are the x and y coordinates of a vertex of the minimum bounding rectangle on the original image, and x2 and y2 are the x and y coordinates of the vertex diagonally opposite to that vertex.

[0178] 5) Compare the positional relationship between the geometric centers of the two screws and the geometric center of the safety rope;

[0179] The comparison rule follows the formula below:

[0180]

[0181] If the product of x and y is positive, the safety rope is installed correctly. If the product of x and y is negative, the safety rope is installed incorrectly. If either x or y is 0, it is considered to be installed correctly.

[0182] like Figure 8As shown, the left side is an example of a correctly installed safety rope, where the center coordinates of the minimum circumscribed rectangles of the two screws are located on both sides of the center coordinates of the minimum circumscribed rectangle of the safety rope, which meets the criteria for correct installation. The right side is an example of an incorrectly installed safety rope, where the center coordinates of the minimum circumscribed rectangles of the two screws are located on the same side as the center coordinates of the minimum circumscribed rectangle of the safety rope, which meets the criteria for incorrect installation.

[0183] 6) Mark the workpiece for correctness based on the result of step 5). If it is determined to be an incorrect configuration, mark the assembly and generate an anomaly report to prompt the operator to make adjustments.

[0184] In summary, the present invention provides an automated assembly anomaly detection method and system based on machine vision. Through steps such as feature extraction, filtering, clustering, and position determination, it achieves accurate segmentation of screw and safety rope images and discrimination of installation position relationships, thereby improving the efficiency and reliability of automated detection.

[0185] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0186] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0187] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0188] In the embodiments provided by this invention, it should be understood that the disclosed devices / terminals and methods can be implemented in other ways. For example, the device / terminal embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0189] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0190] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0191] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0192] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0193] The above content is only for illustrating the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solution based on the technical concept proposed in this invention shall fall within the scope of protection of the claims of this invention.

Claims

1. A machine vision-based automated assembly anomaly detection method, characterized in that, Includes the following steps: S1. The network structure pre-trained by the DINO model is used to extract features from the collected screw and safety rope images. Image features are extracted through self-supervised learning and knowledge distillation to generate a core set. S2. Use a greedy selection algorithm to sample the core set, filter and aggregate the core set to complete the extraction of the safety rope image; S3. Cluster the feature set filtered in step S2 using the K-means algorithm, and then extract the safety rope image using a fully connected Gaussian conditional random field, threshold segmentation, and area filtering algorithm. Specifically: The obtained core sets are concatenated to construct a memory bank. Perform K-means on the memory Clustering is performed on the feature vectors in the data to obtain K clusters and their central features. ; Computational center characteristics Compared with the original feature map The cosine similarity between the images is calculated, and then the similarity map is interpolated to the original image size to obtain a preliminary segmentation map. Post-processing was performed using a fully connected Gaussian conditional random field, and the safety rope segmentation map was obtained through image threshold binarization and area thresholding. ; S4. Using the SAM large model and the safety rope image obtained in step S3, input the prior prompt box to complete the extraction of the screw image. S5. Based on the safety rope and screw images extracted in steps S3 and S4, use position judgment knowledge to determine the installation of the screw and safety rope, thereby realizing assembly anomaly detection.

2. The automated assembly anomaly detection method based on machine vision according to claim 1, characterized in that, Step S1 is as follows: Acquire a set of images of normal screws and safety ropes Following the strategy of the DINO network structure model, the pre-trained DINOVisualTransformer-S / 8 model is used to compute the results for each image. Original feature map , original feature map eigenvectors Stored in memory library middle.

3. The automated assembly anomaly detection method based on machine vision according to claim 1, characterized in that, In step S2, the sampling of the core set using the greedy selection algorithm is specifically as follows: Where M is the set of total image features before core set sampling. Let n be the sampled core set. One of the samples, m is One of the samples, This refers to the new core set samples found after each algorithm iteration.

4. The automated assembly anomaly detection method based on machine vision according to claim 1, characterized in that, The CRF is implemented using multiple iterations of DenseCRF. The specific paired potential function is as follows: in, and It is a pixel node. and These are their corresponding positions. and They are their corresponding colors. , , , , This is a hyperparameter.

5. The automated assembly anomaly detection method based on machine vision according to claim 1, characterized in that, In step S4, the large SAM model includes: The image encoder, consisting of multiple convolutional and pooling layers, is designed to extract image features by using a visual transformer pre-trained with MAE and minimizing it. The prompt encoder uses the coordinates of the prior bounding box as prompt input; The lightweight mask decoder predicts the segmentation mask by combining information from the image encoder and the cue encoder, restores the feature map to the original image size, and generates the segmentation result.

6. The automated assembly anomaly detection method based on machine vision according to claim 1, characterized in that, In step S5, if the centers of the two screws are located on both sides of the smallest circumscribed rectangle of the safety rope, the configuration is considered correct; otherwise, it is considered incorrect.

7. The automated assembly anomaly detection method based on machine vision according to claim 6, characterized in that, The process of extracting the screw image is as follows: Based on the image segmentation and feature extraction results, obtain the specific segmentation images of the screw and safety rope; Image preprocessing is performed on the segmented images of the screw and safety rope to identify the contours of the screw and fuse, and to calculate the geometric center of the minimum bounding rectangle of the screw and safety rope contours. ; Compare the positional relationship between the geometric centers of the two screws and the geometric center of the safety rope; if the product of x and y is positive, the safety rope is correctly installed; if the product of x and y is negative, the safety rope is incorrectly installed; if either x or y is 0, it is a normal installation. Based on the judgment results, mark the workpiece as correct or incorrect. If it is determined to be an incorrect configuration, mark the assembly and generate an anomaly report to prompt the operator to make adjustments.

8. The automated assembly anomaly detection method based on machine vision according to claim 1, characterized in that, In steps S3 and S4, an anomaly detection mechanism is introduced during feature extraction and analysis. An alarm is triggered when data deviation or feature set anomalies are detected. The specific anomaly detection conditions are as follows: Significant deviations in the feature clustering results indicate assembly errors; The core feature set failed to pass the greedy algorithm screening, indicating potential image damage or occlusion; Positional relationship analysis shows that the relative positions of the screw and the safety rope do not conform to the design specifications.

9. An automated assembly anomaly detection system based on machine vision, characterized in that, include: The acquisition module uses a pre-trained network structure of the DINO model to extract features from the acquired screw and safety rope images. It extracts image features through self-supervised learning and knowledge distillation to generate a core set. The filtering module uses a greedy selection algorithm to sample the core set and then filters the core set. The first extraction module clusters the filtered feature set using the K-means algorithm, and then uses a fully connected Gaussian conditional random field, threshold segmentation, and area filtering algorithm to extract the safety rope image. Specifically: The obtained core sets are concatenated to construct a memory bank. Perform K-means on the memory Clustering is performed on the feature vectors in the data to obtain K clusters and their central features. ; Computational center characteristics Compared with the original feature map The cosine similarity between the images is calculated, and then the similarity map is interpolated to the original image size to obtain a preliminary segmentation map. Post-processing was performed using a fully connected Gaussian conditional random field, and the safety rope segmentation map was obtained through image threshold binarization and area thresholding. ; The second extraction module uses the SAM large model and the obtained safety rope image as input to the prior prompt box to complete the extraction of the screw image; The detection module uses positional judgment knowledge to determine the installation of the screw and safety rope based on the images of the safety rope and screw, thereby realizing the detection of assembly abnormalities.

Images