An assembly quality inspection method based on deep learning and object matching

Abstract

The application relates to an assembly quality inspection method based on deep learning and object matching, which comprises the following steps: acquiring an assembly site image to be measured; inputting the assembly site image to be measured into a trained complete lightweight neural network to determine predicted key point information and a predicted positioning frame; determining a conversion relationship between the predicted positioning frame and a standard positioning frame in a standard assembly template according to the predicted key point information; and matching the predicted positioning frame and the standard positioning frame to judge matching quality. The application can check assembly missing and misassembly problems in real time, ensure consistent product quality, and improve assembly efficiency.

Description

Technical Field

[0001] This invention relates to the field of complex product assembly technology, and in particular to an assembly quality inspection method based on deep learning and object matching. Background Technology

[0002] Assembly is a crucial process in manufacturing. In modern industrial production, the high complexity and variability of products significantly increase the probability of assembly errors, necessitating quality inspection of assembled products. Traditional assembly inspection methods involved manually comparing the assembled product with a two-dimensional process manual or a three-dimensional model to check for missing parts and whether the installation type, position, and angle of parts met assembly requirements. Because assembly parts are often of uniform color and diverse types, assembly quality inspection was a time-consuming and labor-intensive task. Furthermore, previous assembly inspection processes lacked monitoring and control of each assembly step. When quality problems such as omissions or misassemblies occurred, the assembled object needed to be disassembled again, severely impacting assembly efficiency and quality.

[0003] Deep learning has achieved remarkable results in image processing, and convolutional neural network-based methods are increasingly being applied to assembly quality inspection. However, the similar colors and lack of texture of mechanical parts pose challenges to feature extraction, localization, and recognition. Therefore, previous convolutional-based methods typically use large, deep, and complex networks with high hardware resource requirements, making algorithm deployment difficult. Furthermore, because the relative poses of the camera and parts may change during assembly through rotation or translation, previous assembly inspection methods suffer from sensitivity to camera viewpoint and poor robustness. Therefore, how to achieve accurate, efficient, and rapid product assembly monitoring is an urgent problem to be solved. Summary of the Invention

[0004] In view of this, it is necessary to provide an assembly quality inspection method based on deep learning and object matching to overcome the problem of difficulty in efficiently monitoring product assembly quality in existing technologies.

[0005] To address the aforementioned technical problems, this invention provides an assembly quality inspection method based on deep learning and object matching, comprising:

[0006] Acquire images of the assembly site to be tested;

[0007] The assembly site image to be tested is input into a fully trained lightweight neural network to determine the key point information and the predicted location box.

[0008] Based on the predicted key point information, determine the conversion relationship between the predicted positioning frame and the standard positioning frame in the standard assembly template;

[0009] The predicted bounding box and the standard bounding box are matched according to the transformation relationship, and the matching quality is judged.

[0010] Furthermore, the determination of the standard positioning frame includes: using a virtual camera to collect the standard assembly template for each assembly step, and marking the corresponding standard key point information and standard positioning frame.

[0011] Furthermore, the training process of the lightweight neural network includes:

[0012] Obtain an assembly image sample set containing annotation information, wherein the annotation information includes actual key point information and actual positioning box corresponding to each assembly site image in the assembly image sample set;

[0013] The assembly image sample set is input into the constructed lightweight neural network to determine the predicted key point information and the predicted localization box, and the loss function is determined based on the actual key point information, the actual localization box, the predicted key point information and the predicted localization box.

[0014] Adjust the parameters of the lightweight neural network according to the value of the loss function until the convergence condition is met, and determine the fully trained lightweight neural network.

[0015] Furthermore, the constructed lightweight neural network includes an encoder and a decoder, wherein the encoder is used to extract local and global features from the assembly site image, and the decoder is used to determine the predicted key point information and the predicted location box based on the extracted features.

[0016] Furthermore, the encoder includes multiple first convolutional structures and second convolutional structures connected in sequence, wherein:

[0017] The first convolutional structure is used to extract local features from the assembly site image using depthwise separable convolution;

[0018] The second convolutional structure is used to extract global features from the assembly site image using a Transformer structure.

[0019] Furthermore, the decoder includes a convolutional sampling layer, a keypoint detection structure, and a part detection structure, wherein:

[0020] The convolutional sampling layer is used to perform multiple convolutions and samplings on the extracted features output by the encoder, and output convolutional features.

[0021] The key point detection structure is used to detect key points based on the convolutional features and output predicted key point information.

[0022] The part detection structure is used to detect parts based on the convolutional features and output predicted localization boxes.

[0023] Furthermore, the key point detection structure includes a first branch structure and a second branch structure, wherein:

[0024] The first branch structure is used to regress the key point heatmap based on the convolutional features;

[0025] The second branch structure is used to perform regression on the coordinate offset of each key point as compensation based on the convolutional features, and output predicted key point information.

[0026] Furthermore, the part detection structure includes a first output branch, a second output branch, and a third output branch, wherein:

[0027] The first output branch is used to output the center point of the predicted localization box based on the convolutional features;

[0028] The second output branch is used to output the width and height of the predicted localization box based on the convolutional features;

[0029] The third output branch is used to output the angle information of the predicted localization box based on the convolution features.

[0030] Further, determining the conversion relationship between the predicted positioning frame and the standard positioning frame in the standard assembly template based on the predicted key point information includes:

[0031] Using the pixel coordinates of the predicted key point information as the center, a pixel region within a preset radius is selected as the feature region;

[0032] A two-dimensional Gaussian distribution is used to generate regional values ​​within the feature region, thereby determining the two-dimensional Gaussian probability distribution.

[0033] Based on the two-dimensional Gaussian probability distribution, the conversion relationship between the predicted positioning frame and the standard positioning frame in the standard assembly template is determined.

[0034] Further, the step of matching the predicted bounding box and the standard bounding box to determine the matching quality includes:

[0035] Based on the area and number of pixels of the predicted bounding box and the standard bounding box, perform IoU matching on the predicted bounding box and the standard bounding box to determine the IoU matching value;

[0036] If the IoU matching value meets the preset conditions, then there is a quality problem in the current assembly step.

[0037] Compared with existing technologies, the beneficial effects of this invention include: First, using the assembly site image to be tested as input, it simultaneously predicts key points and positioning bounding boxes in the assembly site image; second, using the key point prediction results, it solves the transformation relationship between the on-site assembly image and the standard template; finally, based on the prediction results of the assembly part positioning boxes, it performs intersection-over-union (IoU) matching between the detected part positioning boxes and the positioning boxes in the standard assembly template to assess whether there are quality problems in the current step. In summary, this invention, by collecting assembly site images on-site and inputting them into a trained lightweight network, predicts key points and part positioning boxes in the images, matches them with the standard assembly template, and then determines missing and incorrect assembly problems. This allows for real-time checking of missing and incorrect assembly problems, ensuring consistent product quality and improving assembly efficiency. Attached Figure Description

[0038] Figure 1 This is a flowchart illustrating an embodiment of the assembly quality inspection method based on deep learning and object matching provided by the present invention.

[0039] Figure 2 A schematic diagram of an embodiment of the standard assembly template image provided by the present invention;

[0040] Figure 3 A schematic flowchart illustrating an embodiment of the training process of the lightweight neural network provided by the present invention;

[0041] Figure 4 A schematic diagram of an embodiment of the lightweight neural network structure provided by the present invention;

[0042] Figure 5 This is a schematic diagram of an embodiment of the Vit Block network structure provided by the present invention;

[0043] Figure 6 A schematic diagram of a structure for an embodiment of the angle information provided by the present invention;

[0044] Figure 7 Provided by the present invention Figure 1 A flowchart illustrating an embodiment of step S103;

[0045] Figure 8 Provided by the present invention Figure 1 A flowchart illustrating an embodiment of step S104;

[0046] Figure 9 A schematic diagram of an embodiment of IoU matching provided by the present invention;

[0047] Figure 10 A schematic diagram of an embodiment of the assembly quality inspection device based on deep learning and object matching provided by the present invention;

[0048] Figure 11 A schematic diagram of the structure of an embodiment of the electronic device provided by the present invention. Detailed Implementation

[0049] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0050] In the description of this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. Furthermore, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0051] In the description of this invention, reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the described embodiments can be combined with other embodiments.

[0052] This invention provides an assembly quality inspection method based on deep learning and object matching. By combining a lightweight neural network, it simultaneously determines the predicted key point information and the predicted bounding box, and judges the matching quality, providing a new approach for further achieving efficient assembly quality inspection.

[0053] Before describing the embodiments, the relevant terms are defined as follows:

[0054] An autoencoder (AE) is a neural network that uses backpropagation to make the output equal to the input. It first compresses the input into a latent space representation, and then reconstructs the output using this representation. Simply put, it's a network model that can automatically perform encoding and decoding operations, and its input x is the same as its label y. The autoencoder model is optimized by minimizing the error between the label y and the output r.

[0055] Encoder: A part of autoencoder, it can compress the input into a latent space representation, which can be represented by the encoding function h = f(x);

[0056] The decoder, a part of the autoencoder, reconstructs the input from the latent space representation and can be represented by the decoding function r = g(h).

[0057] Based on the above technical terms, existing assembly inspection methods suffer from drawbacks such as sensitivity to camera viewpoint, poor robustness, and high hardware resource requirements, making algorithm deployment difficult. Furthermore, the relative pose of the camera and parts may change due to rotation or translation during assembly. Therefore, this invention aims to propose an efficient and accurate visual inspection method for assembly quality to solve the aforementioned problems.

[0058] The specific embodiments are described in detail below:

[0059] This invention provides an assembly quality inspection method based on deep learning and object matching, combined with... Figure 1 Let's take a look. Figure 1 A flowchart illustrating an embodiment of the assembly quality inspection method based on deep learning and object matching provided by the present invention includes steps S101 to S104, wherein:

[0060] In step S101, an image of the assembly site to be tested is acquired;

[0061] In step S102, the assembly site image to be tested is input into a fully trained lightweight neural network to determine the predicted key point information and the predicted positioning box.

[0062] In step S103, the conversion relationship between the predicted positioning frame and the standard positioning frame in the standard assembly template is determined based on the predicted key point information.

[0063] In step S104, the predicted localization box and the standard localization box are matched according to the transformation relationship, and the matching quality is judged.

[0064] In this embodiment of the invention, firstly, the assembly site image to be tested is used as input, and key points and positioning box information in the assembly site image are predicted simultaneously; secondly, the conversion relationship between the on-site assembly image and the standard template is solved using the key point prediction results; finally, based on the prediction results of the assembly part positioning boxes, the detected part positioning boxes are matched with the positioning boxes in the standard assembly template based on the intersection-union ratio (IoU) to evaluate whether there are quality problems in the current step.

[0065] As a preferred embodiment, combined with Figure 2 Let's take a look. Figure 2 This is a schematic diagram of an embodiment of the standard assembly template image provided by the present invention. The determination of the standard positioning frame includes: using a virtual camera to collect the standard assembly template under each assembly step, and marking the corresponding standard key point information and standard positioning frame.

[0066] In this embodiment of the invention, a virtual camera is used in CAD to collect the standard assembly template for each assembly step, and key point information and standard positioning frame information of assembly parts are marked.

[0067] In a specific embodiment of the present invention, the above method mainly includes two stages: an offline preparation stage and an online inspection stage. In the offline preparation stage, a standard assembly state template and a neural network training dataset are established for assembly quality inspection. The training dataset contains key point coordinate information for solving the viewpoint differences between the standard assembly template and on-site images, as well as bounding box information for locating assembly parts. During the construction of the dataset, images of the assembly site are captured using a camera in a real assembly scenario, key point information and part bounding box information are labeled, and data augmentation is performed on the images.

[0068] As a preferred embodiment, combined with Figure 3 Let's take a look. Figure 3 This is a flowchart illustrating an embodiment of the training process of the lightweight neural network provided by the present invention. The training process of the lightweight neural network includes steps S301 to S303, wherein:

[0069] In step S301, an assembly image sample set containing annotation information is obtained, wherein the annotation information includes the actual key point information and actual positioning box corresponding to each assembly site image in the assembly image sample set;

[0070] In step S302, the assembly image sample set is input into the constructed lightweight neural network to determine the predicted key point information and the predicted localization box, and a loss function is determined based on the actual key point information, the actual localization box, the predicted key point information, and the predicted localization box.

[0071] In step S303, the parameters of the lightweight neural network are adjusted according to the value of the loss function until the convergence condition is met, and the fully trained lightweight neural network is determined.

[0072] In this embodiment of the invention, a lightweight neural network is effectively trained.

[0073] In a specific embodiment of the present invention, the collection of standard assembly templates and the creation of the CNN-Transformer training dataset are as follows:

[0074] In terms of template image acquisition, the virtual camera in CAD is used to acquire images of each assembly step under standard assembly conditions, and the categories, coordinates, and standard positioning boundary box information of key points and assembly parts are marked.

[0075] For the training dataset, images of on-site assembly were collected as the training set. These images were taken against a realistic assembly background, at different angles and under varying lighting conditions. Each training image was annotated with keypoints and bounding box information for the parts. Keypoint information included coordinates and category, while bounding box information included center point, width, height, and rotation angle. To improve the model's generalization ability, four data augmentation methods were employed: image rotation, image flipping, adding noise, and color space transformation.

[0076] As a preferred embodiment, combined with Figure 4 Let's take a look. Figure 4 This is a schematic diagram of an embodiment of the lightweight neural network structure provided by the present invention. The constructed lightweight neural network includes an encoder and a decoder. The encoder is used to extract local and global features from the assembly site image, and the decoder is used to determine the predicted key point information and the predicted positioning box based on the extracted features.

[0077] In this embodiment of the invention, a lightweight neural network CNN-Transformer is designed for detecting key points of an assembly and part positioning boxes. The lightweight neural network includes an encoding part and a decoding part. The encoding part is an image feature extraction network part, and the decoding part includes a key point detection head and a part detection head.

[0078] In a specific embodiment of the present invention, a novel lightweight neural network model, CNN-Transformer, is designed, wherein the CNN-Transformer employs an encoder-decoder structure. The encoder part is the backbone feature extraction network used to extract features from the assembly image. The detection head part includes a keypoint detection head for solving the camera viewpoint transformation relationship, and a part detection head for locating and identifying the assembly components.

[0079] As a preferred embodiment, it still combines Figure 4 The encoder comprises multiple first convolutional structures and second convolutional structures connected in sequence, wherein:

[0080] The first convolutional structure is used to extract local features from the assembly site image using depthwise separable convolution;

[0081] The second convolutional structure is used to extract global features from the assembly site image using a Transformer structure.

[0082] In this embodiment of the invention, the encoding part consists of Conv Block and Vit Block. Conv Block is responsible for extracting local feature information in the image, and Vit Block is used to capture global feature information of the image.

[0083] In a specific embodiment of the present invention, combined with Figure 5 Let's take a look. Figure 5 This is a schematic diagram of an embodiment of the Vit Block network structure provided by the present invention. The encoding part is composed of stacked Vit Blocks and Conv Blocks. The Conv Block and Vit Block use depthwise separable convolution and Transformer structures to learn local and global features in the image, respectively.

[0084] The Conv Block employs depthwise separable convolution. For a feature map with an input size of H×W×C, it first undergoes a 1×1 point convolution to map the feature map to a higher dimension H×W×4C. Then, a 3×3 depthwise separable convolution is used to extract local features from the image. In this convolution, the stride is controlled to determine whether to downsample the feature map. Finally, a 1×1 point convolution maps the channel dimensions back to the input dimension. In the encoding part, the Conv Block is responsible for both feature extraction and downsampling of the feature map.

[0085] Specifically, for the Vit Block, a Transformer structure is used to perform global processing on the input feature map, replacing the local processing in the convolution operation. For example... Figure 3 As shown, for the H×W×C feature map input into the Vit Block, a patch embedding operation is first performed on the feature map. A 1×1 point convolutional layer is used to map the feature map's dimension to a higher dimension H×W×d by performing multiple linear combinations on the input channels. Each H×W feature map is divided into N h×w blocks. These blocks are then expanded to obtain a feature map with a dimension of (h×w)×N×d, thus implementing the patch embedding operation. The feature map is then input into the Transformer structure along the h×w dimension for feature extraction.

[0086] The Transformer encoder consists of a multi-head self-attention module (MSA) and a feed-forward module. Layer Norm is applied before each module, and residual connections and layer normalization are applied after each block. The feed-forward block is a multilayer perceptron module that maps the feature map to four times its original dimension and then back to the original dimension. For the self-attention module, for each input feature map with dimension N×d, three quantities, Q (query), K (key), and V (value), are obtained by multiplying with three learnable matrices. The final self-attention output is then calculated according to the following formula.

[0087]

[0088] To enable the Transformer to attend to information from different subspaces and capture richer feature information, a multi-head attention mechanism is used. Multiple sets of learnable matrices Wq, Wk, and Wv are multiplied with the input feature map. Finally, the results are concatenated and projected onto the input dimension through a fully connected layer.

[0089] MSA(Q,K,V)=Concate(head 1, head2,…,head h W o

[0090] In this process, w×h feature maps of dimension N×d are input into the Transformer structure for processing. The resulting feature map still has dimensions (h×w)×N×d. The feature map is then folded into H×W×d and projected back to the original H×W×C dimensions through a 1×1 point convolution.

[0091] As a preferred embodiment, it still combines Figure 4 The decoder includes a convolutional sampling layer, a keypoint detection structure, and a part detection structure, wherein:

[0092] The convolutional sampling layer is used to perform multiple convolutions and samplings on the extracted features output by the encoder, and output convolutional features.

[0093] The key point detection structure is used to detect key points based on the convolutional features and output predicted key point information.

[0094] The part detection structure is used to detect parts based on the convolutional features and output predicted localization boxes.

[0095] In this embodiment of the invention, key point detection and part detection output heads are set according to the detection task. Weight parameters are shared in the encoding part and upsampling part is shared in the decoding part, forming a multi-task learning network, which reduces the number of network parameters.

[0096] In a specific embodiment of the present invention, for the decoding part, after feature extraction of the input image by the encoding part, the resulting feature map has a size of 16×16×96. Then, it undergoes two 1×1 convolutions to adjust the channel dimension to 384. To enable the network to achieve pixel-to-pixel level prediction, a deconvolution operation is used to upsample the feature map three times, resulting in a feature map of size 128×128×64. Keypoint detection and part detection output heads are set according to the detection task. They share weight parameters in the encoding part and share the upsampling part in the decoding part, forming a multi-task learning network and reducing network parameters.

[0097] As a preferred embodiment, it still combines Figure 4 The key point detection structure includes a first branch structure and a second branch structure, wherein:

[0098] The first branch structure is used to regress the key point heatmap based on the convolutional features;

[0099] The second branch structure is used to perform regression on the coordinate offset of each key point as compensation based on the convolutional features, and output predicted key point information.

[0100] In this embodiment of the invention, the key point detection head outputs two heatmap branches. One branch is used to regress the key point heatmap, and the other branch regresses the coordinate offset of each key point to compensate for the coordinate error caused by image downsampling.

[0101] In a specific embodiment of the present invention, two output branches are designed in the CNN-Transformer decoding part for keypoint detection. One branch is used to regress the heatmap of the keypoints. The heatmap output size is 128×128×C, where C is the keypoint type. The original image size is 512×512, and it is downsampled by a factor of four when the heatmap is obtained after passing through the feature extraction network. To reduce the error in the center point coordinate prediction accuracy caused by downsampling, another branch is set to regress the coordinate offset of each keypoint as compensation. The output size of this branch is 128×128×2, where 2 represents the two offsets of the keypoint in the width and height directions.

[0102] As a preferred embodiment, combined with Figure 6 Let's take a look. Figure 6 This is a schematic diagram of a structural embodiment of the angle information provided by the present invention. The part detection structure includes a first output branch, a second output branch, and a third output branch, wherein:

[0103] The first output branch is used to output the center point of the predicted localization box based on the convolutional features;

[0104] The second output branch is used to output the width and height of the predicted localization box based on the convolutional features;

[0105] The third output branch is used to output the angle information of the predicted localization box based on the convolution features.

[0106] In this embodiment of the invention, the part detection structure outputs three heatmap branches: center point, width and height, and angle, which correspond to the center point information, width and height information, and rotation angle of the part positioning frame, respectively.

[0107] In a specific embodiment of the present invention, the decoding section includes three output branches for the CNN-Transformer part detection head: center point, width / height, and angle information branches. The center point branch outputs a heatmap representing the center point of the part's bounding box, with dimensions of 128×128×D, where D is the number and type of assembled parts. The width / height branch outputs a heatmap representing the width and height information of the part's bounding box, with dimensions of 128×128×2. To ensure the detected part bounding box fits the part better, rotation information is added to the detection box. Figure 4 As shown, the rotation information is defined as the angle between the longer side of the part and the y-axis, with the angle range being [0,π].

[0108] For the loss function of the CNN-Transformer detector, the L1 loss function is used for predicting offset, width, height, and angle information. For the prediction of keypoints and the center points of the part's bounding box, Focal Loss is used to reduce the problem of class imbalance. The calculation formula is as follows:

[0109]

[0110] As a preferred embodiment, combined with Figure 7 Let's take a look. Figure 7 Provided by the present invention Figure 1 A flowchart illustrating an embodiment of step S103, wherein step S103 specifically includes steps S701 to S703, wherein:

[0111] In step S701, the pixel coordinates of the predicted key point information are used as the center, and a pixel region within a preset radius is selected as the feature region.

[0112] In step S702, a two-dimensional Gaussian distribution is used to generate regional values ​​within the feature region, thereby determining the two-dimensional Gaussian probability distribution;

[0113] In step S703, the conversion relationship between the predicted positioning frame and the standard positioning frame in the standard assembly template is determined according to the two-dimensional Gaussian probability distribution.

[0114] In this embodiment of the invention, the pixel coordinates of the key point are used as the center of a circle, and the area with a radius of 10 pixels is used as the feature region. The values ​​within the feature region of the key point are generated using a two-dimensional Gaussian distribution, and the values ​​outside the feature region of the key point are set to 0. Within the feature region of the key point, the Gaussian probability value decreases from the center to the edge.

[0115] It should be noted that the perspective projection method is used to solve the homography transformation matrix between the assembly site image and the standard assembly template.

[0116] In a specific embodiment of the present invention, according to the perspective transformation theory, the minimum number of points required to obtain the homography matrix between two images is 4. Therefore, let C be 4, corresponding to 4 key points. Using the pixel coordinates of the key points as the center, a region with a radius of 10 pixels is selected as the feature region. A two-dimensional Gaussian distribution is used to generate values ​​within the key point feature region, while values ​​outside the key point feature region are set to 0. Within the key point feature region, the Gaussian probability value decreases from the center to the edge. When a key point in the image is not visible, all probability values ​​within the radius of that key point are set to 0. The two-dimensional Gaussian probability distribution of the key point is calculated as follows:

[0117]

[0118] The transformation matrix between the assembly site images and the standard assembly template is calculated using the following formula:

[0119]

[0120] As a preferred embodiment, combined with Figure 8 Let's take a look. Figure 8 Provided by the present invention Figure 1 A flowchart illustrating an embodiment of step S104 is provided. Step S104 specifically includes steps S801 to S802, wherein:

[0121] In step S801, based on the area and number of pixels of the predicted localization box and the standard localization box, the predicted localization box and the standard localization box are matched by IoU to determine the IoU matching value.

[0122] In step S802, if the IoU matching value meets the preset conditions, then there is a quality problem in the current assembly step.

[0123] In this embodiment of the invention, based on the transformation relationship between the assembly site image and the standard assembly template, the part positioning frame in the assembly site image detected by the part detection head is matched with the pre-marked part positioning frame in the standard assembly template based on the intersection-union ratio to determine whether there are any missing or incorrect assembly problems in the current assembly step.

[0124] In a specific embodiment of the present invention, combined with Figure 9 Let's take a look. Figure 9 This is a schematic diagram of an embodiment of IoU matching provided by the present invention. After obtaining the positioning box of the part to be tested, the positioning box of the part to be tested is matched with the positioning box in the template image based on IoU. The matching formula is as follows:

[0125]

[0126] Among them, such as Figure 5 As shown, Sr S is the area of ​​the standard bounding box of a part in a standard assembly template. v This is the area of ​​the actual inspection frame of the part after mapping. Calculate S. r The number of pixels in SumB, S v The number of pixels in the array is SumC, then S is calculated. r ∪S v If the number of pixels in the middle is SumA, then S r ∩S v The number of pixels can be calculated using the following formula:

[0127] S r ∩S v =SumB + SumC - SumA

[0128] When performing quality inspections for missing or incorrect components, it is necessary to set a quality scoring threshold. When the calculated IoU... 2D If the value is greater than the threshold, then there are no quality issues such as missing or incorrect assembly in the current assembly step. When the calculated IoU... 2D If the value is less than the threshold, it indicates that there is a quality problem in the current assembly step.

[0129] This invention also provides an assembly quality inspection device based on deep learning and object matching, combined with... Figure 10 Let's take a look. Figure 10 This is a schematic diagram of an embodiment of the assembly quality inspection device based on deep learning and object matching provided by the present invention. The assembly quality inspection device 1000 based on deep learning and object matching includes:

[0130] Acquisition unit 1001 is used to acquire images of the assembly site to be tested;

[0131] The processing unit 1002 is used to input the assembly site image to be tested into a fully trained lightweight neural network to determine the predicted key point information and the predicted positioning box; it is also used to determine the conversion relationship between the predicted positioning box and the standard positioning box in the standard assembly template based on the predicted key point information.

[0132] The matching unit 1003 is used to match the predicted positioning box and the standard positioning box and determine the matching quality.

[0133] For more specific implementation details of the various units of the assembly quality inspection device based on deep learning and object matching, please refer to the description of the assembly quality inspection method based on deep learning and object matching described above. It has similar beneficial effects and will not be repeated here.

[0134] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the assembly quality inspection method based on deep learning and object matching as described above.

[0135] Generally, computer instructions for implementing the methods of the present invention can be carried on any combination of one or more computer-readable storage media. Non-transitory computer-readable storage media can include any computer-readable medium except for signals themselves that are temporarily propagating.

[0136] Computer-readable storage media can be, for example—but not limited to—electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0137] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—as well as conventional procedural programming languages—such as the "C" language or similar programming languages. In particular, Python, suitable for neural network computation, and platform frameworks based on TensorFlow, PyTorch, etc., can be used. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0138] This invention also provides an electronic device, combined with Figure 11 Let's take a look. Figure 11This is a schematic diagram of an embodiment of the electronic device provided by the present invention. The electronic device 1100 includes a processor 1101, a memory 1102, and a computer program stored in the memory 1102 and executable on the processor 1101. When the processor 1101 executes the program, it implements the assembly quality inspection method based on deep learning and object matching as described above.

[0139] In a preferred embodiment, the electronic device 1100 further includes a display 1103 for displaying the assembly quality inspection method based on deep learning and object matching as described above, which is executed by the processor 1101.

[0140] For example, a computer program can be divided into one or more modules / units, one or more of which are stored in memory 1102 and executed by processor 1101 to complete the present invention. One or more modules / units can be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in electronic device 1100. For example, the computer program can be divided into various units as described in the above embodiments, and the specific functions of each unit are as described above, and will not be repeated here.

[0141] Electronic device 1100 can be a desktop computer, laptop, PDA, or smartphone with an adjustable camera module.

[0142] The processor 1101 may be an integrated circuit chip with signal processing capabilities. The processor 1101 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor or any conventional processor.

[0143] The memory 1102 may be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), etc. The memory 1102 stores programs, and the processor 1101 executes these programs upon receiving execution instructions. The process definition method disclosed in any of the foregoing embodiments of the present invention can be applied to the processor 1101, or implemented by the processor 1101.

[0144] The display 1103 can be an LCD screen or an LED screen. For example, the display screen on a mobile phone.

[0145] Understandable, Figure 11 The structure shown is only a schematic diagram of one possible structure of electronic device 1100. Electronic device 1100 may also include more than one of the following: Figure 11 Show more or fewer components. Figure 11 The components shown can be implemented using hardware, software, or a combination thereof.

[0146] The computer-readable storage medium and electronic device provided in the above embodiments of the present invention can be implemented with reference to the content specifically described in the present invention for implementing the assembly quality inspection method based on deep learning and object matching, and have similar beneficial effects as implementing the assembly quality inspection method based on deep learning and object matching as described above, which will not be repeated here.

[0147] This invention discloses an assembly quality inspection method based on deep learning and object matching. First, the assembly site image to be tested is used as input, and key points and positioning box information in the assembly site image are predicted simultaneously. Second, the conversion relationship between the on-site assembly image and the standard template is solved using the key point prediction results. Finally, based on the prediction results of the assembly part positioning boxes, the detected part positioning boxes are matched with the positioning boxes in the standard assembly template based on the intersection-union ratio (IoU) to evaluate whether there are quality problems in the current step.

[0148] This invention provides an assembly quality inspection method that offers high real-time performance, accuracy, and low hardware resource requirements. It employs deep learning and object matching algorithms to inspect assembled parts in on-site images after assembly steps are completed. The 2D bounding box of the currently assembled object in the on-site image is matched with 2D bounding boxes in a standard state template extracted from CAD using intersection-union (IU) ratio to assess whether assembly quality issues exist in the current assembly step. This invention acquires assembly site images on-site, inputs them into a trained lightweight network, predicts key points and part location boxes in the images, and matches them with a standard assembly template to identify missing and incorrect assembly issues. This allows for real-time inspection of missing and incorrect assembly, ensuring consistent product quality and improving assembly efficiency.

[0149] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A method of assembly quality inspection based on deep learning and object matching, characterized by, The method comprises the following steps: acquiring an image of an assembly site to be measured; inputting the image of the assembly site to be measured into a trained lightweight neural network to determine predicted key point information and a predicted positioning frame, wherein the lightweight neural network comprises an encoder and a decoder, the encoder is configured to extract local features and global features of the image of the assembly site, and the decoder is configured to determine the predicted key point information and the predicted positioning frame according to the extracted features; the decoder comprises a convolution sampling layer, a key point detection structure and a part detection structure, wherein the convolution sampling layer is configured to perform multiple convolution and sampling on the extracted features output by the encoder to output convolution features; the key point detection structure is configured to perform key point detection according to the convolution features to output the predicted key point information; and the part detection structure is configured to perform part detection according to the convolution features to output the predicted positioning frame; the key point detection structure comprises a first branch structure and a second branch structure, wherein the first branch structure is configured to regress a key point heat map according to the convolution features, and the second branch structure is configured to regress a coordinate offset of each key point as compensation according to the convolution features to output the predicted key point information; and the part detection structure comprises a first output branch, a second output branch and a third output branch, wherein the first output branch is configured to output a center point of the predicted positioning frame according to the convolution features, the second output branch is configured to output a width and a height of the predicted positioning frame according to the convolution features, and the third output branch is configured to output angle information of the predicted positioning frame according to the convolution features; determining a conversion relationship between the predicted positioning frame and a standard positioning frame in a standard assembly template according to the predicted key point information; matching the predicted positioning frame and the standard positioning frame according to the conversion relationship to determine a matching quality, including: performing IoU matching on the predicted positioning frame and the standard positioning frame according to areas and pixel numbers of the predicted positioning frame and the standard positioning frame to determine an IoU matching value; and if the IoU matching value meets a preset condition, there is a quality problem in a current assembly step. 2.The deep learning and object matching based assembly quality inspection method according to claim 1, wherein, The determination of the standard positioning frame comprises: collecting a standard assembly template under each assembly step by using a virtual camera and labeling corresponding standard key point information and a standard positioning frame. 3.The deep learning and object matching based assembly quality inspection method according to claim 1, wherein, The training process of the lightweight neural network comprises: acquiring an assembly image sample set containing labeling information, wherein the labeling information comprises actual key point information and an actual positioning frame corresponding to each image of the assembly image sample set; inputting the assembly image sample set into the constructed lightweight neural network to determine predicted key point information and a predicted positioning frame, and determining a loss function according to the actual key point information, the actual positioning frame, the predicted key point information and the predicted positioning frame; adjusting parameters of the lightweight neural network according to a value of the loss function until the parameters meet a convergence condition to determine a trained lightweight neural network. 4.The deep learning and object matching based assembly quality inspection method according to claim 1, wherein, The encoder comprises a plurality of first convolution structures and second convolution structures connected in sequence, wherein: The first convolutional structure is configured to extract local features of the assembly site image using a depth separable convolution. The second convolutional structure is configured to extract global features of the assembly site image using a Transformer structure.

5. The assembly quality inspection method based on deep learning and object matching according to claim 1, wherein determining the conversion relationship between the predicted positioning frame and the standard positioning frame in the standard assembly template according to the predicted key point information comprises: selecting a pixel region within a preset radius as a feature region with the pixel coordinates of the predicted key point information as the center; generating region values in the feature region using a two-dimensional Gaussian distribution to determine a two-dimensional Gaussian probability distribution; determining the conversion relationship between the predicted positioning frame and the standard positioning frame in the standard assembly template according to the two-dimensional Gaussian probability distribution.

Images