Assembly change detection method based on 3D attention and bilateral filtering

Abstract

The present invention relates to an assembly change detection method based on three-dimensional attention and bilateral filtering, comprising the following steps: S1, acquiring a first image and a second image, wherein the first image is an image of a mechanical assembly at a previous moment, and the second image is an image of the mechanical assembly at a second moment; S2, constructing and training a change detection model, wherein the change detection model comprises an encoding module, a decoding module, a first three-dimensional attention module, and a second three-dimensional attention module, wherein the encoding module comprises a first sub-encoding network and a second sub-encoding network; S3, inputting the first image and the second image into the change detection model to obtain a change image.

Description

Technical Field

[0001] The invention relates to an assembly change detection method based on three-dimensional attention and bilateral filtering, and belongs to the field of multi-view change detection of mechanical assembly images. Background Art

[0002] Assembly is the production process of assembling mechanical parts together according to the design sequence to form a product. The assembly sequence is the key to the quality of product assembly. In the assembly process of complex products, detecting the newly installed parts in each assembly step helps to obtain relevant information about the mechanical assembly process, timely discover assembly sequence errors, and improve the production efficiency of mechanical products. Therefore, intelligent detection of the assembly process of mechanical assemblies has important research value. However, there is currently a lack of effective methods for accurately monitoring the assembly sequence of products (that is, monitoring the changes in mechanical assemblies).

[0003] Image change detection can detect information that changes over time in an image by analyzing two images at different times. Image change detection is currently mainly used in remote sensing image analysis, and has important practical value in land use change detection, disaster assessment, and environmental monitoring. Traditional change detection methods can be divided into two categories: pixel-based change detection methods and object-based change detection methods. The pixel-based change detection method generates a difference image based on the pixel information of the image, and obtains the change image through threshold segmentation or clustering. This method is simple to implement, but ignores spatial information, and a lot of noise will appear in the image processing process. The object-based change detection method uses the texture, structure, and geometric information of the image to obtain the change image. Although this method makes full use of the spatial information of the image, the feature extraction method is relatively complex and has poor robustness.

[0004] In recent years, deep learning methods have been widely used in change detection and have achieved good results. At present, the input of change detection models is mostly color images. This is because the color and texture information of deep images are single, and it is difficult to extract image features. The labels of real data sets used in multi-view change detection of mechanical assemblies need to be manually labeled, which has the problems of time-consuming labeling and inaccurate labeling.

[0005] The patent with publication number CN113269236A discloses an assembly change detection method based on multi-model integration, which includes the following steps: establishing a three-dimensional model of the assembly, adding labels to each part in the three-dimensional model, setting a number of assembly nodes, obtaining depth images of the three-dimensional model under each assembly node at different perspectives, and obtaining the change category labels of the newly added parts of each assembly node; selecting two depth images at different perspectives at two moments before and after as training samples; subjecting the training samples to feature extraction, selection of target candidate areas, feature information comparison, classification prediction, and semantic segmentation in turn, training the detection model, saving the model parameters with the best similarity during the training process, and completing the training; obtaining the depth images of the two assembly nodes before and after the assembly process of the assembly to be detected, inputting them into the trained detection model, and outputting the category and image of the changed parts during the assembly process. Summary of the invention

[0006] In order to overcome the problems existing in the prior art, the present invention designs an assembly change detection method based on three-dimensional attention and bilateral filtering.

[0007] In order to achieve the above object, the present invention adopts the following technical solution:

[0008] Technical solution 1:

[0009] The assembly change detection method based on three-dimensional attention and bilateral filtering includes the following steps:

[0010] S1. Acquire a first image and a second image, wherein the first image is an image of the mechanical assembly at a previous moment, and the second image is an image of the mechanical assembly at a second moment;

[0011] S2. Construct and train a change detection model, wherein the change detection model includes an encoding module, a decoding module, a first three-dimensional attention module, and a second three-dimensional attention module, and the encoding module includes a first sub-encoding network and a second sub-encoding network;

[0012] S3, inputting the first image and the second image into a change detection model to obtain a change image, specifically:

[0013] S31, input the first image into the first sub-coding network to obtain the first shallow feature and the first deep feature; input the second image into the second sub-coding network to obtain the second shallow feature and the second deep feature; fuse the first deep feature and the second deep feature to obtain the first fused feature, and output it to the decoding module; output the first shallow feature and the second shallow feature to the first three-dimensional attention module and the second three-dimensional attention module respectively;

[0014] S32, the first three-dimensional attention module outputs the first three-dimensional attention feature to the encoding network; the second three-dimensional attention module outputs the second three-dimensional attention feature to the encoding network;

[0015] S33. The decoding module outputs a changed image based on the input first fusion feature, the first three-dimensional attention feature and the second three-dimensional attention feature.

[0016] Furthermore, it also includes a bidirectional filtering module; the bidirectional filtering module is used to perform bidirectional filtering on the changed image.

[0017] Furthermore, the specific steps of obtaining the first three-dimensional attention feature are:

[0018] The three-dimensional attention weight is calculated, and the importance of each neuron of the first shallow feature map is determined according to the three-dimensional attention weight, and the features of neurons whose importance is higher than the threshold are enhanced to obtain the first three-dimensional attention feature.

[0019] Furthermore, the step S33 is specifically as follows:

[0020] The decoding module performs several transposed convolution operations on the first fused features of the input to obtain the decoded features;

[0021] Perform feature fusion on the decoding feature, the first three-dimensional attention feature, and the second three-dimensional attention feature to obtain a second fused feature;

[0022] Perform several transposed convolution operations on the second fused feature to obtain a changed image.

[0023] Furthermore, the training change detection model is specifically:

[0024] Obtain a synthetic depth image dataset and a real color image dataset; use the synthetic depth image dataset to train a change detection model, and save the obtained first optimal model; use the real color image dataset to train the first optimal model through a transfer learning method, and save the obtained second optimal model as the trained change detection model.

[0025] Technical solution 2:

[0026] An assembly change detection device based on three-dimensional attention and bilateral filtering includes a memory and a processor, wherein the memory stores instructions, and the instructions are suitable for being loaded by the processor and executing the following steps:

[0027] S1. Acquire a first image and a second image, wherein the first image is an image of the mechanical assembly at a previous moment, and the second image is an image of the mechanical assembly at a second moment;

[0028] S2. Construct and train a change detection model, wherein the change detection model includes an encoding module, a decoding module, a first three-dimensional attention module, and a second three-dimensional attention module, and the encoding module includes a first sub-encoding network and a second sub-encoding network;

[0029] S3, inputting the first image and the second image into a change detection model to obtain a change image, specifically:

[0030] S31, input the first image into the first sub-coding network to obtain the first shallow feature and the first deep feature; input the second image into the second sub-coding network to obtain the second shallow feature and the second deep feature; fuse the first deep feature and the second deep feature to obtain the first fused feature, and output it to the decoding module; output the first shallow feature and the second shallow feature to the first three-dimensional attention module and the second three-dimensional attention module respectively;

[0031] S32, the first three-dimensional attention module outputs the first three-dimensional attention feature to the encoding network; the second three-dimensional attention module outputs the second three-dimensional attention feature to the encoding network;

[0032] S33. The decoding module outputs a changed image based on the input first fusion feature, the first three-dimensional attention feature and the second three-dimensional attention feature.

[0033] Furthermore, it also includes a bidirectional filtering module; the bidirectional filtering module is used to perform bidirectional filtering on the changed image.

[0034] Furthermore, the specific steps of obtaining the first three-dimensional attention feature are:

[0035] The three-dimensional attention weight is calculated, and the importance of each neuron of the first shallow feature map is determined according to the three-dimensional attention weight, and the features of neurons whose importance is higher than the threshold are enhanced to obtain the first three-dimensional attention feature.

[0036] Furthermore, the step S33 is specifically as follows:

[0037] The decoding module performs several transposed convolution operations on the first fused features of the input to obtain the decoded features;

[0038] Perform feature fusion on the decoding feature, the first three-dimensional attention feature, and the second three-dimensional attention feature to obtain a second fused feature;

[0039] Perform several transposed convolution operations on the second fused feature to obtain a changed image.

[0040] Furthermore, the training change detection model is specifically:

[0041] Obtain a synthetic depth image dataset and a real color image dataset; use the synthetic depth image dataset to train a change detection model, and save the obtained first optimal model; use the real color image dataset to train the first optimal model through a transfer learning method, and save the obtained second optimal model as the trained change detection model.

[0042] Compared with the prior art, the present invention has the following characteristics and beneficial effects:

[0043] In the present invention, the first three-dimensional attention module and the second three-dimensional attention module take into account the spatial dimension information and the channel dimension information. The output first three-dimensional attention features and the second three-dimensional attention features can focus on the important areas and important features in the first fusion feature information and ignore the unimportant information therein, thereby effectively improving the change detection efficiency and reducing the complexity of the change detection model. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 Schematic diagram of an assembly change detection method based on three-dimensional attention and bilateral filtering provided by the present invention;

[0045] Figure 2 is a schematic diagram of the encoding module provided by the present invention;

[0046] Figure 3 is a schematic diagram of a decoding module provided by the present invention;

[0047] Figure 4 is the three-dimensional attention model provided by the present invention;

[0048] Figure 5 It is the change detection model training flow chart. DETAILED DESCRIPTION

[0049] The present invention is described in more detail below with reference to the embodiments.

[0050] Embodiment 1

[0051] The assembly change detection method based on three-dimensional attention and bilateral filtering has the following steps:

[0052] S1. Acquire a first image and a second image, wherein the first image is an image of the mechanical assembly at a previous moment, and the second image is an image of the mechanical assembly at a second moment;

[0053] S2. Construct and train a change detection model, wherein the change detection model includes an encoding module, a decoding module, a first three-dimensional attention module, and a second three-dimensional attention module, and the encoding module includes a first sub-encoding network and a second sub-encoding network;

[0054] S3, inputting the first image and the second image into a change detection model to obtain a change image, specifically:

[0055] S31, input the first image into the first sub-coding network to obtain the first shallow feature and the first deep feature; input the second image into the second sub-coding network to obtain the second shallow feature and the second deep feature; fuse the first deep feature and the second deep feature to obtain the first fused feature, and output it to the decoding module; output the first shallow feature and the second shallow feature to the first three-dimensional attention module and the second three-dimensional attention module respectively;

[0056] S32, the first three-dimensional attention module outputs the first three-dimensional attention feature to the encoding network; the second three-dimensional attention module outputs the second three-dimensional attention feature to the encoding network;

[0057] S33. The decoding module outputs a changed image based on the input first fusion feature, the first three-dimensional attention feature and the second three-dimensional attention feature.

[0058] The beneficial effect of this embodiment is that the first three-dimensional attention module and the second three-dimensional attention module take into account the spatial dimension information and the channel dimension information, and the output first three-dimensional attention features and the second three-dimensional attention features can focus on important areas and important features in the first fusion feature information and ignore unimportant information therein, thereby effectively improving the change detection efficiency and reducing the complexity of the change detection model.

[0059] Embodiment 2

[0060] In this embodiment, the change detection model further includes a bilateral filtering module; the change image described in the first embodiment is input into the bilateral filtering module to obtain a filtered change image.

[0061] Define the change image as I, and define the change image after filtering as Apply the following bilateral filter function to image I for filtering:

[0062]

[0063] Among them, f is the pixel value of the template point (that is, the value of the pixel point in image I), and h is the final pixel value of the target point (that is, the value of the pixel point in image I). The value of the pixel in the image), ω is the weight coefficient, i and j are the pixel indexes, and (k, l) is the center point of the template. When all the pixels in image I are filtered, the output is the changed image

[0064] Since the components of the assembly are closely connected, the change area detected by the change detection model is prone to unclear boundaries; and the real image of the assembly is easily disturbed by the surrounding environment during the acquisition process, resulting in noise in the detected change image, and the detected change image is inaccurate. This embodiment reduces the noise in the change image through a bilateral filtering module. The advantage is that it not only considers the Euclidean distance between pixels, but also considers the similarity in pixel brightness and depth, so that the filtered image can reduce the noise in the image and keep the edge information of the image clear.

[0065] Embodiment 3

[0066] like Figure 2 As shown in Figure 1, the first sub-coding network includes three convolutional layers (A0, A1, A2), a pooling layer (B), and four residual blocks (C1, C2, C3, C4). Each residual block contains two residual units, and each residual unit contains two convolutional layers. Skip connections are used inside the residual unit to alleviate the gradient vanishing problem caused by the depth of the sub-coding network.

[0067] Similarly, the second sub-encoding network also includes three convolutional layers, one pooling layer, and four residual blocks. Each residual block contains two residual units, and each residual unit contains two convolutional layers. Skip connections are used inside the residual unit to alleviate the gradient vanishing problem caused by the depth of the sub-encoding network.

[0068] like Figure 3 As shown, the decoding module includes three convolutional layers (A0, A1, A2), two upsampling layers and four decoding blocks (Res4, Res3, Res2, Res1), each decoding block contains two decoding units, and each decoding unit contains two convolutional layers.

[0069] The first sub-coding network and the fourth residual block of the second sub-coding network output the first shallow feature and the second shallow feature to the first three-dimensional attention module and the second three-dimensional attention module respectively, and obtain the first three-dimensional attention feature and the second three-dimensional attention feature. The first sub-coding network and the third convolutional layer of the second sub-coding network output the first deep feature and the second deep feature respectively, and fuse the first deep feature and the second deep feature to obtain the first fused feature and output it to the decoding module.

[0070] After the decoding module performs transposed convolution and upsampling operations on the first fusion feature, the decoded feature is obtained, which is fused with the first three-dimensional attention feature and the second three-dimensional attention feature, and then input into the four decoding blocks for transposed convolution to obtain the changed image.

[0071] Embodiment 4

[0072] The features obtained by the fourth residual block in the sub-coding network are output to the three-dimensional attention module. Based on the theory of neuroscience, the three-dimensional attention weights are calculated using the following energy function to determine the importance of neurons. The formula of the energy function is as follows:

[0073]

[0074] Among them, t and x i They represent the input feature X mapping neurons and other neurons respectively, i represents the index of the spatial dimension, M = H × W is the number of neurons on the channel, and w t and b t Refers to the weights and biases of neurons when they are transformed.

[0075] The importance of neurons is determined by the energy of each neuron. The lower the energy of neuron t, the greater the difference between it and the surrounding neurons, and the higher the importance of neuron t. For neurons with high importance, features need to be enhanced to highlight the key areas in the image. The output of the first three-dimensional attention feature three-dimensional attention is calculated as follows: The calculation formula is:

[0076]

[0077] Among them, E represents the grouping of low-energy neurons across channels and spatial dimensions; ⊙ represents the scaling operator; sigmoid is used to limit excessive values ​​in E without affecting the importance of neurons.

[0078] There are two main types of existing attention: channel attention and spatial attention. The feature map extracted by the encoding module of the present invention not only contains a large amount of useful information on the channel, but also contains a large amount of useful information between the feature map pixels inside the channel. Using channel attention or spatial attention alone is prone to losing information. The improvement of this embodiment is that the ability to extract detail features (including channel information and feature map pixel information) is improved through the three-dimensional attention module, thereby improving the efficiency of change detection.

[0079] Embodiment 5

[0080] Based on the first to fourth embodiments, multi-view change detection is performed on a mechanical assembly, including three stages: data set establishment stage, training stage, and testing stage. The specific process is as follows:

[0081] Dataset creation phase:

[0082] (1) Creation of synthetic dataset

[0083] The synthetic image dataset includes a synthetic depth image set and a synthetic color image set (the synthetic dataset is simple and efficient, which is conducive to the rapid detection of the performance of the change detection model): First, the three-dimensional model of the mechanical assembly is established using SolidWorks software, and n assembly nodes and n-1 assembly steps are set, and each step assembles a component; then the three-dimensional model of the mechanical assembly is loaded into the depth image synthesis software, and the virtual camera is set to image each assembly node at different angles to obtain a synthetic depth image of each assembly node, and a specific color is set for the changed component as the label value of the synthetic depth image. The synthetic depth image is input into the semantic segmentation network (such as FCN, U-Net, etc.) to segment the assembly components and obtain a semantic image; the synthetic depth image and the semantic image are fused to obtain a synthetic color image.

[0084] (2) Establishment of a real color image dataset

[0085] The assembly steps of the real color image dataset are the same as those of the synthetic dataset. The color image dataset first uses an RGB-D camera to collect physical color images, fixes the camera position, rotates the physical mechanical assembly to obtain multi-view images of the assembly, and finally uses image processing software to mark the change label images of the newly installed parts.

[0086] The change detection data set used in the present invention includes images of each assembly node at different viewing angles and change label images of each assembly step.

[0087] Training phase:

[0088] refer to Figure 5 The specific training process of the multi-view change detection method for mechanical assembly of the present invention is as follows:

[0089] S1: Use synthetic depth image dataset and synthetic color image dataset to train the change detection model respectively.

[0090] S2: When the set training batch is reached, the optimal model of the synthetic depth image dataset and the optimal model of the synthetic color image dataset are saved.

[0091] S3: Based on the principle of transfer learning, the optimal model trained on the synthetic depth image dataset is used as the pre-trained model of the real color image dataset for iterative training. When the set training batch is reached, the optimal model of the real color image dataset is saved.

[0092] Testing phase:

[0093] Input two images from different perspectives into the above optimal model to generate a changed image. If the accuracy of the changed image meets the requirement, the training is completed; otherwise, the training is repeated.

[0094] The improvement of this embodiment is that the change detection model can process both synthetic depth images that are more convenient to obtain and real color images with more significant features and better detection effects. At the same time, based on the principle of transfer learning, the change detection model is pre-trained with synthetic depth images and then trained with real color images, so that the change detection model has better feature extraction capabilities and can extract enough features from fewer training samples to complete change detection. At the same time, the training speed is accelerated, the accuracy of the network model is improved, and the workload of manual labeling is reduced.

[0095] Embodiment five.

[0096] In order to verify the effectiveness of the assembly change detection method based on three-dimensional attention and bilateral filtering proposed in this invention, the existing change detection models STANet and SNUNet-CD are compared with the method of this invention. The data sets are the three data sets established in the above steps, and the evaluation index is F1. The results are shown in the following table.

[0097]

[0098] F1 is a comprehensive evaluation index calculated based on precision and recall. From the above table, it can be seen that the evaluation indexes of the method proposed in the present invention in the three data sets are better than those of the prior art.

[0099] Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

Claims

1. An assembly change detection method based on three-dimensional attention and bilateral filtering, characterized in that: The steps include: S1. Acquire a first image and a second image, wherein the first image is an image of the mechanical assembly at a previous moment, and the second image is an image of the mechanical assembly at a second moment; S2. Construct and train a change detection model, wherein the change detection model includes an encoding module, a decoding module, a first three-dimensional attention module, and a second three-dimensional attention module, and the encoding module includes a first sub-encoding network and a second sub-encoding network; S3, inputting the first image and the second image into a change detection model to obtain a change image, specifically: S31, input the first image into the first sub-coding network to obtain the first shallow feature and the first deep feature; input the second image into the second sub-coding network to obtain the second shallow feature and the second deep feature; fuse the first deep feature and the second deep feature to obtain the first fused feature, and output it to the decoding module; output the first shallow feature and the second shallow feature to the first three-dimensional attention module and the second three-dimensional attention module respectively; S32, the first three-dimensional attention module outputs the first three-dimensional attention feature to the encoding network; the second three-dimensional attention module outputs the second three-dimensional attention feature to the encoding network; S33, the decoding module outputs a change image according to the input first fusion feature, the first three-dimensional attention feature and the second three-dimensional attention feature; It also includes a bidirectional filtering module; the bidirectional filtering module is used to perform bidirectional filtering on the changed image; The specific steps of obtaining the first three-dimensional attention feature are: Calculating a three-dimensional attention weight, judging the importance of each neuron of the first shallow feature map according to the three-dimensional attention weight, enhancing the features of neurons whose importance is higher than a threshold, and obtaining a first three-dimensional attention feature; The training change detection model is specifically: Obtain a synthetic depth image dataset and a real color image dataset; use the synthetic depth image dataset to train a change detection model, and save the obtained first optimal model; use the real color image dataset to train the first optimal model through a transfer learning method, and save the obtained second optimal model as the trained change detection model.

2. The assembly change detection method based on three-dimensional attention and bilateral filtering according to claim 1, characterized in that: The step S33 is specifically as follows: The decoding module performs several transposed convolution operations on the first fused features of the input to obtain the decoded features; Perform feature fusion on the decoding feature, the first three-dimensional attention feature, and the second three-dimensional attention feature to obtain a second fused feature; Perform several transposed convolution operations on the second fused feature to obtain a changed image.

3. An assembly change detection device based on three-dimensional attention and bilateral filtering, comprising a memory and a processor, wherein the memory stores instructions, characterized in that: The instructions are suitable for being loaded by a processor and executing the following steps: S1. Acquire a first image and a second image, wherein the first image is an image of the mechanical assembly at a previous moment, and the second image is an image of the mechanical assembly at a second moment; S2. Construct and train a change detection model, wherein the change detection model includes an encoding module, a decoding module, a first three-dimensional attention module, and a second three-dimensional attention module, and the encoding module includes a first sub-encoding network and a second sub-encoding network; S3, inputting the first image and the second image into a change detection model to obtain a change image, specifically: S31, input the first image into the first sub-coding network to obtain the first shallow feature and the first deep feature; input the second image into the second sub-coding network to obtain the second shallow feature and the second deep feature; fuse the first deep feature and the second deep feature to obtain the first fused feature, and output it to the decoding module; output the first shallow feature and the second shallow feature to the first three-dimensional attention module and the second three-dimensional attention module respectively; S32, the first three-dimensional attention module outputs the first three-dimensional attention feature to the encoding network; the second three-dimensional attention module outputs the second three-dimensional attention feature to the encoding network; S33, the decoding module outputs a change image according to the input first fusion feature, the first three-dimensional attention feature and the second three-dimensional attention feature; It also includes a bidirectional filtering module; the bidirectional filtering module is used to perform bidirectional filtering on the changed image; The specific steps of obtaining the first three-dimensional attention feature are: Calculating a three-dimensional attention weight, judging the importance of each neuron of the first shallow feature map according to the three-dimensional attention weight, enhancing the features of neurons whose importance is higher than a threshold, and obtaining a first three-dimensional attention feature; The training change detection model is specifically: Obtain a synthetic depth image dataset and a real color image dataset; use the synthetic depth image dataset to train a change detection model, and save the obtained first optimal model; use the real color image dataset to train the first optimal model through a transfer learning method, and save the obtained second optimal model as the trained change detection model.

4. The assembly change detection device based on three-dimensional attention and bilateral filtering according to claim 3 is characterized in that: The step S33 is specifically as follows: The decoding module performs several transposed convolution operations on the first fused features of the input to obtain the decoded features; Perform feature fusion on the decoding feature, the first three-dimensional attention feature, and the second three-dimensional attention feature to obtain a second fused feature; Perform several transposed convolution operations on the second fused feature to obtain a changed image.

Images