A method, an electronic device and a medium for instance segmentation of mechanical assembly images

Abstract

The present invention relates to a method for instance segmentation of mechanical assembly images, an electronic device, and a medium, comprising the following steps: constructing an instance segmentation network including a backbone feature extraction network, a feature multi-scale fusion network, and a mask segmentation network, wherein the mask segmentation network includes a Transfiner module and a DCT module; inputting a mechanical assembly image dataset to train the constructed instance segmentation network to obtain a trained mechanical assembly instance segmentation model; inputting the mechanical assembly image to be detected into the trained mechanical assembly instance segmentation model to output an instance segmentation result; this method can more accurately segment mechanical assembly images and accurately identify the categories and positions of each mechanical part.

Description

Technical Field

[0001] The present invention relates to a method for instance segmentation of mechanical assembly images, an electronic device, and a medium, belonging to the technical fields of computer image recognition and intelligent manufacturing. Background Art

[0002] In recent years, personalized customization and flexible manufacturing have gradually become important development trends in industrial production. Different types of mechanical products can be produced on the same assembly line. This production mode has led to an increase in product complexity, making it easier for human errors to occur during the assembly monitoring of mechanical products. Once errors (such as incorrect assembly sequence, missing parts, misassembly, etc.) that occur during the assembly process are not detected in a timely manner, it will directly affect the assembly efficiency and service life of mechanical products. Using instance segmentation technology in the field of assembly monitoring can efficiently and accurately segment and identify each part in the mechanical assembly image, effectively monitoring errors or non-standard operations during the assembly process, thereby reducing the unqualified rate of products and shortening the production cycle of products, which has important research significance for realizing the intelligent detection of mechanical assemblies.

[0003] Although there are currently a large number of instance segmentation network models, in actual scenarios, mechanical assemblies have the characteristics of diverse part shapes and overlapping installation positions, resulting in difficulties for the model to accurately segment each part. Therefore, the current image instance segmentation technology has less research and application in the field of mechanical assembly monitoring.

[0004] The prior art, such as the invention patent with the application number "202211689162.1", discloses an automatic marking method for tooth cracks based on Mask R-CNN instance segmentation. By using a training set to train the Mask R-CNN network structure, an optimal weight model is obtained; the Mask R-CNN network structure has a not-high-enough average mask accuracy, resulting in insufficient accuracy of the segmented image and prone to incorrect marking. Therefore, a new technical solution is needed to improve the accuracy of image segmentation. Summary of the Invention

[0005] In order to solve the problems existing in the above prior art, the present invention proposes a method for instance segmentation of mechanical assembly images, an electronic device, and a medium.

[0006] The technical solution of the present invention is as follows:

[0007] On the one hand, the present invention provides a method for instance segmentation of mechanical assembly images, including the following steps:

[0008] Construct an instance segmentation network including a backbone feature extraction network, a feature multi-scale fusion network, and a mask segmentation network; the backbone feature extraction network is used to initially extract the initial feature map of the input image, the feature multi-scale fusion network performs feature fusion processing based on the initial feature map to generate a multi-level feature map; the mask segmentation network outputs the corresponding mask segmentation image based on the multi-level feature map;

[0009] Among them, the mask segmentation network includes a Transfiner module and a DCT module. The Transfiner module detects loss information in the target edge region of the multi-level feature map and outputs a part edge mask with high precision as the first mask; the DCT module performs high-resolution regression on the low-resolution mask output by the Transfiner module to obtain a second mask; the first mask and the second mask are fused to obtain the final mask segmentation image;

[0010] Input the mechanical assembly image dataset to train the constructed instance segmentation network to obtain a trained mechanical assembly instance segmentation model;

[0011] Input the mechanical assembly image to be detected into the trained mechanical assembly instance segmentation model and output the instance segmentation result.

[0012] As a preferred embodiment of the present invention, the training steps of the instance segmentation network are as follows:

[0013] Respectively establish a synthetic depth image dataset and a real color image dataset of the mechanical assembly;

[0014] Divide the synthetic depth image dataset and the real color image dataset into a training set and a test set to obtain a synthetic depth image training set, a real color image training set, a synthetic depth image test set, and a real color image test set;

[0015] Train the instance segmentation network through the synthetic depth image training set and the real color image training set, train the network to the set training batch, and save the optimal model during the training process.

[0016] Test and analyze the optimal model through the synthetic depth image test set and the real color image test set to obtain the qualitative and quantitative results of the instance segmentation.

[0017] As a preferred embodiment of the present invention, the synthetic depth image dataset is obtained by using modeling software to establish a three-dimensional model of the mechanical assembly, using annotation software to annotate each part instance with different colors, then using dataset synthesis software to generate depth images and label images, and finally converting them into a coco format dataset;

[0018] The real color image dataset is obtained by using an RGB-D camera to capture images of a mechanical assembly, processing the captured images to obtain color images of appropriate size, then using annotation software to annotate each part in the color images to obtain labeled images, and finally generating a dataset in the COCO format.

[0019] As a preferred embodiment of the present invention, the backbone feature extraction network is used to extract the information in the original image into a feature map containing rich semantic information, wherein:

[0020] The backbone feature extraction network uses the ResNet101 network as the backbone network. The ResNet101 network includes five stages of Res1, Res2, Res3, Res4, and Res5 in sequence. Among them, in the Res1 and Res2 stages, conventional convolution operations are performed on the input features, and in the Res3, Res4, and Res5 stages, deformable convolution operations are performed on the input features;

[0021] The method of the deformable convolution operation is: adding an offset variable that adaptively adjusts the sampling position according to the target geometric feature at the sampling point position in the conventional convolution operation.

[0022] As a preferred embodiment of the present invention, the feature multi-scale fusion network performs feature fusion processing based on the preliminary feature map. The specific steps for generating a multi-level feature map include:

[0023] Performing convolution on the feature map C5 output by the Res5 stage to generate a feature map P5, upsampling the feature map P5 and performing feature fusion with the feature map C4 output by the Res4 stage to generate a feature map P4, upsampling the feature map P4 and performing feature fusion with the feature map C3 output by the Res3 stage to generate a feature map P3;

[0024] Obtaining a multi-level feature map including the feature map P5, the feature map P4, and the feature map P3.

[0025] As a preferred embodiment of the present invention, the specific steps for the Transfiner module to detect the loss information in the target edge region of the multi-level feature map and output a part edge mask with high precision as the first mask are:

[0026] Predicting a low-resolution mask of the multi-level feature map through the rough mask head network in the module, inputting the low-resolution mask and the low-level features into the information loss region detection structure, predicting a low-resolution information loss mask using a fully convolutional network, upsampling this mask and fusing it with a higher-level feature, and then using a single convolution to predict a finer information loss mask;

[0027] Taking the information loss points detected by the lowest-level features as the root nodes, expand 4 child nodes from top to bottom, combine multi-scale features, construct an m-layer quadtree structure to represent different discrete points at multiple levels, select child nodes from higher-level feature maps through the quadtree structure to obtain more local part detail information, then convert these discrete points into an unordered pixel sequence, input it into a refinement network based on Transformer, and finally output the first mask.

[0028] As a preferred embodiment of the present invention, the specific steps of performing high-resolution regression on the low-resolution mask output by the Transfiner module through the DCT module to obtain the second mask are as follows:

[0029] Use n convolutional layers to extract the low-resolution mask of the multi-level feature map predicted by the rough mask head network in the Transfiner module, then use multiple connection layers for high-resolution regression and encode it into a DCT mask vector to participate in the training phase, and finally perform inverse DCT regression on the high-resolution mask as the second mask.

[0030] As a preferred embodiment of the present invention, the method of encoding into a DCT mask vector is specifically as follows:

[0031] Obtain the binary ground truth mask M based on the low-resolution mask output by the Transfier module gt , and use bilinear interpolation to adjust the size of M gt to a fixed K×K, then transform M K×K to the frequency domain space M DCT , and the expression of M DCT is as follows:

[0032]

[0033] where: K×K is the size of the mask; when w = 0, otherwise C(w) = 1;

[0034] Finally, obtain an n-dimensional DCT mask vector v from M DCT in a zigzag manner;

[0035] The method of performing inverse DCT regression on the high-resolution mask is specifically as follows:

[0036] Reconstruct with the DCT mask vector v Fill with 0 at the defective places, and then obtain through inverse DCT transformation. The expression is as follows:

[0037]

[0038] Finally, use bilinear interpolation upsampling to Restore to the high-resolution regression mask M rec 。

[0039] On the other hand, the present invention also provides a mechanical assembly image instance segmentation system, including the following modules:

[0040] An image dataset construction module, which establishes a synthetic depth image dataset and a real color image dataset of the mechanical assembly through the input mechanical assembly images; divides the synthetic depth image dataset and the real color image dataset into a training set and a test set;

[0041] An instance segmentation network construction module, which constructs an instance segmentation network including a backbone feature extraction network, a feature multi-scale fusion network, and a mask segmentation network;

[0042] A training module, which inputs the synthetic depth image training set and the real color image training set to train the constructed instance segmentation network to obtain a trained mechanical assembly instance segmentation model;

[0043] A test module, which conducts test analysis on the mechanical assembly instance segmentation model through the synthetic depth image test set and the real color image test set to obtain qualitative and quantitative results of instance segmentation;

[0044] An instance segmentation module, which inputs the mechanical assembly image to be detected into the trained mechanical assembly instance segmentation model and outputs the instance segmentation result.

[0045] As a preferred embodiment of the present invention, the mask segmentation network includes a Transfiner module and a DCT module. The Transfiner module detects loss information in the target edge region of the multi-level feature map and outputs a part edge mask with high precision as the first mask; the DCT module performs high-resolution regression on the low-resolution mask output by the Transfiner module to obtain a second mask; and fuses the first mask and the second mask to obtain the final mask segmentation image.

[0046] On the other hand, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the mechanical assembly image instance segmentation method as described in any embodiment of the present invention.

[0047] On the other hand, the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, it implements the mechanical assembly image instance segmentation method as described in any embodiment of the present invention.

[0048] The present invention has the following beneficial effects:

[0049] 1. The present invention provides a method for instance segmentation of mechanical assembly images, which introduces deformable convolution into the backbone extraction network to more flexibly and effectively obtain local information of mechanical assembly images.

[0050] 2. The present invention provides a method for instance segmentation of mechanical assembly images, which uses the Transfiner module as the main body of the mask branch and introduces the DCT module to improve the segmentation accuracy of the edge parts of parts. At the same time, high-resolution regression is performed on the initially predicted low-resolution masks to enhance the segmentation ability of the model for mechanical assemblies. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 is a schematic diagram of the method of the present invention;

[0052] Figure 2 is a multi-scale feature fusion structure diagram of the present invention;

[0053] Figure 3 is a schematic diagram of the encoding and decoding principle of the DCT module of the present invention;

[0054] Figure 4 is a flowchart of the method of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0055] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without making creative efforts fall within the protection scope of the present invention.

[0056] It should be understood that the step numbers used in the text are only for convenient description and do not limit the execution order of the steps.

[0057] It should be understood that the terms used in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include the plural forms.

[0058] The terms "include" and "comprise" indicate the presence of the described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or their combinations.

[0059] The term " / and" refers to any combination and all possible combinations of one or more of the related listed items, and includes these combinations.

[0060] Example 1:

[0061] Refer to Figure 1 , a method for instance segmentation of mechanical assembly images, comprising the following steps:

[0062] Build an instance segmentation network including a backbone feature extraction network, a feature multi-scale fusion network, and a mask segmentation network; the backbone feature extraction network is used to initially extract the initial feature map of the input image, the feature multi-scale fusion network performs feature fusion processing based on the initial feature map to generate a multi-level feature map; the mask segmentation network outputs the corresponding mask segmentation image based on the multi-level feature map;

[0063] Among them, the mask segmentation network includes a Transfiner module and a DCT module. The Transfiner module detects loss information in the target edge region of the multi-level feature map and outputs a part edge mask with high precision as the first mask; the DCT module performs high-resolution regression on the low-resolution mask output by the Transfiner module to obtain a second mask; the first mask and the second mask are fused to obtain the final mask segmentation image;

[0064] Input the mechanical assembly image dataset to train the constructed instance segmentation network to obtain a trained mechanical assembly instance segmentation model;

[0065] Input the mechanical assembly image to be detected into the trained mechanical assembly instance segmentation model and output the instance segmentation result.

[0066] Refer to Figure 4 , as a preferred embodiment of the present invention, the training steps of the instance segmentation network are:

[0067] Respectively establish a synthetic depth image dataset and a real color image dataset of the mechanical assembly;

[0068] Divide the synthetic depth image dataset and the real color image dataset into a training set and a test set to obtain a synthetic depth image training set, a real color image training set, a synthetic depth image test set, and a real color image test set;

[0069] Train the instance segmentation network through the synthetic depth image training set and the real color image training set, train the network to the set training batch, and save the optimal model during the training process.

[0070] Test and analyze the optimal model through the synthetic depth image test set and the real color image test set to obtain the qualitative and quantitative results of instance segmentation.

[0071] As a preferred implementation of this embodiment, the synthesized depth image dataset is obtained by using modeling software to build a three-dimensional model of a mechanical assembly, using annotation software to annotate each part instance with different colors, and then using data set synthesis software to generate depth images and label images, and finally converting them into a Cocoa format dataset;

[0072] The real color image dataset is obtained by using an RGB-D camera to shoot mechanical assembly images, processing them to obtain color images of appropriate size, then using annotation software to annotate each part in the color image to obtain a label image, and finally generating a coco format dataset.

[0073] As a preferred implementation of this embodiment, the backbone feature extraction network is used to extract information from the original image into a feature map containing rich semantic information, wherein:

[0074] The backbone feature extraction network is used to extract information from the original image into a feature map containing rich semantic information, where:

[0075] The backbone feature extraction network uses the ResNet101 network as the backbone network. The ResNet101 network includes five stages in sequence: Res1, Res2, Res3, Res4, and Res5. In the Res1 and Res2 stages, conventional convolution operations are performed on the input features, and in the Res3, Res4, and Res5 stages, deformable convolution operations are performed on the input features.

[0076] The method of the deformable convolution operation is: adding an offset variable to the sampling point position in the conventional convolution operation to adaptively adjust the sampling position according to the target geometric features.

[0077] As a preferred implementation of this embodiment, see Figure 2 The multi-scale feature fusion network performs feature fusion processing based on the preliminary feature map, and the specific steps of generating a multi-level feature map include:

[0078] Convolution is performed on the feature map C5 output from the Res5 stage to generate the feature map P5. The feature map P5 is upsampled and fused with the feature map C4 output from the Res4 stage to generate the feature map P4. The feature map P4 is upsampled and fused with the feature map C3 output from the Res3 stage to generate the feature map P3.

[0079] Obtain a multi-level feature map including feature map P5, feature map P4 and feature map P3;

[0080] The 2* upsampling in the figure indicates 2-step upsampling.

[0081] As a preferred implementation manner of this embodiment, the specific steps for the Transfiner module to detect loss information in the target edge region of the multi-level feature map and output a part edge mask with high precision as the first mask are as follows:

[0082] Predict a low-resolution mask of the multi-level feature map through the rough mask head network in the module, input the low-resolution mask and the low-level features into the information loss region detection structure, use a fully convolutional network to predict the low-resolution information loss mask, upsample this mask and fuse it with the features of the higher layer, and then use a single convolution to predict a finer information loss mask;

[0083] Take the information loss points detected by the lowest-level features as the root nodes, expand 4 child nodes from top to bottom, combine multi-scale features, construct an m-layer quadtree structure to represent different discrete points at multiple levels, select child nodes from the higher-level feature maps through the quadtree structure to obtain more local part detail information, then convert these discrete points into an unordered pixel sequence, input it into the refinement network based on Transformer, and finally output the first mask.

[0084] As a preferred implementation manner of this embodiment, the specific steps for the DCT module to perform high-resolution regression on the low-resolution mask output by the Transfiner module to obtain the second mask are as follows:

[0085] Use n convolutional layers to extract the low-resolution mask of the multi-level feature map predicted by the rough mask head network in the Transfiner module, then use multiple connection layers for high-resolution regression and encode it as a DCT mask vector to participate in the training stage, and finally perform inverse DCT regression on the high-resolution mask as the second mask.

[0086] As a preferred implementation manner of this embodiment, the method for encoding as a DCT mask vector is specifically as follows:

[0087] Obtain a binary ground truth mask M based on the low-resolution mask output by the Transfier module gt , use bilinear interpolation to adjust the size of M gt to a fixed K×K, and then transform M K×K to the frequency domain space M DCT , and the expression of M DCT is as follows:

[0088]

[0089] where: K×K is the size of the mask; when w = 0, otherwise C(w) = 1;

[0090] Finally, in a zigzag manner from MDCT obtain an n-dimensional DCT mask vector v from

[0091] The method for regressing the high-resolution mask through the inverse DCT is specifically as follows:

[0092] reconstruct with the DCT mask vector v fill with 0 at the defective places, and then obtain through the inverse DCT transform The expression is as follows:

[0093]

[0094] Finally, use bilinear interpolation upsampling to restore to the high-resolution regression mask M rec .

[0095] Embodiment 2:

[0096] A mechanical assembly image instance segmentation system includes the following modules:

[0097] An image dataset construction module, which establishes a synthetic depth image dataset and a real color image dataset of the mechanical assembly through the input mechanical assembly image; divides the synthetic depth image dataset and the real color image dataset into a training set and a test set;

[0098] An instance segmentation network construction module, which constructs an instance segmentation network including a backbone feature extraction network, a feature multi-scale fusion network, and a mask segmentation network;

[0099] A training module, which inputs the synthetic depth image training set and the real color image training set to train the constructed instance segmentation network, and obtains a trained mechanical assembly instance segmentation model;

[0100] A test module, which performs test analysis on the mechanical assembly instance segmentation model through the synthetic depth image test set and the real color image test set, and obtains the qualitative and quantitative results of the instance segmentation;

[0101] An instance segmentation module, which inputs the mechanical assembly image to be detected into the trained mechanical assembly instance segmentation model and outputs the instance segmentation result.

[0102] As a preferred implementation method of this embodiment, the mask segmentation network includes a Transfiner module and a DCT module. The Transfiner module detects the loss information of the target edge region of the multi-level feature map and outputs a high-precision part edge mask as the first mask; the DCT module performs high-resolution regression on the low-resolution mask output by the Transfiner module to obtain a second mask; the first mask and the second mask are fused to obtain the final mask segmentation image.

[0103] Example 3:

[0104] This embodiment provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the method described in any embodiment of the present invention is implemented.

[0105] Example 4:

[0106] This embodiment provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the method described in any embodiment of the present invention is implemented.

[0107] In the embodiments of the present application, "at least one" means one or more, and "a plurality" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent the cases of A existing alone, A and B existing simultaneously, and B existing alone. Where A and B may be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after. "At least one of the following" and its similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, and c may represent: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, and c may be single or multiple.

[0108] Those of ordinary skill in the art can realize that the units and algorithm steps described in the embodiments disclosed herein can be implemented by a combination of electronic hardware, computer software, and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0109] Those skilled in the art can clearly understand that for the convenience and conciseness of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the foregoing method embodiments and will not be described herein again.

[0110] In several embodiments provided by the present application, if any function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or a part of this technical solution can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present application. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks, or optical discs that can store program codes.

[0111] The above are only the embodiments of the present invention, and do not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the content of the specification and drawings of the present invention, or directly or indirectly applied in other related technical fields, shall be equally included in the patent protection scope of the present invention.

Claims

1. A method for instance segmentation of mechanical assembly images, characterized in that Including the following steps: Construct an instance segmentation network including a backbone feature extraction network, a feature multi-scale fusion network, and a mask segmentation network; the backbone feature extraction network is used to initially extract a preliminary feature map of the input image, the feature multi-scale fusion network performs feature fusion processing based on the preliminary feature map to generate a multi-level feature map; the mask segmentation network outputs a corresponding mask segmentation image based on the multi-level feature map; Among them, the mask segmentation network includes a Transfiner module and a DCT module. The Transfiner module detects loss information in the target edge region of the multi-level feature map and outputs a high-precision part edge mask as the first mask; the DCT module performs high-resolution regression on the low-resolution mask output in the Transfiner module to obtain a second mask; the first mask and the second mask are fused to obtain the final mask segmentation image; Input the mechanical assembly image dataset to train the constructed instance segmentation network to obtain a trained mechanical assembly instance segmentation model; Input the mechanical assembly image to be detected into the trained mechanical assembly instance segmentation model and output the instance segmentation result.

2. The method for instance segmentation of a mechanical assembly image according to claim 1, wherein The backbone feature extraction network is used to extract the information in the original image into a feature map containing rich semantic information, where: The backbone feature extraction network uses the ResNet101 network as the backbone network. The ResNet101 network includes five stages of Res1, Res2, Res3, Res4, and Res5 in sequence. Among them, the Res1 and Res2 stages perform conventional convolution operations on the input features, and the Res3, Res4, and Res5 stages perform deformable convolution operations on the input features; The method of the deformable convolution operation is: adding an offset variable that adaptively adjusts the sampling position according to the target geometric feature at the sampling point position in the conventional convolution operation.

3. A method for instance segmentation of mechanical assembly images according to claim 2, characterized in that, The specific steps for the feature multi-scale fusion network to perform feature fusion processing based on the preliminary feature map to generate a multi-level feature map include: Convolve the feature map C5 output in the Res5 stage to generate a feature map P5, upsample the feature map P5 and fuse it with the feature map C4 output in the Res4 stage to generate a feature map P4, upsample the feature map P4 and fuse it with the feature map C3 output in the Res3 stage to generate a feature map P3; Obtain a multi-level feature map including the feature map P5, the feature map P4, and the feature map P3.

4. A method for instance segmentation of mechanical assembly images according to claim 1, characterized in that, The specific steps for the Transfiner module to detect loss information in the target edge region of the multi-level feature map and output a high-precision part edge mask as the first mask are: Predict the low-resolution mask of the multi-level feature map through the rough mask head network in the module, input the low-resolution mask and the low-level features into the information loss region detection structure, use a fully convolutional network to predict the low-resolution information loss mask, upsample the mask and fuse it with the feature of the higher layer, and then use a single convolution to predict a finer information loss mask; Take the information loss points detected by the lowest-level features as the root nodes, expand 4 child nodes from top to bottom, combine multi-scale features, and construct an m-layer quadtree structure to represent different discrete points at multiple levels. Select child nodes from the higher-level feature maps through the quadtree structure to obtain more local part detail information, then convert these discrete points into an unordered pixel sequence, input it into a refinement network based on Transformer, and finally output the first mask.

5. A method for instance segmentation of mechanical assembly images according to claim 4, characterized in that, The specific steps for obtaining the second mask by performing high-resolution regression on the low-resolution mask output in the Transfiner module through the DCT module are as follows: Use n convolutional layers to extract the low-resolution mask of the multi-level feature maps predicted by the rough mask head network in the Transfiner module, then use multiple connection layers for high-resolution regression and encode it as a DCT mask vector to participate in the training phase, and finally perform inverse DCT regression on the high-resolution mask as the second mask.

6. A method for instance segmentation of mechanical assembly images according to claim 5, characterized in that, The method for encoding as a DCT mask vector is specifically as follows: Obtain the binary ground truth mask M based on the low-resolution mask output by the Transfier module gt , and use bilinear interpolation to resize M gt to a fixed size of K×K, and then transform M K×K to the frequency domain space M DCT , and the expression of M DCT is as follows: where: K×K is the size of the mask; when w = 0, otherwise C(w) = 1; Finally, obtain an n-dimensional DCT mask vector v from M in a zigzag manner; DCT ​ The method for performing inverse DCT regression on the high-resolution mask is specifically as follows: Reconstruct using the DCT mask vector v Fill with 0 at the defective positions and then obtain through inverse DCT transform The expression is as follows: Finally, use bilinear interpolation upsampling to restore to the high-resolution regression mask M rec .

7. A mechanical assembly image instance segmentation system, characterized in that, It includes the following modules: An image dataset construction module, which establishes a synthetic depth image dataset and a real color image dataset of the mechanical assembly by inputting the mechanical assembly image; divides the synthetic depth image dataset and the real color image dataset into a training set and a test set; An instance segmentation network construction module, which constructs an instance segmentation network including a backbone feature extraction network, a feature multi-scale fusion network, and a mask segmentation network; A training module, which inputs the synthetic depth image training set and the real color image training set to train the constructed instance segmentation network to obtain a trained mechanical assembly instance segmentation model; A test module, which performs test analysis on the mechanical assembly instance segmentation model through the synthetic depth image test set and the real color image test set to obtain qualitative and quantitative results of the instance segmentation; An instance segmentation module, which inputs the mechanical assembly image to be detected into the trained mechanical assembly instance segmentation model and outputs the instance segmentation result.

8. A mechanical assembly image instance segmentation system according to claim 7, wherein The mask segmentation network includes a Transfiner module and a DCT module. The Transfiner module detects the loss information of the target edge region of the multi-level feature maps and outputs a part edge mask with high precision as the first mask; the DCT module performs high-resolution regression on the low-resolution mask output in the Transfiner module to obtain the second mask; the first mask and the second mask are fused to obtain the final mask segmentation image.

9. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method according to any one of claims 1 to 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method according to any one of claims 1 to 6.

Images