A complex scene bolt data synthesis and recognition method based on domain adaptation

Abstract

The application discloses a kind of based on field adaptation's complex scene bolt data synthesis and identification method, it is related to engineering computer vision technical field.In virtual simulation environment, construct three-dimensional model library, the geometric form and surface material of component are parameterized modeling;Again, the component arrangement mode, illumination condition and camera pose of simulation scene are set to field randomization, render and generate the synthesis image data with instance segmentation annotation;Subsequently, the synthesis data are input instance segmentation network training, introduce field adaptation mechanism to reduce the feature distribution difference between synthesis data and real data;Finally, the model of training completion is used to realize the identification and instance segmentation of bolt component in real complex scene.The application generates high-fidelity synthesis data by parameterized modeling and field randomization, improves field adaptation effect by combining adversarial feature alignment and contrast constraint, strengthens the adaptability of model to complex scene, without relying on large-scale real labeled data, reduces data acquisition cost.

Description

Technical Field

[0001] This invention relates to the field of engineering computer vision technology, specifically to a method for synthesizing and recognizing bolt data in complex scenes based on domain adaptation. Background Technology

[0002] As bolted components serve as core connecting elements in industrial manufacturing and construction engineering, their installation quality and condition monitoring directly impact equipment operational stability and structural safety. In complex environments such as steel structure construction sites and industrial equipment assembly workshops, bolts, nuts, and washers are numerous and densely distributed, often facing challenges such as varying lighting, background interference, component occlusion, and surface contamination. Traditional manual inspection methods are inefficient and prone to errors, failing to meet the demands of automated production and intelligent operation and maintenance. With the development of computer vision and deep learning technologies, image-based bolt component recognition and segmentation methods have become a research hotspot. Automated inspection methods enable accurate identification, positioning, and condition assessment of components, providing data support for subsequent fastening inspections and quality evaluations, and possessing significant engineering application value.

[0003] Existing deep learning-based bolt recognition technologies primarily rely on real-world image datasets for model training. However, the acquisition and annotation of these datasets face numerous challenges. Firstly, acquiring bolt images in complex scenarios requires specialized equipment and on-site coordination, which is time-consuming, labor-intensive, and costly. Secondly, pixel-level instance segmentation and annotation demands a high level of expertise, has a long annotation cycle, and is prone to human error, making it difficult to acquire high-quality real-world datasets on a large scale. Furthermore, significant differences in lighting, background, and component states across different scenarios result in insufficient generalization ability for models trained on single-scenario datasets. This leads to a substantial decrease in recognition accuracy when applied across different scenarios, limiting the practical application of the technology.

[0004] Chinese Patent Publication No. CN119693411A proposes a bolt pose estimation method based on deep learning and template matching. This patent acquires images of overhead contact line cantilever bolts and annotates bounding boxes and hexagonal region information. It then combines deep learning and template matching techniques to identify bolt corner points and 6D pose, effectively improving the accuracy and detection efficiency of bolt pose estimation in specific scenarios and overcoming the weakness of traditional template matching methods in generalization. However, this technology still has significant shortcomings: its synthetic data generation process lacks comprehensive simulation of complex scene elements, failing to consider dynamic changes in illumination intensity and camera pose, as well as the real-world occlusion of components, resulting in significant differences in feature distribution between synthetic and real data. Furthermore, the model training does not incorporate an effective domain adaptation mechanism, making it difficult to eliminate the domain offset problem between synthetic and real data, leading to insufficient recognition stability in multi-scenario applications.

[0005] To address the issue of misalignment between synthetic and real data domains in the aforementioned patents, Chinese Patent Publication No. CN116922385A proposes an optimized and improved method and apparatus for bolt identification and installation / removal. This patent constructs an identification model library and a parameter database. The collected bolt data is input into the model library to obtain model information, and then the parameter database is retrieved to generate operation information. This improves the targeting and efficiency of bolt identification to some extent, indirectly alleviating the limitations of training with a single data domain. However, this technology still has shortcomings: it relies on a pre-set parameter database for matching and identification, failing to achieve dynamic generation and adaptive optimization of synthetic data, and thus cannot flexibly adapt to bolt components of different specifications and scenarios. Furthermore, it does not employ a feature-level domain alignment strategy, meaning that in complex scenarios with sudden changes in lighting and severe background interference, the identification accuracy and robustness still need improvement, making it difficult to meet the application requirements of high-precision instance segmentation.

[0006] Therefore, there is an urgent need for a bolt component data synthesis and recognition technology that can comprehensively simulate the characteristics of complex scenarios and achieve efficient matching between synthetic data and real data. Summary of the Invention

[0007] Based on the aforementioned technical problems, this application discloses a data synthesis and recognition method for bolted components in complex scenarios, specifically including:

[0008] A 3D model library containing bolts, nuts, and washers is established in a virtual simulation environment, and the geometry and surface material of bolt components are parametrically modeled.

[0009] Domain randomization is applied to the component arrangement, lighting conditions, and camera pose in the simulation scene to generate diverse scene combinations;

[0010] The simulation scene is rendered based on the randomization settings to generate synthetic image data with instance segmentation annotation information;

[0011] The synthesized image data is input into an instance segmentation network for training, and a domain adaptation mechanism is introduced during the training process to reduce the feature distribution difference between the synthesized data and the real data.

[0012] The trained instance segmentation model is used to identify and segment bolt components in real-world complex scenarios.

[0013] Preferably, the parametric modeling of the surface material is based on physically based rendering theory, and the optical properties of the metallic material are characterized by a bidirectional reflectance distribution function, the formula of which is: ,in It is a two-way reflection distribution function. Let be the incident light direction vector. Let be the direction vector of the emitted light. For half-range vectors, For the surface normal vector, For Fresnel reflection, For geometric occlusion, This is the micro-surface distribution term.

[0014] Preferably, the component arrangement, lighting conditions, and camera pose in the simulation scene are set using domain randomization. The domain randomization of the component arrangement is achieved through a spatial probability distribution model, and the spatial position coordinates of the components are... Satisfying three-dimensional uniform distribution, attitude angle The probabilistic model for occlusion relationships, generated through random sampling of Euler angles, is as follows: ,in The occlusion coefficient is... The area of ​​the overlapping region of the structural components. The total surface area of ​​the structural components.

[0015] Preferably, the domain randomization of the illumination conditions adopts an illumination intensity attenuation model, and the illumination intensity of the point light source varies with distance to satisfy: ,in The initial intensity of the light source, The distance from the light source to the surface of the component. This is an attenuation correction factor, and the light color is mapped to RGB values ​​using a color temperature conversion formula: ,in This is the color temperature conversion matrix. These are CIE standard chromaticity coordinates.

[0016] Preferably, in the domain randomization of the camera pose, the camera intrinsic focal length With field of view Satisfies geometric relations: ,in For the image width, the camera rotation matrix in the extrinsic parameters is obtained through quaternion transformation: ,in , , For unit quaternions, This is the camera rotation matrix.

[0017] Preferably, the total loss function of the instance segmentation network is a multi-task loss fusion form, and the formula is: ,in For category classification loss, For bounding box regression loss, For pixel-level mask loss, For domain adaptive loss, To compare the losses, and This is the loss weighting coefficient.

[0018] Preferably, the domain adaptation mechanism adopts an adversarial feature alignment architecture, which eliminates domain differences through a minimax game between the domain discriminator and the feature extractor. The domain adaptation loss function is: ,in For the synthetic data domain, For the real data domain, For feature extractor, For domain discriminator, For domain adaptive loss, For synthetic data samples, This is a real data sample.

[0019] Preferably, the domain adaptation mechanism further includes a feature-level contrast constraint module, which improves the domain adaptation effect by strengthening the aggregation of similar features and the separation of dissimilar features. The contrast loss formula is as follows: ,in The number of samples used for comparison. The cosine similarity function is used. For samples of the same type, As an outlier sample, This is the temperature coefficient.

[0020] Preferably, the bounding box coordinates in the instance segmentation annotation are obtained through a 3D bounding box projection transformation, and the projection formula is: , ,in For two-dimensional image coordinates, In three-dimensional space coordinates, Focal length The image principal point coordinates are used, and the pixel-level mask is determined by a depth threshold. If and only if ,otherwise ,in For pixels The corresponding depth value, For pixels The mask value, The depth range of the structural components.

[0021] Preferably, the feature fusion of the instance segmentation network adopts an attention weighting mechanism, and the fused feature map is as follows: ,in For the first Layer feature map, The number of feature levels participating in the fusion. The attention weights are calculated using the following formula: , , For the first , The saliency score of the layer features is calculated through global average pooling and fully connected layers.

[0022] Compared with the prior art, the technical solution of this application has the following technical effects:

[0023] This invention parametrically models the geometry and surface material of bolt components, combining physical rendering theory and bidirectional reflection distribution function to accurately simulate the optical properties of metal materials, significantly improving the realism of the synthetic data. By modeling spatial probability distribution, illumination intensity attenuation, and camera pose geometry, it achieves full-domain randomization of component arrangement, illumination conditions, and camera pose. The generated synthetic data covers diverse features in complex scenes, providing a rich and high-fidelity data source for model training, fundamentally solving the problem of scarce real-world labeled data.

[0024] The domain adaptation mechanism of this invention adopts an adversarial feature alignment architecture and a feature-level contrast constraint module. Through the minimax game between the domain discriminator and the feature extractor, and in conjunction with contrast loss, it strengthens the aggregation of similar features and the separation of dissimilar features, effectively reducing the feature distribution difference between synthetic data and real data. This dual domain adaptation design significantly reduces the impact of domain offset on model performance, enabling the model to quickly adapt to problems such as lighting changes, background interference, and component occlusion in real complex scenes, and greatly improves the generalization ability of recognition and segmentation.

[0025] The instance segmentation network of this invention integrates category classification, bounding box regression, pixel-level mask loss, domain adaptation, and contrastive loss through a multi-task fusion loss function, achieving collaborative optimization of each task. Feature fusion employs an attention-weighted mechanism, dynamically allocating feature weights at each level based on saliency scores, thus enhancing the feature extraction capability for small-scale, structurally similar bolt components. These designs enable the model to accurately distinguish bolts, nuts, and washers in complex scenes, accurately outputting the component's category, spatial location, and pixel-level mask, thereby improving the accuracy of recognition and segmentation.

[0026] This invention does not rely on large-scale real-world labeled data. By efficiently generating synthetic data with instance segmentation annotations, it reduces the cost of data acquisition and annotation. The model training and inference process is simple and efficient, and it is suitable for complex application scenarios in fields such as industrial manufacturing and building steel structures. It can provide reliable visual perception support for the automated assembly, fastening inspection and quality assessment of bolted components, promote the automation and intelligent upgrading of related industries, and has significant engineering application value.

[0027] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the preferred embodiments of this application are described in detail below with reference to the accompanying drawings.

[0028] The above and other objects, advantages and features of this application will become more apparent to those skilled in the art from the following detailed description of specific embodiments in conjunction with the accompanying drawings. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In all drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn to scale.

[0030] Based on the description of the figures and their corresponding technical content in the document, the titles of the figures are as follows:

[0031] Figure 1 A schematic diagram illustrating the process steps of the data synthesis and identification method for bolt components;

[0032] Figure 2 A schematic diagram of 3D models of bolts, nuts, and washers, along with scene domain randomization and annotation.

[0033] Figure 3 A comparative diagram of the rendered scene image of the bolt component and the instance segmentation annotations;

[0034] Figure 4 A schematic diagram illustrating the structure and data flow of an instance segmentation network (Encoder-Decoder architecture);

[0035] Figure 5 : Curves showing the changes in recognition accuracy and precision for each experimental group under different confidence thresholds;

[0036] Figure 6 Comparison curves of the comprehensive performance scores of each experimental group under different interference intensity levels;

[0037] Figure 7 : Curves showing the change in average recognition accuracy of each experimental group under different training data volumes;

[0038] Figure 8 A display diagram showing the segmentation and annotation results of bolt, nut, and washer examples in different scenarios. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. In the following description, specific details such as specific configurations and components are provided merely to help fully understand the embodiments of this application. Therefore, those skilled in the art should understand that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. In addition, for clarity and brevity, descriptions of known functions and structures are omitted in the embodiments.

[0040] It should be understood that the phrase "an embodiment" or "this embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "an embodiment" or "this embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments.

[0041] Furthermore, reference numerals and / or letters may be repeated in different examples within this application. Such repetition is for the purpose of simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or settings discussed.

[0042] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, B exists alone, and A and B exist simultaneously. The term " / and" in this article describes another type of relationship between related objects, indicating that two relationships can exist. For example, A / and B can mean: A exists alone, and A and B exist alone. In addition, the character " / " in this article generally indicates that the related objects before and after it are in an "or" relationship.

[0043] In this article, the term "at least one" is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, "at least one of A and B" can mean: A exists alone, A and B exist simultaneously, or B exists alone.

[0044] It should also be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion.

[0045] Example 1

[0046] This embodiment mainly describes a method for synthesizing and recognizing bolt data in complex scenarios based on domain adaptation, such as... Figures 1-2 As shown, it specifically includes:

[0047] A 3D model library containing bolts, nuts, and washers is established in a virtual simulation environment, and the geometry and surface material of bolt components are parametrically modeled.

[0048] Domain randomization is applied to the component arrangement, lighting conditions, and camera pose in the simulation scene to generate diverse scene combinations;

[0049] The simulation scene is rendered based on the randomization settings to generate synthetic image data with instance segmentation annotation information;

[0050] The synthesized image data is input into an instance segmentation network for training, and a domain adaptation mechanism is introduced during the training process to reduce the feature distribution difference between the synthesized data and the real data.

[0051] The trained instance segmentation model is used to identify and segment bolt components in real-world complex scenarios.

[0052] Furthermore, the 3D model library focuses on three core components: bolts, nuts, and washers. All parameters are defined based on an industrial standardization system. Bolt parameters include nominal diameter, pitch, effective length, head thickness, head width across flats, thread angle, and chamfer size. The thread angle is uniformly set to 60°, and the chamfer size is 1 / 10 of the nominal diameter. Nut parameters include thread specification (perfectly matching the bolt), width across flats, thickness, internal chamfer, and effective thread depth. The width across flats and thickness meet a standardized ratio of 1.5:1. Washer parameters include inner diameter (0.3-0.5mm larger than the bolt's nominal diameter), outer diameter (2-2.5 times the inner diameter), thickness, and edge chamfer radius, which is fixed at 1 / 2 of the thickness. All parameters are bound through parametric association functions. When any core parameter is modified, the associated parameters are automatically updated to ensure the model conforms to national standards such as GB / T196 and GB / T6170.

[0053] A material property library is built based on physically based rendering (PBR) theory, covering commonly used fastener materials such as carbon steel, stainless steel, and alloy steel. The optical properties of metallic materials are accurately characterized using the bidirectional reflectance distribution function (BRDF), with the following formula: ,in, This is a bidirectional reflection distribution function used to quantify the reflection distribution of incident light after passing through the surface of a material; Let be the incident light direction vector, and its unit vector form represents the direction in which the light ray strikes the material surface. Let be the direction vector of the outgoing light, and let be the unit vector form representing the propagation direction of the reflected light. For a half-range vector, the calculation formula is: , used to connect the incident light and the outgoing light directions; n is the surface normal vector, perpendicular to the material surface and pointing outward; For the Fresnel reflection term, the Schlick approximation formula is used. ,in Fresnel reflectivity of normal incident light (carbon steel) =0.15, stainless steel =0.20, alloy steel =0.18); For geometric occlusion, the Smith-Schlick model is used. , These are roughness conversion parameters; For the micro-surface distribution term, the GGX model is used. , It is the surface roughness coefficient of the material (with a value range of 0.05-0.3, corresponding to smooth to rough surfaces). and These are the dot products of the incident light direction vector, the outgoing light direction vector, and the surface normal vector, respectively, quantifying the angle between the light direction and the surface normal.

[0054] Furthermore, by randomly combining multi-dimensional parameters, the diverse characteristics of real-world complex scenarios are fully simulated, and a spatial probability distribution model is used to achieve the dynamic arrangement of components: the spatial position coordinates of the components. Satisfying three-dimensional uniform distribution ,in =-500mm、 =500mm, =-500mm、 =500mm, =0mm、 =800mm, ensuring uniform distribution within the preset scene area; attitude angle Generated by random sampling using Euler angles, each following a uniform distribution. This allows for multi-dimensional free rotation; occlusion relationships are quantitatively controlled through a probabilistic model, with the formula being... ,in This is the occlusion coefficient (valued between 1.2 and 1.8, dynamically adjusted based on scene density). Let be the area of ​​the overlapping region between any two components on the image plane. The model calculates the total surface area of ​​a single component (for bolts, it is calculated as the sum of the surface areas of the head and the shank; for nuts, it is calculated as the surface area of ​​a hexahedron; and for washers, it is calculated as the surface area of ​​a ring). This model makes the occlusion probability change non-linearly with the overlap ratio, simulating the natural overlap state of components in a real scene. The background environment supports random replacement and includes three basic backgrounds: metal components (grayscale value 180-220), concrete (grayscale value 120-160), and wood (grayscale value 80-120), as well as five mixed backgrounds (metal + concrete, metal + wood, etc.). The background texture resolution is uniformly 1024×1024 pixels.

[0055] Multi-dimensional randomization of lighting parameters is achieved using a physical lighting model: the intensity of a point light source varies with distance following an attenuation model. ,in The initial intensity of the light source (values ​​range from 500 to 1500 cd, corresponding to low-light to high-light environments). This is the distance from the light source to the surface of the component (values ​​range from 500 to 2000 mm). An attenuation correction factor (fixed at 10000 mm²) is used to ensure that the change in light intensity with distance conforms to physical laws; the light color is mapped to RGB values ​​using a color temperature conversion formula, which is: ,in This is the color temperature conversion matrix (constructed according to the CIE1931 standard colorimetric system). Using CIE standard chromaticity coordinates, the color temperature ranges from 2700K to 6500K (corresponding to warm to cool light). This formula enables precise color conversion of light at different color temperatures. Light source configuration supports random combinations of 1-4 light sources, with the light source type randomly selected from parallel light, point light, and area light. The illumination direction of parallel light is determined by a unit direction vector. Randomly generated The positions of point light sources and area light sources in the scene space Random sampling is used within the scene; ambient occlusion (AO) technology is introduced for all lighting scenes, with AO intensity ranging from 0.3 to 0.7, to enhance the sense of light and shadow in the scene and simulate the shadow effect of gaps on the surface of objects in a real environment.

[0056] Camera intrinsic and extrinsic parameters are co-randomized to simulate imaging effects under different shooting conditions: focal length in intrinsic parameters. With field of view Satisfy geometric relations Where w is the image sensor width (fixed at 36mm, corresponding to a full-frame camera), and the field of view is... The value range is 30°-120° (corresponding to telephoto to wide-angle shooting), focal length The camera principal point coordinates are then dynamically adjusted within the range of 17mm-108mm. , Fixed at the image center, i.e. The image resolution is fixed at 1920×1080 pixels; the rotation matrix in the extrinsic parameters is obtained through unit quaternion conversion, with the formula:

[0057]

[0058] in, , , For a unit quaternion, satisfying Quaternion components in Random sampling and normalization are performed within the range to avoid the singularity problem of the rotation matrix; the camera's spatial position is in three-dimensional space. Random sampling within the scene; shooting distance (distance from camera to scene center) ranges from 800-3000mm, corresponding to close-up, medium-range, and long-range shots; shooting angle is determined by azimuth. and elevation angle Random combinations are used to cover multiple perspectives, including top-down, bottom-up, and side views, ensuring that components occupy 5%-30% of the image.

[0059] Furthermore, such as Figure 3 As shown, based on a physically based rendering engine and a precise annotation algorithm, high-fidelity, precisely annotated synthetic image data is generated, specifically as follows:

[0060] Image rendering employs a path tracing rendering algorithm to render the virtual scene. Rendering parameters are set as follows: sampling rate of 64-128 samples / pixel, anti-aliasing mode of TAA (temporal anti-aliasing), rendering resolution of 1920×1080 pixels, color space of sRGB, and dynamic range of 16-bit floating-point. During rendering, the material optical properties of components, scene lighting distribution, and camera imaging models are fully integrated to accurately simulate the reflection, highlights, and shadows of metallic materials, as well as color changes under different lighting conditions. The resulting composite image closely resembles the real image in terms of texture detail, lighting effects, and color representation.

[0061] Instance segmentation annotations are generated synchronously with the rendering process, and include three core types of information: category labels, bounding box coordinates, and pixel-level masks.

[0062] Category tags: Using a unique thermal coding format, bolts are coded as [1,0,0], nuts as [0,1,0], and washers as [0,0,1], clearly distinguishing the three types of core components;

[0063] Bounding box coordinates: obtained through 3D bounding box projection transformation, the projection formula is as follows: , ,in Two-dimensional image coordinates (in pixels). The 3D spatial coordinates of the vertices of the 3D bounding box of the component in the camera coordinate system. For camera focal length, The coordinates of the principal point of the image (in pixels); the bounding box is axis-aligned, taking the minimum of all projected vertices. Minimum ,maximum ,maximum As bounding box coordinates ;

[0064] Pixel-level mask: generated by combining depth thresholding and semantic attribution determination, using the following formula: If and only if ,otherwise ,in For pixels The corresponding depth value, , The depth range of the component is determined by the minimum and maximum values ​​of the 3D bounding box in the Z-axis direction of the camera coordinate system. A mask value of 1 indicates that the pixel belongs to the target component, and 0 indicates that it belongs to the background or other components.

[0065] The labeled data format follows the COCO dataset standard, is stored in JSON file format, and contains three core fields: image information, label information, and category information. It can be directly used for training mainstream instance segmentation models.

[0066] Furthermore, such as Figure 4 As shown, the instance segmentation network is trained using an Encoder-Decoder architecture. Through multi-task loss collaborative optimization and feature enhancement strategies, the segmentation accuracy and generalization ability of the model are improved.

[0067] The Encoder uses ResNeSt-50 as its backbone network, containing four stages of convolutional blocks, each composed of multiple residual units. It enhances feature extraction capabilities through grouped convolutions and attention mechanisms. The Feature Pyramid Network (FPN) serves as the neck structure, upsampling and fusing the C2, C3, C4, and C5 feature maps output by the Encoder to generate feature maps at five scales: P2, P3, P4, P5, and P6. P2-P5 are used to detect small, medium, and large-sized targets, while P6 is used to generate candidate regions. The Decoder consists of deconvolutional and convolutional layers, upsampling the P2 feature map to the original image resolution while fusing feature information from each scale. The network head contains three branches: a classification branch (using fully connected layers and Softmax activation to output class probabilities), a bounding box regression branch (using fully connected layers and SmoothL1 loss to output bounding box offsets), and a mask generation branch (using convolutional layers and Sigmoid activation to output pixel-level masks).

[0068] Feature fusion employs an attention-weighted mechanism, and the formula for the fused feature map is as follows: ,in For the first Layer feature map ( =1,2,3,4,5 (corresponding feature maps), L is the number of feature levels participating in the fusion. The attention weights are calculated using the following formula: , For the first The saliency score of the layer features is calculated by performing global average pooling on the feature map of that layer to obtain the feature vector, and then inputting it into two fully connected layers (256 hidden neurons, ReLU activation function). The higher the score, the greater the contribution of the layer features to the target segmentation.

[0069] The network training employs a multi-task loss fusion approach, and the total loss function formula is as follows: ,in, For category classification loss, the cross-entropy loss function is used. , For the sample size, For the number of categories, For the sample Category The true label, To predict probabilities; For bounding box regression loss, the SmoothL1 loss function is used. , This is the actual bounding box offset. To predict the offset; For pixel-level mask loss, the Dice loss function is used. , For the real mask, For predicting the mask; For domain adaptive loss, To compare the losses; and These are the loss weighting coefficients, with values ​​of 0.5 and 0.1 respectively, used to adjust the contribution ratio of each loss item to the total loss.

[0070] Furthermore, by combining adversarial learning and contrastive learning, the difference in feature distribution between synthetic data and real data is reduced, thereby improving the model's adaptability in real-world scenarios.

[0071] An adversarial feature alignment architecture is adopted, with the core consisting of a feature extractor G and a domain discriminator D. Domain discriminator D achieves domain difference elimination through a minimax game. The domain discriminator D uses a fully connected network structure, taking as input a high-dimensional feature vector (2048 dimensions) output by the feature extractor G, and outputting the probability that a feature belongs to the true data domain. During training, the feature extractor G aims to learn domain-invariant features, making it impossible for the domain discriminator D to distinguish the feature source, while the domain discriminator D aims to accurately determine the feature source. The domain adaptive loss function formula is: ,in, This is the synthetic data domain, which contains all the generated synthetic image data; This is the real data domain, containing real-world bolt image data collected from actual scenes; This is the feature extractor (i.e., the Encoder part of the instance segmentation network). For domain discriminator; Domain-adaptive loss; For synthetic data samples; This is a real data sample; The feature vector obtained from the synthetic data sample by the feature extractor; The feature vector is obtained from real data samples by a feature extractor. The discrimination result of the domain discriminator on the features of the synthetic data; This represents the discrimination result of the domain discriminator on the features of the real data.

[0072] A feature-level contrast constraint module is introduced to enhance the aggregation of features of similar components and the separation of features of dissimilar components. The contrast loss formula is as follows: ,in, is the sample size; sim(·) is the cosine similarity function. This is the current anchor point sample; To and Positive samples belonging to the same category (randomly selected from components of the same category); To and Negative samples belonging to different categories (randomly selected from the other two categories of components); τ is a temperature coefficient (value 0.1) used to adjust the smoothness of the similarity distribution; through this loss function, the feature vectors of similar components are brought closer to each other in the high-dimensional space, while the feature vectors of dissimilar components are moved further apart, thereby further improving the domain adaptation effect.

[0073] Based on the trained instance segmentation model, efficient and accurate identification and segmentation of bolt components in real complex scenes are achieved. The input real scene image is resized (scaled to 1920×1080 pixels), normalized (pixel value divided by 255 to map to the [0,1] interval), and channel order converted (from BGR to RGB) while maintaining the original aspect ratio of the image and avoiding stretching and deformation.

[0074] The preprocessed image is input into the Encoder part of the model to extract multi-scale semantic features, and multi-scale feature maps are generated through FPN.

[0075] Candidate regions are generated based on feature maps using a sliding window strategy. The scale of the candidate regions ranges from 32×32 pixels to 1024×1024 pixels. Candidate regions with a confidence level higher than 0.5 are selected by non-maximum suppression (NMS).

[0076] Candidate regions are mapped to feature maps at various scales. The category and confidence of the candidate regions are predicted by the classification branch. The coordinates of the candidate regions are corrected by the bounding box regression branch. NMS (IOU threshold 0.3) is used again to remove overlapping candidate regions.

[0077] For the selected candidate regions, a pixel-level segmentation mask is generated through a mask generation branch. The mask threshold is set to 0.5, and pixels greater than the threshold are determined to be target components.

[0078] Output component category information (bolts, nuts, washers), bounding box coordinates, and instance identifier (unique ID) to provide data support for subsequent automated operations.

[0079] This implementation details how high-fidelity synthetic data is generated through parametric modeling and domain randomization, combined with a domain adaptation mechanism of adversarial feature alignment and contrast constraints, effectively reducing the domain difference between synthetic and real data. The instance segmentation network employs a multi-task loss fusion and attention feature fusion strategy to enhance the feature extraction capability of small-scale, structurally similar components, enabling accurate identification and segmentation of bolt components in complex scenarios without relying on large-scale real-world labeled data, reducing data acquisition costs, and providing reliable visual perception support for industrial automation and intelligent operation and maintenance.

[0080] Based on Embodiment 1, this embodiment describes in detail the verification of the technical solution of this application. Two mainstream and advanced bolt recognition technologies in the current industrial vision field are selected as control groups, and a complete verification system including hardware acquisition equipment and a software training platform is built. The hardware uses an industrial camera with a resolution of 1920×1080 pixels (lens focal length 25mm), developed based on the PyTorch 2.0 deep learning framework. All tests are conducted on the same hardware platform. The verification data is divided into synthetic datasets and real datasets: the synthetic dataset is generated using the method of this invention, covering 12 component specifications, 8 background types, a color temperature range of 2700K-6500K, and a field of view of 30°-120°; the real dataset is collected from building steel structure construction sites, industrial equipment assembly workshops, and bridge maintenance sites, totaling 8000 images (6000 for verification and 2000 for testing), all of which have been manually annotated at the pixel level.

[0081] The experimental group adopted the complete technical solution of this invention. Control group 1 adopted Mask R-CNN recognition technology based on real data augmentation (the current high-precision industrial-grade recognition solution, which expands real data through 10 data augmentation methods such as random cropping, flipping, and illumination distortion). Control group 2 adopted domain adaptive recognition technology based on general synthetic data (the mainstream cross-domain recognition solution, which uses publicly available industrial parts synthetic datasets and traditional domain adaptive methods). Validation metrics included recognition accuracy, segmentation intersection-over-union (IoU) ratio, inference speed, data acquisition cost, and robustness in complex scenarios (recognition accuracy under three extreme scenarios: sudden illumination changes, heavy occlusion, and mixed backgrounds). Specific test results are shown in the table below.

[0082] Evaluation indicators Experimental group (this invention) Control group 1 Control group 2 Bolt identification accuracy 98.7% 96.5% 91.3% Nut identification accuracy 97.9% 95.8% 89.7% Washer identification accuracy 96.8% 94.2% 88.5% Average recognition accuracy 97.8% 95.5% 89.8% Average Intersection over Union (IoU) 92.4% 90.3% 84.1% Inference speed (frames / second) 32 28 30 Data acquisition cost (ten thousand yuan) 0.3 4.8 1.2 Accuracy in scenes with sudden changes in lighting 96.5% 93.8% 85.2% Accuracy in heavily occluded scenes (50% occlusion rate) 94.2% 90.1% 82.6% Accuracy in mixed background scenes 95.7% 92.9% 86.4%

[0083] The table above clearly demonstrates the significant advantages of the technical solution of this invention in various core indicators. Regarding recognition accuracy, the experimental group achieved an average recognition accuracy of 97.8%, an improvement of 2.3 percentage points compared to control group 1 (95.5%) and 8.0 percentage points compared to control group 2 (89.8%). In particular, for small-scale, easily occluded washers, the experimental group achieved a recognition accuracy of 96.8%, 2.6 percentage points higher than control group 1, demonstrating the effectiveness of the attention feature fusion strategy. In terms of segmentation accuracy, the experimental group achieved an average IoU of 92.4%, exceeding control group 1 and control group 2 by 2.1 and 8.3 percentage points respectively, verifying the rationality of the multi-task loss fusion design of the instance segmentation network in this invention.

[0084] In terms of engineering practicality, the experimental group achieved an inference speed of 32 frames per second, higher than control group 1 (28 frames per second) and close to control group 2 (30 frames per second), fully meeting the needs of real-time industrial detection. The data acquisition cost was only 0.3 million yuan, only 6.25% of control group 1 (4.8 million yuan) and 25% of control group 2 (1.2 million yuan), demonstrating a significant cost advantage. This is attributed to the efficient generation capability of the high-fidelity synthetic data of this invention. In robustness tests in complex scenarios, the experimental group maintained an accuracy rate above 94% in three extreme scenarios: sudden changes in illumination, severe occlusion, and mixed backgrounds. This represents an average improvement of 2.5 percentage points compared to control group 1 and an average improvement of 10.8 percentage points compared to control group 2, fully demonstrating that the domain randomization setting and improved domain adaptive mechanism of this invention can effectively cope with complex interference in real-world scenarios.

[0085] From the perspective of the impact of confidence threshold on model recognition performance, such as Figure 5 As shown, the experimental group achieved a leading position in both accuracy and precision across the entire confidence threshold range: when the confidence threshold was 0.5 (a commonly used industrial threshold), the experimental group achieved an average accuracy of 97.8% and a precision of 98.2%, which were 2.3 and 1.9 percentage points higher than control group 1, respectively, and 8.0 and 7.5 percentage points higher than control group 2, respectively. Even when the confidence threshold was increased to 0.8 (a stringent identification standard), the experimental group's accuracy remained at 95.3%, far exceeding the 92.1% of control group 1 and the 83.6% of control group 2, and the precision remained consistently above 97%, demonstrating the accurate identification capability of the model under high confidence requirements. The smoothness of the curve also reflects the stability of the model's prediction results.

[0086] To visually demonstrate the differences in robustness among different schemes under varying interference intensities, such as Figure 6 As shown, the experimental group curve exhibited the smallest downward slope, with the overall performance score decreasing only from 98.5 to 91.2 points from level 1 to level 5, a drop of only 7.3 points. Control group 1 decreased from 96.2 to 87.5 points, a drop of 8.7 points; and control group 2 decreased from 90.3 to 78.8 points, a drop of 11.5 points. Notably, under level 5 extreme interference (light intensity change ±40% + occlusion rate 60% + mixed background noise), the experimental group score still exceeded 90 points, while control group 2 fell below 80 points. Furthermore, the confidence interval width of the experimental group curve was consistently less than 2.0, significantly narrower than that of control group 1 (3.2) and control group 2 (4.5). This indicates that the method of the present invention has stronger performance stability, smaller data fluctuations, and more significant robustness advantages under different interference intensities.

[0087] By analyzing the correlation between the amount of training data and the model's recognition accuracy, such as... Figure 7As shown, the performance improvement rate of the experimental group was significantly faster than that of the control group. When the amount of synthetic data reached 50,000 images, the accuracy rate reached 96.1%, which is close to the accuracy level of the control group 1 using 8,000 real data images. When the amount of data increased to 100,000 images, the accuracy rate further improved to 97.8%, far exceeding the two control groups. This fully demonstrates that the synthetic data generated by this invention has extremely high training value and can replace expensive real labeled data with large-scale, low-cost data.

[0088] The actual recognition performance of the model was verified by selecting two typical real-world scenarios. The technology presented in this application allows for a direct and intuitive demonstration of the results. Figure 8 As shown, it contains two subgraphs, left and right (denoted as...). Figure 8 a, Figure 8 (b) Both demonstrate the recognition and annotation effects of bolt components in real-world scenarios: Figure 8 The background is a wood-textured surface. Multiple bolts (ID10, 11, 12), nuts (ID13, 14, 15), and washers (ID16, 17, 18) are distributed in the scene. Some components are partially occluded. The model accurately selects all targets with red bounding boxes. The component category and unique ID are marked next to the box. The bounding box fits the target outline very well. Figure 8 The background is a cement-textured surface. The scene contains multiple long bolts (ID38, 39), nuts (ID40, 41), and washers of different sizes (ID43, 44, 45). Some bolts are stacked in a crisscross pattern. The model also marks all components completely with red bounding boxes. The categories and ID information are clearly correlated, with no missed detections, false detections, or bounding box offsets. This intuitively demonstrates the accurate recognition capability of the method of this invention under different backgrounds and different degrees of occlusion.

[0089] In summary, this invention, through parametric modeling and global randomization to generate high-fidelity synthetic data, combined with an improved domain adaptive mechanism and attention feature fusion network, comprehensively surpasses existing advanced technologies in terms of recognition accuracy, inference speed, cost control, and adaptability to complex scenarios. It effectively solves the shortcomings of traditional technologies, such as reliance on real data, high cost, and insufficient robustness, and provides a better solution for the automated identification of bolted components in industrial manufacturing and building steel structures.

[0090] The above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. For those skilled in the art, the present invention can have various modifications and variations. Any changes, modifications, substitutions, integrations, and parameter changes made to these embodiments within the spirit and principles of the present invention, without departing from the principles and spirit of the present invention, through conventional substitutions or to achieve the same function, fall within the scope of protection of the present invention.

Claims

1. A method for synthesizing and recognizing bolt data in complex scenarios based on domain adaptation, characterized in that, include: A 3D model library containing bolts, nuts, and washers is established in a virtual simulation environment, and the geometry and surface material of bolt components are parametrically modeled. Domain randomization is applied to the component arrangement, lighting conditions, and camera pose in the simulation scene to generate diverse scene combinations; The simulation scene is rendered based on the randomization settings to generate synthetic image data with instance segmentation annotation information; The synthesized image data is input into an instance segmentation network for training, and a domain adaptation mechanism is introduced during the training process to reduce the feature distribution difference between the synthesized data and the real data. The trained instance segmentation model is used to identify and segment bolt components in real-world complex scenarios.

2. The method for synthesizing and recognizing bolt data in complex scenes based on domain adaptation according to claim 1, characterized in that, The parametric modeling of the surface material is based on physically based rendering theory, using a bidirectional reflectance distribution function to characterize the optical properties of the metallic material. The formula is: ,in It is a two-way reflection distribution function. Let be the incident light direction vector. Let be the direction vector of the emitted light. For half-range vectors, For the surface normal vector, For Fresnel reflection, For geometric occlusion, This is the micro-surface distribution term.

3. The method for synthesizing and recognizing bolt data in complex scenes based on domain adaptation according to claim 1, characterized in that, The component layout, lighting conditions, and camera pose in the simulation scene are set using domain randomization. The domain randomization of the component layout is achieved through a spatial probability distribution model, and the spatial position coordinates of the components are... Satisfying three-dimensional uniform distribution, attitude angle The probabilistic model for occlusion relationships, generated through random sampling of Euler angles, is as follows: ,in The occlusion coefficient is... The area of ​​the overlapping region of the structural components. The total surface area of ​​the structural components.

4. The method for synthesizing and recognizing bolt data in complex scenes based on domain adaptation according to claim 3, characterized in that, The domain randomization of the illumination conditions adopts an illumination intensity attenuation model, and the illumination intensity of the point light source varies with distance as follows: ,in The initial intensity of the light source, The distance from the light source to the surface of the component. This is an attenuation correction factor, and the light color is mapped to RGB values ​​using a color temperature conversion formula: ,in This is the color temperature conversion matrix. These are CIE standard chromaticity coordinates.

5. The method for synthesizing and recognizing bolt data in complex scenes based on domain adaptation according to claim 4, characterized in that, In the domain randomization of the camera pose, the camera intrinsic focal length With field of view Satisfies geometric relations: ,in For the image width, the camera rotation matrix in the extrinsic parameters is obtained through quaternion transformation: ,in , , For unit quaternions, This is the camera rotation matrix.

6. The method for synthesizing and recognizing bolt data in complex scenes based on domain adaptation according to claim 1, characterized in that, The total loss function of the instance segmentation network is a multi-task loss fusion form, and the formula is: ,in For category classification loss, For bounding box regression loss, For pixel-level mask loss, For domain adaptive loss, To compare the losses, and This is the loss weighting coefficient.

7. The method for synthesizing and recognizing bolt data in complex scenes based on domain adaptation according to claim 1, characterized in that, The domain adaptation mechanism employs an adversarial feature alignment architecture, achieving domain difference elimination through a minimax game between the domain discriminator and the feature extractor. The domain adaptation loss function is: ,in For the synthetic data domain, For the real data domain, For feature extractor, For domain discriminator, For domain adaptive loss, For synthetic data samples, This is a real data sample.

8. The method for synthesizing and recognizing bolt data in complex scenes based on domain adaptation according to claim 7, characterized in that, The domain adaptation mechanism also includes a feature-level contrast constraint module, which improves the domain adaptation effect by strengthening the aggregation of similar features and the separation of dissimilar features. The contrast loss formula is as follows: ,in The number of samples used for comparison. The cosine similarity function is used. For samples of the same type, As an outlier sample, This is the temperature coefficient.

9. The method for synthesizing and recognizing bolt data in complex scenes based on domain adaptation according to claim 1, characterized in that, The bounding box coordinates in the instance segmentation annotation are obtained through a 3D bounding box projection transformation, and the projection formula is: , ,in For two-dimensional image coordinates, In three-dimensional space coordinates, Focal length The image principal point coordinates are used, and the pixel-level mask is determined by a depth threshold. If and only if ,otherwise ,in For pixels The corresponding depth value, For pixels The mask value, The depth range of the structural components.

10. The method for synthesizing and recognizing bolt data in complex scenes based on domain adaptation according to claim 1, characterized in that, The feature fusion of the instance segmentation network adopts an attention-weighted mechanism, and the fused feature map is as follows: ,in For the first Layer feature map, The number of feature levels participating in the fusion. The attention weights are calculated using the following formula: , , For the first , The saliency score of the layer features is calculated through global average pooling and fully connected layers.

Images