An industrial scene automatic semantic segmentation labeling method and system

Abstract

This invention discloses an automatic semantic segmentation and annotation method and system for industrial scenes, belonging to the field of automatic semantic segmentation and annotation. The method specifically includes: collecting an image sample set of an industrial scene and extracting sample information from each sample in the image sample set; inputting the sample information from each sample in the first sample set into a first large-scale visual model, manually annotating the output of the large-scale visual model to obtain an initial segmentation dataset; adjusting a second large-scale visual model, inputting a second sample set into the adjusted second large-scale visual model, manually correcting the output pre-annotated dataset to obtain a fine-tuned dataset; training the second large-scale visual model, collecting a set of real industrial images, and inputting the real industrial images into the trained second large-scale visual model to obtain a real automatic semantic segmentation and annotation dataset. This application improves the recognition accuracy of targets such as semiconductor components and enhances the clarity of region segmentation boundaries.

Description

Technical Field

[0001] This invention belongs to the field of automatic semantic segmentation and annotation, specifically relating to an automatic semantic segmentation and annotation method and system for industrial scenarios. Background Technology

[0002] Automatic semantic segmentation technology for industrial scenarios plays an important role in the field of intelligent manufacturing. This method can perform semantic parsing, region segmentation, and region type labeling on industrial images. Especially when facing complex industrial scenarios in semiconductors, automatic semantic segmentation technology can accurately identify and segment targets such as chips and metal wires, which is an important guarantee for completing subsequent tasks such as defect detection and quality assessment.

[0003] In existing technologies, automatic semantic segmentation and annotation methods for complex industrial scenarios in the semiconductor industry typically employ pre-trained models based on convolutional neural networks or Transformer architectures. These models undergo supervised training using manually labeled datasets and are then globally optimized using Markov random fields. However, in complex industrial semiconductor environments, existing automatic semantic segmentation and annotation methods suffer from the following problems: weak global optimization capabilities, leading to blurred region segmentation boundaries. Summary of the Invention

[0004] The purpose of this invention is to provide an automatic semantic segmentation and annotation method and system for industrial scenarios to solve problems such as blurred segmentation boundaries in existing technologies.

[0005] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0006] An automatic semantic segmentation and annotation method for industrial scenarios includes:

[0007] Collect an image sample set of an industrial scene, divide the image sample set into a first sample set and a second sample set, and extract the sample information of each sample in the image sample set. The sample information includes RGB images and depth maps.

[0008] The RGB images of each sample in the first sample set are input into the first large visual model. The output of the large visual model is manually labeled based on the depth map to obtain the initial segmentation dataset.

[0009] The second visual large model is adjusted based on the initial segmentation dataset. The second sample is input into the adjusted second visual large model, and the pre-labeled dataset output by the adjusted second visual large model is manually corrected to obtain the fine-tuned dataset. The adjustment is to iteratively update the second training model in the second visual large model by generating attention maps and feature weighted fusion. The pre-labeled dataset is obtained by applying a rule-aware function to optimize the labeling of the region adjacency graph constructed based on the output of the second training model.

[0010] The second vision model is trained based on the fine-tuned dataset. A real industrial image set is collected and input into the trained second vision model to obtain a real automatic semantic segmentation and annotation dataset.

[0011] Specifically, the RGB images of each sample in the first sample set are input into the first large-scale visual model. The output of the large-scale visual model is manually labeled based on the depth map to obtain the initial segmentation dataset, including:

[0012] Input the RGB images of each sample in the first sample set into the first visual large model, and obtain the initial mask set of the RGB images of each sample through interactive prompts;

[0013] Based on depth maps and RGB images, semantic segmentation is performed on each initial mask in the initial mask set through manual annotation, and the segmentation initial dataset is constructed based on the results of semantic segmentation.

[0014] Specifically, based on depth maps and RGB images, semantic segmentation is performed on each initial mask in the initial mask set through manual annotation. Based on the semantic segmentation results, a segmentation initial dataset is constructed, including:

[0015] Set a height threshold, calculate the height feature value of the region corresponding to each initial mask in the initial mask set based on the depth map of each sample in the first sample set, and compare the height feature value with the height threshold.

[0016] When the height feature value is greater than the height threshold, the boundary of the current initial mask is corrected by manual annotation based on the depth map and RGB image, and the region type is determined. The region of the current initial mask and the corresponding region type label are recorded to obtain the first region mask set and the first region type label set.

[0017] When the height feature value is less than or equal to the height threshold, the boundary of the current initial mask is corrected by manual annotation based on the RGB image, and the region type is determined. The region of the current initial mask and the corresponding region type label are recorded to obtain the second region mask set and the second region type label set.

[0018] An initial segmentation dataset is constructed based on a set of region masks and a set of region type labels. The set of region masks includes a first set of region masks and a second set of region masks. The set of region type labels includes a first set of region type labels and a second set of region type labels.

[0019] Specifically, the second vision model includes:

[0020] The second training model and the second inference model are integrated to obtain the second large visual model.

[0021] Specifically, the second visual model is adjusted based on the initial segmentation dataset. The second sample is input into the adjusted second visual model, and the pre-labeled dataset output by the adjusted second visual model is manually corrected to obtain a fine-tuned dataset, including:

[0022] Based on the sample information of each sample in the first sample set and the segmentation of the initial dataset, the second training model is iterated until the parameters of the second training model after iteration meet the preset iteration termination condition. The iteration is then completed, and the adjusted second training model and the second inference model are integrated to obtain the adjusted second visual large model.

[0023] The sample information of each sample in the second sample set is input into the adjusted second visual large model to obtain the pre-labeled dataset;

[0024] The pre-labeled dataset is manually corrected to obtain a segmentation correction dataset. The segmentation correction dataset and the initial segmentation dataset are then merged to obtain a fine-tuned dataset.

[0025] Specifically, based on the sample information of each sample in the first sample set and the initial segmentation dataset, the second training model is iteratively tested until the parameters of the iterated second training model meet the preset iteration termination condition. The iteration is then complete. The adjusted second training model and the second inference model are integrated to obtain the adjusted second visual large model, including:

[0026] The sample information of each sample in the first sample set is input into the second training model, and the general feature map of each sample in the first sample set is obtained through feature extraction.

[0027] A query vector is defined, and the query vector is interacted with a general feature map to generate an attention map, which includes: a region focus attention map, a region geometry attention map, and a region material attention map;

[0028] Based on the region focus attention map, region geometric attention map, and region material attention map, spatial weighted feature extraction and feature fusion are performed on the general feature map to obtain path features that associate geometric and material information of different regions.

[0029] The path features are input into the pre-trained decoder to obtain a predicted segmentation mask set. Based on the predicted segmentation mask set and the initial segmentation dataset, the total task loss value is calculated.

[0030] Set a differentiated learning rate strategy and calculate the gradient of the total task loss with respect to all parameters in the second training model;

[0031] Based on the differentiated learning rate strategy and gradient, all parameters in the second training model are iteratively updated until the parameters after iteration meet the preset iteration termination condition, at which point the iteration is completed and the adjusted second training model is obtained.

[0032] Based on the adjusted second training model and second inference model, the adjusted second visual large model is obtained through model ensemble.

[0033] Specifically, the sample information of each sample in the second sample set is input into the adjusted second vision model to obtain a pre-labeled dataset, including:

[0034] The sample information of each sample in the second sample set is input into the adjusted second training model to obtain the first prediction segmentation mask set and the first attention map;

[0035] A region adjacency graph is constructed based on the first predicted segmentation mask set, wherein nodes in the region adjacency graph represent segmentation mask regions and edges represent adjacent region relationships.

[0036] Set up a rule base, calculate the perception function value of each edge in the region adjacency graph, and construct a rule perception function based on the perception function value;

[0037] The region adjacency graph is globally optimized using a rule-aware function to obtain optimized region type labels.

[0038] Based on the optimized region type labels, the regions and region types corresponding to each segmentation mask in the first predicted segmentation mask set are pre-labeled to construct a pre-labeled dataset.

[0039] Specifically, the perceptual function value of each edge in the region adjacency graph is calculated, and a rule-based perceptual function is constructed based on the perceptual function value, including:

[0040] Calculate the cost of assigning each region type label to each region to obtain the data item;

[0041] Calculate the geometric data of the merged region after merging each pair of adjacent regions, and evaluate the geometric data based on the rule base;

[0042] Set a material similarity threshold, extract the material feature confidence of each pair of adjacent regions, and evaluate the material similarity of each pair of adjacent regions based on the material similarity threshold;

[0043] Based on the evaluation results of geometric data and material similarity, the perceptual function value of each edge in the region adjacency graph is calculated, and a perceptual smoothing term is designed based on the perceptual function value.

[0044] The data items and the perceptual smoothing items are summed to construct a rule-based perceptual function.

[0045] An automatic semantic segmentation and annotation system for industrial scenarios includes:

[0046] The sample acquisition module is used to acquire image sample sets of industrial scenes, divide the image sample sets into a first sample set and a second sample set, and extract sample information of each sample in the image sample set. The sample information includes RGB images and depth maps.

[0047] The manual annotation module is used to input the RGB images of each sample in the first sample set into the first visual large model, and manually annotate the output of the visual large model based on the depth map to obtain the initial segmentation dataset;

[0048] The fine-tuning and correction module is used to adjust the second visual large model based on the initial segmentation dataset. The second sample is input into the adjusted second visual large model, and the pre-labeled dataset output by the adjusted second visual large model is manually corrected to obtain the fine-tuned dataset.

[0049] The automatic annotation module is used to train the second vision large model based on the fine-tuning dataset. It collects a set of real industrial images and inputs the real industrial images into the trained second vision large model to obtain a real automatic semantic segmentation annotation dataset.

[0050] Specifically, the fine-tuning and correction module includes:

[0051] The model fine-tuning unit is used to iterate on the second training model based on the sample information of each sample in the first sample set and the segmentation of the initial dataset until the parameters of the second training model after iteration meet the preset iteration termination condition. When the iteration is completed, the adjusted second training model and the second inference model are integrated to obtain the adjusted second visual large model.

[0052] The pre-labeling unit is used to input the sample information of each sample in the second sample set into the adjusted second visual large model to obtain the pre-labeled dataset;

[0053] The data merging unit is used to manually correct the pre-labeled dataset to obtain a segmentation correction dataset, and to merge the segmentation correction dataset and the initial segmentation dataset to obtain a fine-tuned dataset.

[0054] Compared with existing technologies, the beneficial effects of this invention include: collecting a sample set of industrial scene images; manually annotating the first sample set using a first visual large model to construct an initial segmentation dataset; iteratively updating the second visual large model based on the initial segmentation dataset by generating multi-type attention maps and feature weighted fusion, constructing a region adjacency graph and a rule-aware function to pre-annotate the second sample set, and obtaining a fine-tuned dataset after manual correction; training the second visual large model using the fine-tuned dataset to achieve automatic semantic segmentation and annotation of real industrial scene images; solving the problem of blurred region segmentation boundaries caused by insufficient global optimization capabilities in complex industrial scenes of semiconductors in existing technologies, significantly improving the accuracy and clarity of segmentation boundaries, and making the boundaries of targets more explicit. Attached Figure Description

[0055] Figure 1 A flowchart of an automatic semantic segmentation and annotation method for industrial scenarios provided by the present invention;

[0056] Figure 2 A flowchart of the phased semantic segmentation annotation process provided by this invention;

[0057] Figure 3 A flowchart for generating the pre-labeled dataset provided by this invention;

[0058] Figure 4 This invention provides a structural diagram of an automatic semantic segmentation and annotation system for industrial scenarios. Detailed Implementation

[0059] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations thereof. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0060] The term "and / or" simply describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0061] Example 1

[0062] Please see Figures 1-3 The present invention provides an embodiment of an automatic semantic segmentation and annotation method for industrial scenarios, comprising the following specific steps:

[0063] Step S1: Collect an image sample set of industrial scene, divide the image sample set into a first sample set and a second sample set, and extract the sample information of each sample in the image sample set.

[0064] In this embodiment, the image sample set is divided into a first sample set and a second sample set according to a preset ratio. The preset ratio is set by those skilled in the art according to the actual situation. It should be noted that after collecting the image sample set of industrial scene, the images need to be preprocessed. The preprocessing specifically includes: outlier handling, data standardization, missing value imputation, time alignment and feature smoothing.

[0065] Step S2: Input the RGB images of each sample in the first sample set into the first large visual model, and manually annotate the output of the large visual model based on the depth map to obtain the initial segmentation dataset.

[0066] The specific steps of step S2 are as follows:

[0067] Step S201: Input the RGB images of each sample in the first sample set into the first visual large model, and obtain the initial mask set of the RGB images of each sample through interactive prompts.

[0068] In this embodiment, the interactive prompts include: dot prompts, box prompts, and mask prompts. Each mask in the initial mask set corresponds to a target region in the image, and the first large visual model can be the RITM interactive segmentation model.

[0069] Step S202: Based on the depth map and RGB image, perform semantic segmentation on each initial mask in the initial mask set by manual annotation, and construct the segmentation initial dataset based on the semantic segmentation results.

[0070] The specific steps of step S202 are as follows:

[0071] Step S2021: Set a height threshold. Based on the depth map of each sample in the first sample set, calculate the height feature value of the region corresponding to each initial mask in the initial mask set, and compare the height feature value with the height threshold.

[0072] In this embodiment, the height threshold is set by those skilled in the art according to the actual situation. The depth values ​​of all pixels in the region corresponding to each initial mask in the initial mask set are extracted, and the median of the depth values ​​is calculated as the height feature value of each initial mask set.

[0073] Step S2022: When the height feature value is greater than the height threshold, based on the depth map and RGB image, the boundary of the current initial mask is corrected by manual annotation, and the region type is determined. The region of the current initial mask and the corresponding region type label are recorded to obtain the first region mask set and the first region type label set.

[0074] In this embodiment, a label list is set, which includes: grains, metal wires, and contacts, etc. When the height feature value is greater than the height threshold, the annotator switches back and forth between the RGB image and the depth map for observation. On the depth map, the boundary of the object usually shows a large change in pixel value. Based on this, the boundary of each initial mask is corrected. After the correction is completed, the annotator observes the RGB image and uses the depth map as an auxiliary judgment tool to select a region type label for each initial mask, and records the region of each initial mask and the corresponding region type label.

[0075] Integrate the regions and region type labels corresponding to all initial masks with height feature values ​​greater than the height threshold to obtain the first region mask set and the first region type label set.

[0076] Step S2023: When the height feature value is less than or equal to the height threshold, based on the RGB image, the boundary of the current initial mask is corrected by manual annotation, and the region type is determined. The region of the current initial mask and the corresponding region type label are recorded to obtain the second region mask set and the second region type label set.

[0077] In this embodiment, when the height feature value is less than or equal to the height threshold, the annotator zooms in on the RGB image to the pixel level and observes the boundary of color, brightness and texture changes in each initial mask and its surroundings. Based on the observation results, the boundary of each initial mask is corrected. After the correction is completed, the region type corresponding to each initial mask is determined, and the region and the corresponding region type label of each initial mask are recorded.

[0078] By integrating the regions and region type labels corresponding to all initial masks whose height feature values ​​are less than or equal to the height threshold, a second region mask set and a second region type label set are obtained.

[0079] Step S2024: Construct an initial segmentation dataset based on the region mask set and the region type label set. The region mask set includes a first region mask set and a second region mask set. The region type labels include a first region type label set and a second region type label set.

[0080] In this embodiment, the first region mask set and the second region mask set are merged to obtain a region mask set, and the first region type label set and the second region type label set are merged to obtain a region type label set. The region mask set and the region type label set are used as core elements to construct the initial segmentation dataset.

[0081] Step S3: Adjust the second visual large model based on the initial segmentation dataset, input the second sample into the adjusted second visual large model, and manually correct the pre-labeled dataset output by the adjusted second visual large model to obtain the fine-tuned dataset.

[0082] The specific steps of step S3 are as follows:

[0083] Step S301: Based on the sample information of each sample in the first sample set and the segmentation of the initial dataset, perform iterative operations on the second training model until the parameters of the second training model after iteration meet the preset iteration termination condition. When the iteration is completed, the adjusted second training model and the second inference model are integrated to obtain the adjusted second visual large model.

[0084] The specific steps of step S301 are as follows:

[0085] Step S3011: Input the sample information of each sample in the first sample set into the second training model, and obtain the general feature map of each sample in the first sample set through feature extraction.

[0086] In this embodiment, the second visual large model is obtained by integrating the second training model and the second inference model. The second visual large model is a semantic segmentation visual large model (SAM) with the ability to segment with few or even zero samples. The RGB images and depth maps of each sample in the first sample set are stitched together in the channel dimension to obtain a multi-channel tensor.

[0087] The multi-channel tensor is input into the second training model, and the encoder in the second training model is used to extract features, outputting a general feature map of each sample in the first sample set.

[0088] Step S3012: Set a query vector, interact with the query vector and the general feature map to generate an attention map, the attention map including: region focus attention map, region geometry attention map and region material attention map.

[0089] In this embodiment, a query vector is set and initialized. The query vector is an independent learnable parameter row vector, including: region focus query vector, region geometry query vector, and region material query vector.

[0090] Flatten the spatial dimension of the general feature map, perform dot product operations between the query vector and the flattened general feature map in turn, and calculate the similarity score vector between each query vector and the feature vectors at all positions on the flattened general feature map.

[0091] The similarity score vector is reshaped back into a two-dimensional spatial structure, and the two-dimensional graph is normalized to generate a region focus attention map, a region geometric attention map, and a region material attention map.

[0092] Step S3013: Based on the region focus attention map, region geometric attention map, and region material attention map, perform spatial weighted feature extraction and feature fusion on the general feature map to obtain path features that associate geometric and material information of different regions.

[0093] In this embodiment, the dimension of the region focus attention map is expanded to the same dimension as the general feature map. Element-wise multiplication is performed on the expanded region focus attention map and the general feature map, and the general feature map is reweighted to obtain an enhanced feature map.

[0094] Expand the dimensions of the region geometric attention map and the region material attention map to the same dimension as the augmented feature map. Based on the expanded region geometric attention map and the region material attention map, calculate the geometrically weighted augmented feature map and the material-weighted augmented feature map.

[0095] The geometrically weighted enhanced feature map is input into n1 convolutional layers, and geometric features are obtained through feature extraction.

[0096] The material-weighted enhanced feature map is input into n² convolutional layers, and the material features are obtained through feature extraction.

[0097] Geometric and material features are concatenated along the channel dimension to obtain a fused feature map. A single... The convolutional layers perform fusion convolution on the fused feature maps to obtain path features that associate geometric and material information from different regions.

[0098] Step S3014: Input the path features into the pre-trained decoder to obtain the predicted segmentation mask set. Based on the predicted segmentation mask set and the initial segmentation dataset, calculate the total task loss value.

[0099] In this embodiment, the difference between the region mask set in the initial segmentation dataset and the predicted segmentation mask set is calculated using the cross-entropy loss function to obtain the main loss function value.

[0100] Extract the boundary of each instance in the region type label, process the region type label using the Laplacian operator to generate a boundary map, downsample the boundary map to the same resolution as the region geometric attention map, and normalize it to obtain the geometric pseudo-label.

[0101] Set a binary mask with the same shape as the region type label, and initialize the binary mask. In the initialized binary mask, set the attention value of the internal region of all foreground objects to 1, and set the attention value of the background and boundary to 0. The internal region is the instance region minus the boundary region.

[0102] The binary mask is downsampled to the same resolution as the region material attention map and then normalized to obtain the material pseudo-label.

[0103] Calculate the mean squared error loss of the region's geometric attention map and geometric pseudo-labels;

[0104] Calculate the mean absolute error loss of the region material attention map and material pseudo-labels;

[0105] The total loss for regional path supervision is obtained by summing the mean squared error loss and the mean absolute error loss.

[0106] The total task loss value is obtained by weighting and summing the main loss function value and the total loss of the regional path supervision according to a preset weight. The preset weight is set by those skilled in the art according to the actual situation.

[0107] Step S3015: Set a differentiated learning rate strategy and calculate the gradient of the total task loss value relative to all parameters in the second training model.

[0108] In this embodiment, an existing differentiated learning rate strategy is adopted to assign different learning rates to different parameters in the second training model. Based on the learning rate, the gradient of the total task loss value relative to all parameters in the second training model is calculated using the backpropagation algorithm.

[0109] Step S3016: Based on the differentiated learning rate strategy and gradient, iteratively update all parameters in the second training model until the parameters after iteration meet the preset iteration termination condition, and the iteration is completed, resulting in the adjusted second training model.

[0110] In this embodiment, the iteration termination condition is set by those skilled in the art based on the actual situation.

[0111] Step S3017: Based on the adjusted second training model and the second inference model, obtain the adjusted second visual large model through model ensemble.

[0112] Step S302: Input the sample information of each sample in the second sample set into the adjusted second visual large model to obtain the pre-labeled dataset.

[0113] The specific steps of step S302 are as follows:

[0114] Step S3021: Input the sample information of each sample in the second sample set into the adjusted second training model to obtain the first prediction segmentation mask set and the first attention map.

[0115] In this embodiment, the first attention map includes: a first region focus attention map, a first region geometry attention map, and a first region material attention map.

[0116] Step S3022: Construct a region adjacency graph based on the first predicted segmentation mask set. In the region adjacency graph, nodes represent segmentation mask regions, and edges represent adjacent region relationships.

[0117] In this embodiment, the first predicted segmentation mask set is composed of the segmentation mask sets of each sample in the second sample set. Each segmentation mask in the segmentation mask set of each sample is parsed, and the geometric properties of each segmentation mask are calculated. The geometric properties include: area, perimeter, minimum bounding rectangle, and width, etc.

[0118] Based on the position of each segmentation mask in the corresponding RGB image, it is determined whether any two regions are adjacent. Adjacent means that the pixel distance between the boundaries of the two regions is less than a preset pixel threshold. The pixel threshold is set by those skilled in the art according to the actual situation. Based on the adjacency determination results, a region adjacency map of each sample is constructed.

[0119] Step S3023: Set up a rule base, calculate the perception function value of each edge in the region adjacency graph, and construct a rule perception function based on the perception function value.

[0120] The specific steps of step S3023 are as follows:

[0121] Step S30231: Calculate the cost value assigned to each region type label to obtain the data item.

[0122] In this embodiment, the regional features of each region are extracted, and the similarity between the regional features of each region and each region type label is calculated. The specific formula for the data item is as follows:

[0123] ;

[0124] Where D represents a data item, and simA represents the similarity between the regional features of each region and the type label of each region.

[0125] Step S30232: Set up a rule base, calculate the geometric data of the merged region after merging each pair of adjacent regions, and evaluate the geometric data based on the rule base.

[0126] In this embodiment, the rule base is a structured list containing all relevant geometric design rules. These geometric design rules are obtained based on design specifications for the semiconductor industry. The rule base includes: width threshold, spacing threshold, extension threshold, etc.

[0127] Simulate merging any two adjacent regions to generate a mask for the merged new region. Calculate the geometric data of the new region based on the mask. The geometric data includes: minimum bounding rectangle, region area, minimum width, and minimum spacing, etc.

[0128] The geometric data is compared one by one with the thresholds in the rule base. The geometric data is evaluated according to the constraints of each geometric design rule. When all the geometric data meet the constraints of the corresponding rule base, the merging of any two adjacent regions is allowed from a geometric perspective.

[0129] Step S30233: Set a material similarity threshold, extract the material feature confidence of each pair of adjacent regions, and evaluate the material similarity of each pair of adjacent regions based on the material similarity threshold.

[0130] In this embodiment, the material similarity threshold is set by those skilled in the art according to the actual situation. The pixel coordinates of each pair of regions are obtained based on the segmentation mask set of each sample. The average value of the pixel values ​​of each pair of regions is calculated in each channel of the material attention map of each sample. The average value of each channel is combined into a vector as the material feature vector of each pair of regions.

[0131] Calculate the similarity between the material feature vectors of each pair of regions. If the similarity is greater than or equal to the material similarity threshold, the pair of regions is allowed to be merged from the perspective of material.

[0132] Step S30234: Based on the evaluation results of geometric data and material similarity, calculate the perceptual function value of each edge in the region adjacency graph, and design a perceptual smoothing term based on the perceptual function value.

[0133] In this embodiment, when any two adjacent regions are allowed to be merged from both a geometric and material perspective, the perception function value of the edge representing the regional relationship between the two adjacent regions is set to 1; otherwise, it is set to 0.

[0134] The specific formula for the smoothing term in the existing technology is as follows:

[0135] ;

[0136] Where * represents the multiplication sign, and Multiply, This represents the smoothing term in the prior art, where i and j represent any two adjacent regions. The representative weighting coefficients were obtained experimentally by those skilled in the art. The region type label representing region i. The region type label represents region j. This represents an exponential function. Its value is 1 when the region label types of region i and region j are different, and 0 otherwise.

[0137] Based on the perceptual function value and the smoothing term in existing technologies, a perceptual smoothing term is constructed, with the specific formula as follows:

[0138] ;

[0139] in, This represents the perceptual smoothing term, and F is the value of the perceptual function.

[0140] Step S30235: Accumulate the data items and the perceptual smoothing items to construct a rule-based perceptual function.

[0141] In this embodiment, the specific formula for the rule-aware function is:

[0142] ;

[0143] in, This represents the rule-aware function, where D represents the data item. This represents the summation of data items across all regions. This represents the summation of the values ​​of the perceptual smoothing terms represented by all edges in the adjacent region graph.

[0144] Step S3024: Perform global optimization on the region adjacency graph using a rule-aware function to obtain optimized region type labels.

[0145] In this embodiment, the rule-aware function is used as the objective function, through... The extended algorithm seeks a label allocation scheme that minimizes the value of the rule-aware function, resulting in optimized region type labels.

[0146] Step S3025: Based on the optimized region type labels, pre-label the regions and region types corresponding to each segmentation mask in the first predicted segmentation mask set to construct a pre-labeled dataset.

[0147] In this embodiment, the optimized region type labels are mapped back to the pixel-level segmentation map. Based on the first predicted segmentation mask set, the regions and region types corresponding to each segmentation mask are pre-labeled to construct a pre-labeled dataset. The pixel-level segmentation map is obtained by segmenting the image according to the segmentation mask set of each sample in the first predicted segmentation mask set. Each pixel inherits the label of its region.

[0148] Step S303: Manually correct the pre-labeled dataset to obtain a segmentation correction dataset, and merge the segmentation correction dataset and the initial segmentation dataset to obtain a fine-tuned dataset.

[0149] In this embodiment, the annotator reviews the automatic annotation results obtained from the pre-annotated dataset on the image one by one, identifies and corrects the errors in the automatic annotation results, and obtains a segmentation correction dataset. The errors include: label errors, geometric errors, omission errors and duplication errors.

[0150] Step S4: Train the second vision model based on the fine-tuning dataset, collect a real industrial image set, input the real industrial image set into the trained second vision model, and obtain a real automatic semantic segmentation annotation dataset.

[0151] In this embodiment, based on the fine-tuning dataset, the second visual large model is deep-trained in the same way as in step S301 to obtain the deep-trained second visual large model. After collecting a set of real industrial images, the real industrial images are input into the deep-trained second visual large model. Automatic semantic segmentation and annotation of the real industrial images are performed in the same way as in step S302. It should be noted that the images need to be preprocessed after collecting the real industrial images. The preprocessing specifically includes: outlier handling, data standardization, missing value imputation, time alignment, and feature smoothing.

[0152] Example 2

[0153] Please see Figure 4 The present invention provides an embodiment of an automatic semantic segmentation and annotation system for industrial scenarios, comprising: a sample acquisition module, a manual annotation module, a fine-tuning and correction module, and an automatic annotation module.

[0154] The sample acquisition module is used to acquire an image sample set of industrial scene, divide the image sample set into a first sample set and a second sample set, and extract the sample information of each sample in the image sample set. The sample information includes RGB image and depth map.

[0155] The manual annotation module is used to input the RGB images of each sample in the first sample set into the first visual large model, and manually annotate the output of the visual large model based on the depth map to obtain the initial segmentation dataset.

[0156] The fine-tuning and correction module includes: a model fine-tuning unit, a pre-labeling unit, and a data merging unit.

[0157] The model fine-tuning unit is used to iterate on the second training model based on the sample information of each sample in the first sample set and the segmented initial dataset until the parameters of the second training model after iteration meet the preset iteration termination condition. When the iteration is completed, the adjusted second training model and the second inference model are integrated to obtain the adjusted second visual large model.

[0158] The pre-labeling unit is used to input the sample information of each sample in the second sample set into the adjusted second visual large model to obtain the pre-labeled dataset.

[0159] The data merging unit is used to manually correct the pre-labeled dataset to obtain a segmentation correction dataset, and to merge the segmentation correction dataset and the initial segmentation dataset to obtain a fine-tuned dataset.

[0160] The automatic annotation module is used to train the second vision model based on the fine-tuning dataset. It collects a set of real industrial images and inputs the set of real industrial images into the trained second vision model for automatic semantic segmentation and annotation.

[0161] In addition, the parts of the technical solutions provided in the embodiments of this application that are consistent with the implementation principles of the corresponding technical solutions in the prior art have not been described in detail, so as to avoid excessive elaboration.

[0162] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the invention. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An automatic semantic segmentation and annotation method for industrial scenarios, characterized in that, include: Collect an image sample set of an industrial scene, divide the image sample set into a first sample set and a second sample set, and extract the sample information of each sample in the image sample set. The sample information includes RGB images and depth maps. The RGB images of each sample in the first sample set are input into the first large visual model. The output of the large visual model is manually labeled based on the depth map to obtain the initial segmentation dataset. The second visual large model is adjusted based on the initial segmentation dataset. The second sample is input into the adjusted second visual large model, and the pre-labeled dataset output by the adjusted second visual large model is manually corrected to obtain the fine-tuned dataset. The adjustment is to iteratively update the second training model in the second visual large model by generating attention maps and feature weighted fusion. The pre-labeled dataset is obtained by applying a rule-aware function to optimize the labeling of the region adjacency graph constructed based on the output of the second training model. The second vision model is trained based on the fine-tuned dataset. A real industrial image set is collected and input into the trained second vision model to obtain a real automatic semantic segmentation and annotation dataset. The process involves adjusting the second visual model based on the initial segmentation dataset, inputting the second sample into the adjusted second visual model, and manually correcting the pre-labeled dataset output by the adjusted second visual model to obtain a fine-tuned dataset, including: Based on the sample information of each sample in the first sample set and the segmentation of the initial dataset, the second training model is iterated until the parameters of the second training model after iteration meet the preset iteration termination condition. The iteration is then completed, and the adjusted second training model and the second inference model are integrated to obtain the adjusted second visual large model. The sample information of each sample in the second sample set is input into the adjusted second visual large model to obtain the pre-labeled dataset; The pre-labeled dataset is manually corrected to obtain a segmentation correction dataset. The segmentation correction dataset and the initial segmentation dataset are then merged to obtain a fine-tuned dataset. The process involves inputting the sample information of each sample in the second sample set into the adjusted second visual large model to obtain a pre-labeled dataset, including: The sample information of each sample in the second sample set is input into the adjusted second training model to obtain the first prediction segmentation mask set and the first attention map; A region adjacency graph is constructed based on the first predicted segmentation mask set, wherein nodes in the region adjacency graph represent segmentation mask regions and edges represent adjacent region relationships. Set up a rule base, calculate the perception function value of each edge in the region adjacency graph, and construct a rule perception function based on the perception function value; The region adjacency graph is globally optimized using a rule-aware function to obtain optimized region type labels. Based on the optimized region type labels, the regions and region types corresponding to each segmentation mask in the first predicted segmentation mask set are pre-labeled to construct a pre-labeled dataset. The perceptual function value of each edge in the adjacency graph of the computational region is used to construct a rule-based perceptual function, including: Calculate the cost of assigning each region type label to each region to obtain the data item; Calculate the geometric data of the merged region after merging each pair of adjacent regions, and evaluate the geometric data based on the rule base; Set a material similarity threshold, extract the material feature confidence of each pair of adjacent regions, and evaluate the material similarity of each pair of adjacent regions based on the material similarity threshold; Based on the evaluation results of geometric data and material similarity, the perceptual function value of each edge in the region adjacency graph is calculated, and a perceptual smoothing term is designed based on the perceptual function value. The data items and the perceptual smoothing items are summed to construct a rule-based perceptual function.

2. The automatic semantic segmentation and annotation method for industrial scenes according to claim 1, characterized in that, The first step involves inputting the RGB images of each sample in the first sample set into the first large-scale visual model, and manually annotating the output of the large-scale visual model based on the depth map to obtain the initial segmentation dataset, including: Input the RGB images of each sample in the first sample set into the first visual large model, and obtain the initial mask set of the RGB images of each sample through interactive prompts; Based on depth maps and RGB images, semantic segmentation is performed on each initial mask in the initial mask set through manual annotation, and the segmentation initial dataset is constructed based on the results of semantic segmentation.

3. The automatic semantic segmentation and annotation method for industrial scenes according to claim 2, characterized in that, The process involves semantic segmentation of each initial mask in the initial mask set based on depth maps and RGB images through manual annotation. An initial segmentation dataset is then constructed based on the semantic segmentation results, including: Set a height threshold, calculate the height feature value of the region corresponding to each initial mask in the initial mask set based on the depth map of each sample in the first sample set, and compare the height feature value with the height threshold. When the height feature value is greater than the height threshold, the boundary of the current initial mask is corrected by manual annotation based on the depth map and RGB image, and the region type is determined. The region of the current initial mask and the corresponding region type label are recorded to obtain the first region mask set and the first region type label set. When the height feature value is less than or equal to the height threshold, the boundary of the current initial mask is corrected by manual annotation based on the RGB image, and the region type is determined. The region of the current initial mask and the corresponding region type label are recorded to obtain the second region mask set and the second region type label set. An initial segmentation dataset is constructed based on a set of region masks and a set of region type labels. The set of region masks includes a first set of region masks and a second set of region masks. The set of region type labels includes a first set of region type labels and a second set of region type labels.

4. The automatic semantic segmentation and annotation method for industrial scenarios according to claim 3, characterized in that, The second large-scale visual model includes: The second training model and the second inference model are integrated to obtain the second large visual model.

5. The automatic semantic segmentation and annotation method for industrial scenarios according to claim 4, characterized in that, Based on the sample information of each sample in the first sample set and the segmented initial dataset, the second training model is iterated until the parameters of the iterated second training model meet the preset iteration termination condition. The iteration is then complete. The adjusted second training model and the second inference model are integrated to obtain the adjusted second visual large model, including: The sample information of each sample in the first sample set is input into the second training model, and the general feature map of each sample in the first sample set is obtained through feature extraction. A query vector is defined, and the query vector is interacted with a general feature map to generate an attention map, which includes: a region focus attention map, a region geometry attention map, and a region material attention map; Based on the region focus attention map, region geometric attention map, and region material attention map, spatial weighted feature extraction and feature fusion are performed on the general feature map to obtain path features that associate geometric and material information of different regions. The path features are input into the pre-trained decoder to obtain a predicted segmentation mask set. Based on the predicted segmentation mask set and the initial segmentation dataset, the total task loss value is calculated. Set a differentiated learning rate strategy and calculate the gradient of the total task loss with respect to all parameters in the second training model; Based on the differentiated learning rate strategy and gradient, all parameters in the second training model are iteratively updated until the parameters after iteration meet the preset iteration termination condition, at which point the iteration is completed and the adjusted second training model is obtained. Based on the adjusted second training model and second inference model, the adjusted second visual large model is obtained through model ensemble.

6. An automatic semantic segmentation and annotation system for industrial scenes, used to implement the automatic semantic segmentation and annotation method for industrial scenes according to any one of claims 1-5, characterized in that, include: The sample acquisition module is used to acquire image sample sets of industrial scenes, divide the image sample sets into a first sample set and a second sample set, and extract sample information of each sample in the image sample set. The sample information includes RGB images and depth maps. The manual annotation module is used to input the RGB images of each sample in the first sample set into the first visual large model, and manually annotate the output of the visual large model based on the depth map to obtain the initial segmentation dataset; The fine-tuning and correction module is used to adjust the second visual large model based on the initial segmentation dataset. The second sample is input into the adjusted second visual large model, and the pre-labeled dataset output by the adjusted second visual large model is manually corrected to obtain the fine-tuned dataset. The automatic annotation module is used to train the second vision large model based on the fine-tuning dataset. It collects a set of real industrial images and inputs the real industrial images into the trained second vision large model to obtain a real automatic semantic segmentation annotation dataset.

7. The automatic semantic segmentation and annotation system for industrial scenes according to claim 6, characterized in that, The fine-tuning and correction module includes: The model fine-tuning unit is used to iterate on the second training model based on the sample information of each sample in the first sample set and the segmentation of the initial dataset until the parameters of the second training model after iteration meet the preset iteration termination condition. When the iteration is completed, the adjusted second training model and the second inference model are integrated to obtain the adjusted second visual large model. The pre-labeling unit is used to input the sample information of each sample in the second sample set into the adjusted second visual large model to obtain the pre-labeled dataset; The data merging unit is used to manually correct the pre-labeled dataset to obtain a segmentation correction dataset, and to merge the segmentation correction dataset and the initial segmentation dataset to obtain a fine-tuned dataset.

Images