Cross-domain target detection method and device based on sample recombination and style correction

Abstract

The application discloses a cross-domain target detection method and device based on sample recombination and style correction, and belongs to the field of machine vision. The method is used to solve the problem that the prior art cannot simultaneously consider sample expansion, artifact suppression and cross-domain feature correction, and the detection precision is insufficient in the case that target domain labeling is scarce. The method uses a source domain image set to train and obtain source domain pre-training parameters, and repairs the source domain image to obtain a purified source background image. A real target image block to be detected is extracted from an original target domain image, and is implanted into the background image. After local photometric alignment and continuous soft mask fusion, a cross-domain synthetic sample is generated. A mixed fine-tuning data set is constructed by combining the real target domain sample obtained through semantic preservation enhancement, and the detection network is initialized in batches. A preset feature layer is selected, a style correction loss is constructed according to the difference between the batch-level normalized Gram matrices of the two types of samples, and the parameters are updated in combination with the detection loss. After iterative convergence, the final network is obtained, and the target domain image to be detected is input into the final network, and the category and position of the target to be detected are output.

Description

Technical Field

[0001] This invention relates to the field of industrial machine vision image inspection technology, and in particular to a cross-domain target detection method and apparatus based on sample recombination and style correction for defect inspection of workpieces in the manufacturing industry. Background Technology

[0002] Industrial target detection enables the automatic localization and classification of targets within images of industrial products such as welds, metal surfaces, composite materials, and electronic devices. Deep learning-based target detection technology can effectively improve detection efficiency and result consistency; however, this technology heavily relies on a training dataset that is sufficiently large, accurately labeled, and consistent with the data distribution of the actual deployment environment. In real-world industrial scenarios such as small-batch production, new product introduction, equipment replacement, or process switching, the target production line can only acquire a very small number of labeled target domain images. Furthermore, there are differences in data distribution between the source and target production lines in terms of material texture, lighting conditions, imaging devices, background noise, and the appearance of the target object. Directly deploying detectors trained on the source domain, or relying solely on a small number of target domain samples for fine-tuning, makes it difficult to adapt to the target domain's data distribution. This results in low accuracy in cross-domain few-shot target detection, leading to missed detections, false detections, and model overfitting. Existing few-shot detection technologies mainly rely on conventional image enhancement, copy-paste, generative models, meta-learning, and metric learning to expand the training data distribution. When directly copying and pasting real target images into other images, it's easy to retain the original target in the original image, leading to target overlap, semantic conflicts, and missing annotations. Inconsistencies in imaging style, brightness, and texture between different images result in hard boundaries and stitching artifacts after pasting, causing the network to incorrectly learn these synthetic artifacts. Most of these methods focus only on increasing the number of samples or improving category generalization ability, making it difficult to simultaneously reuse the real target morphology in the target domain and the rich background context information in the source domain when target domain annotations are scarce, resulting in limited detection improvement.

[0003] Another type of domain-adaptive object detection method uses adversarial learning, pseudo-labels, teacher-student networks, and feature alignment to reduce inter-domain differences. However, when there are very few labeled samples in the target domain, the data distribution of the target domain cannot be stably estimated in relation to the pseudo-labels. Furthermore, performing excessively strong image-level or instance-level domain alignment operations can destroy key localization features such as the edges and shapes of industrial targets, thereby impairing detection performance.

[0004] In actual manufacturing production lines for workpiece defect inspection, the number of labeled workpiece defect images available after line updates and process changes is scarce. Directly using existing algorithms can lead to missed or false detections of defects such as welds, scratches, and cracks, making it difficult to meet the actual production needs of automated quality inspection. The resulting technical problem is that in low-sample, cross-domain industrial inspection scenarios, the effective information in the source and target domains is asymmetrical. A small number of target domain samples possess realistic target morphology, but their background and imaging conditions have limited coverage. Source domain samples have rich background context, but the original target appearance, background style, and target domain are inconsistent. Current technology lacks a solution that, under conditions of scarce target domain annotations, can fully utilize the realistic target morphology and source domain background context to suppress synthetic artifacts caused by sample recombination, while simultaneously correcting source background style deviations introduced by synthetic samples, thereby improving the accuracy of cross-domain, low-sample target detection.

[0005] In summary, existing small-sample, domain-adaptive detection methods struggle to simultaneously address effective sample expansion, synthetic artifact suppression, and cross-domain feature bias correction, making them ill-suited for cross-domain target detection tasks with scarce annotations in industrial settings. Summary of the Invention

[0006] This invention provides a cross-domain target detection method and apparatus based on sample recombination and style correction. It is used to solve the problem in the manufacturing workpiece defect quality inspection scenario where the number of labeled workpiece defect images that can be obtained from a new production line is scarce, and the imaging styles of workpiece images collected from different production lines are significantly different. Existing detection schemes are difficult to take into account training sample expansion, synthesis artifact suppression, and cross-domain feature deviation correction, resulting in missed or false detection of workpiece defects, and the detection accuracy is difficult to meet the requirements of industrial automated quality inspection production.

[0007] This invention provides a cross-domain target detection method based on sample recombination and style correction, including:

[0008] A source domain image set with a first target annotation and a target domain image set with a first target annotation are obtained. The target detection network is trained using the source domain image set with the first target annotation to obtain source domain pre-training parameters. A source domain target mask is generated based on the first target annotation of the source domain images. Image restoration is performed on the area covered by the source domain target mask to obtain a purified source background image.

[0009] Based on the first target annotation of the original target domain image, extract the real target patch, embed the real target patch into the background image of the purification source and determine the effective placement area, perform geometric constraints, local photometric alignment and continuous soft mask fusion on the real target patch, and generate the second target annotation based on the fused placement position, transformation size and target category to obtain the cross-domain synthetic sample.

[0010] The original target domain image is semantically preserved and enhanced to obtain enhanced real target domain samples. The enhanced real target domain samples and cross-domain synthetic samples are combined to form a hybrid fine-tuning dataset and divided into hybrid fine-tuning batches. The target detection network is initialized using source domain pre-trained parameters.

[0011] Before training iteration, one or more fixed network layers are selected from the shallow to middle layers of the backbone network of the initialized target detection network as preset feature layers, and all the preset feature layers are summarized to form a preset feature layer set.

[0012] During the iteration process, the mixed fine-tuning batch is input into the initialized target detection network. For the real target domain sample subset and the cross-domain synthetic sample subset within the mixed fine-tuning batch, the batch-level normalized Gram matrix is ​​calculated on the original feature map corresponding to the preset feature layer. The style correction loss is constructed based on the difference between the batch-level normalized Gram matrices of the two types of sample subsets.

[0013] The style correction loss is combined with the target detection loss as a weak regularization term to obtain the joint total loss. The trainable parameters of the target detection network are updated by backpropagation based on the joint total loss. The trainable parameters include the unfrozen parameters of the backbone network, the neck network parameters, and the detection head parameters.

[0014] When the iteration stopping condition is met, the final target detection network with converged and updated trainable parameters is obtained. The target domain image to be detected in the real scene is input into the final target detection network, and the target category and target location corresponding to the target domain image are output.

[0015] This invention provides a cross-domain target detection device based on sample recombination and style correction, comprising:

[0016] The first obtaining unit is used to acquire a source domain image set with a first target annotation and a target domain image set with a first target annotation, train a target detection network using the source domain image set with the first target annotation to obtain source domain pre-training parameters; generate a source domain target mask based on the first target annotation of the source domain image, and perform image restoration on the area covered by the source domain target mask to obtain a purified source background image.

[0017] The second obtaining unit is used to extract real target patches based on the first target label of the original target domain image, implant the real target patches into the background image of the purification source and determine the effective placement area, perform geometric constraints, local photometric alignment and continuous soft mask fusion on the real target patches, and generate a second target label based on the fused placement position, transformation size and target category to obtain a cross-domain synthetic sample.

[0018] An initialization unit is used to perform semantic preservation enhancement on the original target domain image to obtain enhanced real target domain samples. The enhanced real target domain samples and cross-domain synthetic samples are combined to form a hybrid fine-tuning dataset and divided into hybrid fine-tuning batches. The target detection network is initialized using source domain pre-trained parameters.

[0019] The selection unit is used to select one or more fixed network layers from the shallow to middle layers of the backbone network of the initialized target detection network as preset feature layers before the training iteration. All the preset feature layers are summarized to form a preset feature layer set.

[0020] The construction unit is used to input the mixed fine-tuning batch into the initialized target detection network during the iteration process, calculate the batch-level normalized Gram matrix on the original feature map corresponding to the preset feature layer for the real target domain sample subset and the cross-domain synthetic sample subset within the mixed fine-tuning batch, respectively, and construct the style correction loss based on the difference between the batch-level normalized Gram matrices of the two types of sample subsets.

[0021] The update unit is used to combine the style correction loss as a weak regularization term with the target detection loss to obtain a joint total loss, and to update the trainable parameters of the target detection network by backpropagation based on the joint total loss. The trainable parameters include the unfrozen parameters of the backbone network, the neck network parameters, and the detection head parameters.

[0022] The output unit is used to obtain the final target detection network with converged update of trainable parameters when the iteration stopping condition is met. The target domain image to be detected in the real scene is input into the final target detection network, and the target category and target location corresponding to the target domain image are output.

[0023] This invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the cross-domain target detection method based on sample recombination and style correction described in any of the above embodiments.

[0024] This invention provides a computer device, which includes a memory and a processor. The memory stores a computer program, and when the computer program is executed by the processor, the processor performs the cross-domain target detection method based on sample recombination and style correction described above.

[0025] Compared with the prior art, the embodiments of the present invention have at least the following beneficial effects:

[0026] (1) In this embodiment of the invention, the real target image is embedded into the cleaned source background image where the original target has been removed. Under the condition of a small number of target domain samples with the first target annotation, the shape of the target is completely preserved in accordance with the actual target domain scene. At the same time, a single target can be adapted to more source domain backgrounds, textures and lighting conditions, which greatly expands the effective sample size of the hybrid fine-tuning dataset that can be used for target domain fine-tuning, and significantly improves the industrial target detection accuracy in cross-domain low-sample scenarios. Meanwhile, the fusion process can automatically generate matching second target annotations according to the fusion position and scaling size of the target, effectively avoiding the problems of target overlap and annotation conflict, without the need for additional manual annotation, and reducing the sample preparation cost of low-sample cross-domain detection.

[0027] (2) The embodiments of the present invention constrain the placement position of the real target image patch by effectively placing the region, and combine the local photometric alignment and continuous soft mask matrix fusion strategy to set differentiated weight transitions for the center and edge regions of the target image patch, effectively eliminating brightness abrupt changes, hard seams and synthesis traces caused by cross-domain patch fusion, greatly improving the visual consistency between the real target image patch and the clean source background image, ensuring the training effectiveness of cross-domain synthesized samples, and further optimizing the target detection accuracy and robustness in cross-domain scenarios with few samples.

[0028] (3) In the target domain fine-tuning stage of this embodiment, a set of preset feature layers is constructed by selecting fixed network layers from the shallow to the middle layers of the backbone network before training iteration, and the feature extraction position of style statistics is fixed. During the iterative training process, the original feature map of the preset feature layer of the network is extracted based on the cross-domain style correction algorithm flow. The batch-level normalized Gram matrix second-order feature statistics difference between the cross-domain synthetic sample subset and the real target domain sample subset in the mixed fine-tuning batch is compared and calculated. The residual source background style deviation introduced by the sample recombination process is corrected by style correction loss, and the feature distribution gap between the two types of training samples is reduced, so that the target detection network can quickly adapt to the target domain data distribution characteristics and stably improve the cross-domain target detection performance with few samples. At the same time, the style correction constraint is only used as a weak regularization method during the training period. It does not require the two types of samples to correspond one-to-one in terms of target category, position and spatial layout. Moreover, the style correction branch is stopped after the iteration stopping condition is reached. It will not change the final target detection network structure in the deployment stage, and there is no additional inference overhead. It takes into account both detection accuracy and deployment efficiency. The embodiments of the present invention improve the detection capability of defects in industrial workpieces such as welds, scratches, and cracks. It can adapt to the automated defect quality inspection needs after changes in production line processes and imaging conditions. It can be directly applied to the automatic identification of workpiece defects in industrial production lines, reducing the workload of manual quality inspection. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 A schematic diagram of the cross-domain target detection method based on sample recombination and style correction provided in an embodiment of the present invention;

[0031] Figure 2 A schematic diagram of the overall process of the cross-domain target detection method based on sample recombination and style correction provided in an embodiment of the present invention;

[0032] Figure 3 This is a schematic diagram of the cross-domain target detection device based on sample recombination and style correction provided in an embodiment of the present invention. Detailed Implementation

[0033] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0034] This invention provides a cross-domain target detection method based on sample recombination and style correction, applicable to machine vision inspection engineering scenarios where labeled samples in the target domain are scarce and there are significant differences in imaging styles between the source and target domains. The method receives images of real-world physical scenes captured by industrial cameras, trains a target detection network adapted to the deployment site, inputs images captured at the deployment site as the target domain images to be inspected into the trained target detection network, and outputs the target category and location, supporting practical business processes such as quality inspection and defective product rejection.

[0035] Specifically, this method is applied to the visual inspection of defects in industrial workpieces: industrial cameras capture images of workpieces on the production line; a large number of labeled workpiece images accumulated from historical production lines serve as the source domain image set with the first target to be inspected labeled; when the production line station, lighting source, or camera model is changed, the imaging domain shifts in the new production line, resulting in fewer manually annotated workpiece defect images, thus forming the target domain image set with the first target to be inspected labeled. This method is used to fine-tune the target domain training of the target detection network; after training, industrial cameras capture real-time images of workpieces from the new production line as target domain images to be inspected, which are then input into the trained target detection network. The network automatically outputs the target category and location corresponding to defects such as scratches, bumps, and defects, providing information for the production line to perform defect sorting, alarms, and defect rejection.

[0036] Figure 1 This is a schematic diagram of the cross-domain target detection method based on sample recombination and style correction provided in an embodiment of the present invention. Figure 2 The following is a detailed schematic diagram of the method architecture provided by the present invention, in conjunction with... Figure 1 and Figure 2 The method provided in the embodiments of the present invention will be explained in detail, such as... Figure 1 As shown, the method includes the following steps:

[0037] Step 101: Obtain a source domain image set with the first target annotation and a target domain image set with the first target annotation; train a target detection network using the source domain image set with the first target annotation to obtain source domain pre-training parameters; generate a source domain target mask based on the first target annotation of the source domain images; perform image restoration on the area covered by the source domain target mask to obtain a purified source background image.

[0038] Step 102: Extract real target patches based on the first target label of the original target domain image, embed the real target patches into the background image of the purification source and determine the effective placement area, perform geometric constraints, local photometric alignment and continuous soft mask fusion on the real target patches, and generate a second target label based on the fused placement position, transformation size and target category to obtain a cross-domain synthetic sample.

[0039] Step 103: Perform semantic preservation enhancement on the original target domain image to obtain enhanced real target domain samples. Combine the enhanced real target domain samples with cross-domain synthetic samples to form a hybrid fine-tuning dataset and divide it into hybrid fine-tuning batches. Initialize the target detection network using source domain pre-trained parameters.

[0040] Step 104: Before the training iteration, select one or more fixed network layers from the shallow to middle layers of the backbone network of the initialized target detection network as preset feature layers, and summarize all the preset feature layers to form a preset feature layer set.

[0041] Step 105: During the iteration process, the mixed fine-tuning batch is input into the initialized target detection network. For the real target domain sample subset and the cross-domain synthetic sample subset within the mixed fine-tuning batch, the batch-level normalized Gram matrix is ​​calculated on the original feature map corresponding to the preset feature layer. The style correction loss is constructed based on the difference between the batch-level normalized Gram matrices of the two types of sample subsets.

[0042] Step 106: Combine the style correction loss as a weak regularization term with the target detection loss to obtain a joint total loss. Update the trainable parameters of the target detection network by backpropagation based on the joint total loss. The trainable parameters include the unfrozen parameters of the backbone network, the neck network parameters, and the detection head parameters.

[0043] Step 107: When the iteration stopping condition is met, the final target detection network with converged and updated trainable parameters is obtained. The target domain image to be detected in the real scene is input into the final target detection network, and the target category and target location corresponding to the target domain image to be detected are output.

[0044] The overall process of the cross-domain target detection method based on sample recombination and style correction of the present invention will be described below with reference to Figure 2.

[0045] like Figure 2 As shown, the method of the present invention includes a cross-domain synthetic sample generation branch and a network training and inference branch.

[0046] In the cross-domain synthetic sample generation branch: A target domain image set with the first target object annotation is acquired, and real target object patches are extracted based on the first target object annotation in the original target domain image; simultaneously, a source domain image set with the first target object annotation is acquired to obtain a purified source background image. The extracted real target object patches are implanted into the effective placement area of ​​the purified source background image, and after local photometric alignment, continuous soft mask fusion is performed to obtain the cross-domain synthetic sample. The cross-domain synthetic sample and the enhanced real target domain sample together constitute a hybrid fine-tuning dataset.

[0047] In the network training and inference branch: Source domain image sets with the first target image annotation are used to obtain source domain pre-training parameters, resulting in an initialized target detection network. Hybrid fine-tuning batches are input into the target detection network, and feature extraction and feature statistics are performed sequentially. A style correction loss is constructed based on the batch-level normalized Gram matrix corresponding to the preset feature layers. The style correction loss and the target detection loss are combined to obtain the joint total loss, and target domain fine-tuning is performed based on the joint total loss. After reaching the iteration stopping condition, the final target detection network is obtained. Detection head prediction is performed on the target images in the target domain, outputting the target category and target location corresponding to the target images in the target domain.

[0048] The cross-domain target detection method based on sample recombination and style correction provided in this embodiment runs on a computer device, which includes a processor and a memory. The memory stores computer programs, various image datasets, and target detection network code. The processor runs the computer programs in the memory and executes the entire process of this scheme: dataset reading, clean source background image generation, cross-domain synthetic sample construction, batch partitioning for fine-tuning, target detection network initialization, determination of preset feature layer set, extraction of original feature maps during iterative training, batch-level normalized Gram matrix calculation, loss construction, and backpropagation to update the trainable parameters of the target detection network. After training, the style correction branch is disabled, and the trained target detection network is used to complete the inference detection of the target domain image, outputting the target category and target location.

[0049] In step 101, a source domain image set with the first target annotation and a target domain image set with the first target annotation are obtained.

[0050] First, the processor reads a set of source domain images labeled with the first target to be detected from memory. This set of source domain images with the first target to be detected consists of multiple source domain images and the first target to be detected label corresponding to each source domain image. The first target to be detected label is used to mark the category and location information of the target to be detected within the source domain image. The source domain image set with the first target to be detected label has a sufficient number of samples and a stable image imaging distribution, which is used to conduct basic detection capability training on the target detection network, enabling the network to learn the general texture and shape features of the target to be detected.

[0051] Then, the processor acquires a target domain image set with the first target annotation. This target domain image set with the first target annotation consists of a limited number of original target domain images and the corresponding first target annotations for each original target domain image. The data distribution of the original target domain images differs from that of the source domain images, manifested in differences in illumination, noise, shooting angle, and imaging style. Furthermore, the number of samples in this target domain image set with the first target annotation is small, insufficient to directly train the target detection network from scratch. Therefore, it is necessary to utilize the pre-training results of the source domain image set with the first target annotation and subsequent cross-domain synthetic sample augmentation methods to complete network adaptation.

[0052] It should be noted that the source domain image set with the first target annotation refers to the image set collected from the source production line that has sufficient samples with the first target annotation, which can be fully used for the basic pre-training of the target detection network. The target domain image set with the first target annotation is the image set collected from the actual deployment production line of the target, and the number of samples with the first target annotation that can be obtained is less than that of the source domain image set with the first target annotation. This difference corresponds to the real-world situation where, after the process switch of the manufacturing production line, it is difficult for the new production line to collect a large number of workpiece defect images with the first target annotation. The solution of this invention is not limited to a certain fixed absolute value of samples. Under typical conditions, the number of samples with the first target annotation available for each type of target to be inspected can be 1 to 5, and it can also be adapted to application scenarios with other sample numbers that have this difference.

[0053] Furthermore, the target detection network is trained using the source domain image set labeled with the first target to be detected, and the source domain pre-training parameters are obtained.

[0054] Specifically, an object detection network is constructed, which consists of three functional modules: a backbone network, a neck network, and a detection head. These three modules are connected sequentially to form a complete forward inference chain for detection.

[0055] Backbone network: Receives the input image, performs image feature extraction, and outputs the image feature map; the backbone network mainly extracts the low-level texture, edges, and high-level semantic features of the image.

[0056] Neck network: Receives image feature maps output by the backbone network, performs multi-scale feature fusion, and performs fusion, upsampling, or downsampling processing on features at different scales.

[0057] Detection head: Receives the fused features output by the neck network, performs category classification prediction and location regression prediction of the target to be detected, and outputs the category and location results of the target to be detected.

[0058] The processor inputs the source domain image set with the first target annotation into the constructed target detection network, trains the target detection network, and obtains the source domain pre-trained parameters after training.

[0059] Furthermore, a source domain target mask is generated based on the first target annotation of the source domain image, and image inpainting is performed on the area covered by the source domain target mask to obtain a purified source background image.

[0060] Specifically, a single source domain image from the source domain image set with the first target to be inspected is read, and a source domain target to be inspected mask is generated based on the first target to be inspected label corresponding to the source domain image; the source domain target to be inspected mask is a binary mask image, and the area in the mask that covers the original target to be inspected in the source domain is marked as the area to be repaired.

[0061] An image inpainting operation is performed on the image region in the source domain image that is marked as the region to be repaired by the source domain target mask to remove the original target to be inspected in the source domain image and obtain a purified source background image. In this embodiment of the invention, the purified source background image is the background image obtained after removing the original target to be inspected in the source domain image. It retains the texture, lighting and imaging context information of the source domain image itself, and the purified source background image no longer contains the original target to be inspected in the source domain.

[0062] In step 102, the true target image patch is extracted based on the first target annotation of the original target domain image. Specifically, based on the target domain image set with the first target annotation, for a single original target domain image, according to its inherent first target annotation, a local image region containing only the target is precisely cropped and separated to obtain the true target image patch. This patch can be determined using the following formula:

[0063] (1)

[0064] in, This represents the actual target image patch after local photometric alignment. Indicates the first A real target image patch extracted from the original target domain image. This function represents the average brightness of an image region. This indicates the effective placement area in the background image of the purification source. This represents a pixel truncation function that restricts the output pixels to the range of valid pixel values ​​in the image.

[0065] The real target image patch is taken from the original target domain image, which fully preserves the appearance, texture, shape and imaging features of the target under inspection in the real deployment scenario, without any extra background information interference from the source domain and target domain.

[0066] Within the entire clean source background image, select a legal pixel area that is centered, far from the image boundary, has complete texture, and is of sufficient size. This area is determined as the effective placement area of ​​the clean source background image, which can be used as the fusion placement position for the subsequent real target image blocks to be inspected.

[0067] Furthermore, the real target image to be inspected is embedded into the background image of the purification source and an effective placement area is determined. Geometric constraints, local photometric alignment and continuous soft mask fusion are performed on the real target image to be inspected. Based on the placement position, size transformation and target category after fusion, a second target label is generated to obtain a cross-domain synthetic sample.

[0068] First, the center of the actual target image is aligned with the center of the effective placement area to complete the initial target localization. If the original size of the actual target image is larger than the size that the effective placement area can accommodate, the actual target image is scaled proportionally to ensure that it can be completely and compliantly embedded within the effective placement area without pixel overflow or target truncation. Through these operations, geometric adaptation between the actual target image and the effective placement area of ​​the clean source background can be achieved, unifying the target scale and placement layout, and ensuring that the subsequent fused and synthesized sample layout is compliant and the imaging is natural.

[0069] Then, the average brightness of the real target image patch and the effective placement area inside the background image of the purification source are calculated separately after geometric constraints. The brightness difference between the real target image patch and the effective placement area is eliminated by photometric alignment. Then, all pixels are constrained to the legal pixel range by the pixel truncation function to avoid overexposure, underexposure and abnormal pixel out-of-bounds, so as to obtain the real target image patch after local photometric alignment.

[0070] Through the above operations, the inconsistency of cross-domain illumination between the real target image patch and the effective placement area inside the cleaned source background image can be eliminated, so that the imaging brightness of the real target image patch can be adapted to the imaging characteristics of the effective placement area, and the authenticity and effectiveness of cross-domain synthesized samples can be greatly improved.

[0071] In this embodiment of the invention, a Gaussian smoothing process is applied to a conventional binary hard mask to generate a continuous soft mask matrix with continuous values ​​and gradual transition. The matrix has the highest weight at the center and the weight decreases continuously towards the edge, which can achieve a seamless gradual transition between the target foreground and background and eliminate hard boundaries of splicing.

[0072] Furthermore, by using a successive soft mask matrix element-wise weighted fusion method, the real target image patch after local photometric alignment is fused with the effective placement area of ​​the clean source background image at the pixel level to generate a local composite region, which is specifically determined by the following formula:

[0073] (2)

[0074] in, This represents the composite region obtained by fusing the real target image patch with the background image of the clean source. This indicates that the size of the target image patch is the same as the actual target image patch, and the value range is... A continuous soft mask matrix is ​​used, and the fusion weights decrease continuously from the center of the patch to the edge. This represents the element-wise multiplication operation of matrices.

[0075] It should be noted that the synthesized area obtained here is only the local fused pixel block corresponding to the effective placement area, not the entire purified source background image.

[0076] The local composite region pixels are backfilled into the effective placement area of ​​the purified source background image, replacing the original background pixels, and finally a completely fused image is obtained.

[0077] Furthermore, a second target label is generated based on the final placement state of the actual target image patch in the effective placement area. Based on the fused image and the second target label, a cross-domain synthetic sample for target domain network fine-tuning is generated.

[0078] The final placement status here includes the final placement position of the actual target block to be inspected, its actual size after scaling, and the category of the target to be inspected; the second target annotation includes the target category, target center coordinates, target width, and target height.

[0079] In step 103, semantic preservation enhancement is performed on the original target domain image to obtain enhanced real target domain samples. The enhanced real target domain samples and cross-domain synthetic samples are combined to form a hybrid fine-tuning dataset and divided into hybrid fine-tuning batches.

[0080] Specifically, semantic preservation enhancement is performed on the original target domain images in the target domain image set with the first target to be detected annotation. Semantic preservation enhancement includes at least one of the operations of rotation and Gaussian blur. The enhancement process is completed by the semantic preservation enhancement operator to obtain the enhanced real target domain sample.

[0081] By merging all augmented real target domain samples and all cross-domain synthetic samples, a hybrid fine-tuning dataset is obtained, which can be determined by the following formula:

[0082] (3)

[0083] in, This indicates that a semantically preserving and enhancing operator is performed on the original target domain image. This represents the target domain image set labeled with the first target to be detected. This represents a sample set consisting of several cross-domain synthesized samples. This represents a hybrid fine-tuning dataset used for fine-tuning the target domain. This represents the union operation of sets.

[0084] It should be noted that the hybrid fine-tuning dataset contains two types of samples: one is the enhanced real target domain samples obtained through semantic preservation enhancement, which comes from real deployment scenarios; the other is the cross-domain synthetic samples generated by the algorithm in step 102, which are used to supplement the number of target domain samples.

[0085] The hybrid fine-tuning dataset is divided into groups to obtain multiple hybrid fine-tuning batches. In this embodiment of the invention, the sample set input to the network in a single iteration, in each hybrid fine-tuning batch, simultaneously contains at least one augmented real target domain sample and at least one cross-domain synthetic sample.

[0086] Furthermore, the source domain pre-trained parameters obtained in step 101 are used to initialize the parameters of the target detection network. As an optional operation, some shallow network parameters of the target detection network can be frozen. The frozen network parameters will not be updated in subsequent backpropagation iterations, and only the unfrozen network parameters will participate in gradient updates, thereby reducing the risk of overfitting under few sample conditions.

[0087] In step 104, before formally starting the iterative training for fine-tuning the target domain, the target detection network that has completed parameter initialization is screened: from the shallow to the middle layers of the backbone network of the target detection network, one or more fixed network layers are selected as preset feature layers, and all preset feature layers are summarized to form a preset feature layer set.

[0088] This operation is a static configuration before iteration, without inputting samples or performing forward operations; the output of the preset feature layer is the original feature map that has not yet been input into the neck network for multi-scale feature fusion, which is used to extract style-related features in subsequent iterations; the preset feature layer set remains fixed throughout the entire training iteration process, defining the range of network layers that will participate in the batch normalization Gram matrix and style correction loss calculation.

[0089] The above steps have completed network initialization, determined the preset feature layer set, and divided into multiple groups of mixed fine-tuning batches, and have officially entered the target domain fine-tuning iteration loop.

[0090] In step 105, during the iteration process, the mixed fine-tuning batch is input into the initialized target detection network. For the real target domain sample subset and the cross-domain synthetic sample subset within the mixed fine-tuning batch, the batch-level normalized Gram matrix is ​​calculated on the original feature map corresponding to the preset feature layer. The style correction loss is constructed based on the difference between the batch-level normalized Gram matrices of the two types of sample subsets.

[0091] Specifically, samples from the mixed fine-tuning batch are fed into the object detection network for forward inference. For each sample in the mixed fine-tuning batch, the backbone network performs feature extraction layer by layer; for each preset feature layer in the preset feature layer set, the original feature map output by that layer is extracted, and the original feature map output by that layer is flattened in spatial dimension to obtain a two-dimensional feature matrix. Each sample independently obtains a corresponding two-dimensional feature matrix.

[0092] In this embodiment of the invention, for each sample in a batch, the normalized Gram matrix of the sample in the preset feature layer is calculated using the two-dimensional feature matrix of the sample. The normalized Gram matrix is ​​used to characterize the second-order statistical correlation between the feature channels of the sample in the feature layer, and is used to quantify the style information of the image.

[0093] The normalized Gram matrix of a sample in this layer is determined by the following formula:

[0094] (4)

[0095] in, Indicates the layer number of the preset feature layer. This represents a single input sample. Indicates sample In the The two-dimensional feature matrix output by a preset feature layer, flattened in terms of spatial dimensions. This represents the matrix transpose operation. Indicates the first The number of channels in a preset feature layer Indicates the first The height of the output feature map of each preset feature layer Indicates the first The width of the output feature map of each preset feature layer. Indicates sample In the The normalized Gram matrix calculated on each preset feature layer represents the second-order statistical correlation between feature channels. .

[0096] The samples within the mixed fine-tuning batch are divided into a subset of real target domain samples and a subset of cross-domain synthetic samples. Within both the real target domain subset and the cross-domain synthetic sample subset, the arithmetic mean of the single-sample normalized Gram matrices of all samples in the real target domain subset and the arithmetic mean of the single-sample normalized Gram matrices of all samples in the cross-domain synthetic sample subset are calculated along the sample dimension. This yields the result of the calculation of the real target domain subset at the [number]th [period]. The batch mean and cross-domain synthetic sample subset of the hierarchical batch normalized Gram matrix at the th level The batch mean of the batch-level normalized Gram matrix is ​​shown below:

[0097] (5)

[0098] (6)

[0099] in, Represents a subset of the real target domain samples in the th case. The batch mean of the batch-level normalized Gram matrix. This represents a subset of the true target domain samples within the current hybrid fine-tuning batch. This represents the number of samples within the subset of the true target domain samples. Represents the first sample in the true target domain subset. One sample, Indicates the layer number of the preset feature layer. Indicates the cross-domain synthetic sample subset in the th domain. The batch mean of the batch-level normalized Gram matrix. This represents the subset of cross-domain synthesized samples within the current hybrid fine-tuning batch. This indicates the number of samples within the cross-domain synthetic sample subset. Represents the first sample in a cross-domain synthetic sample subset. One sample, Indicates the first The cross-domain synthetic sample in the first The normalized Gram matrix calculated on each preset feature layer Indicates the first The real target domain sample at the th The normalized Gram matrix calculated on each preset feature layer For the summation operator, it represents traversing a cross-domain synthetic sample subset. Perform a summation operation on all samples. For the summation operator, it represents traversing a subset of samples from the real target domain. Perform a summation operation on all samples.

[0100] Traverse the preset feature layer set The system uses pre-defined feature layers. For each layer, it calculates the square of the Frobenius norm between the mean of the batch-normalized Gram matrix of the cross-domain synthetic sample subset and the mean of the batch-normalized Gram matrix of the true target domain sample subset. This square is accumulated layer by layer to obtain the style correction loss, as shown below:

[0101] (7)

[0102] in, Indicates style correction loss, Indicates the cross-domain synthetic sample subset in the th domain. The batch mean of the batch-level normalized Gram matrix. Represents a subset of the real target domain samples in the th case. The batch mean of the batch-level normalized Gram matrix. The square of the Frobenius norm of the matrix difference is represented by... Indicates the layer number of the preset feature layer. This represents the set of preset feature layers used for style statistics. For the summation operator, it represents traversing the preset feature layer set. All layer numbers Perform a summation operation.

[0103] In step 106, the style correction loss is used as a weak regularization term and combined with the target detection loss to obtain a joint total loss. The trainable parameters of the target detection network are updated by backpropagation based on the joint total loss. The trainable parameters include the unfrozen parameters of the backbone network, the neck network parameters, and the detection head parameters.

[0104] Specifically, when the processor inputs the mixed fine-tuning batch into the initialized target detection network and completes the forward propagation, the backbone network of the target detection network completes feature extraction, the neck network completes multi-scale feature fusion, the detection head receives the fused features output by the neck network, and outputs the prediction result of the target to be detected for each sample in the batch; the prediction result includes the predicted target category and the predicted target location.

[0105] Since the hybrid fine-tuning batch contains enhanced real target domain samples and cross-domain synthetic samples, the enhanced real target domain samples carry a first target label to be detected, and the cross-domain synthetic samples carry a second target label to be detected. For the enhanced real target domain samples in the hybrid fine-tuning batch, the processor uses the first target label to be detected corresponding to the sample as the ground truth; for the cross-domain synthetic samples in the batch, it uses the second target label to be detected corresponding to the cross-domain synthetic sample as the ground truth.

[0106] The first target label includes the target category, center coordinates, width, and height; the second target label includes the target category, center coordinates, width, and height.

[0107] The processor compares the prediction results output by the detection head with the ground truth labels of the corresponding samples to calculate the target detection loss. In this embodiment, the target detection loss consists of two parts: classification loss and bounding box regression loss. The classification loss is used to measure the prediction deviation of the target category, and the bounding box regression loss is used to measure the prediction deviation of the target location.

[0108] The style correction loss is multiplied by a preset regularization weight coefficient and then added to the object detection loss to obtain the joint total loss, which is determined by the following formula:

[0109] (8)

[0110] in, This represents the object detection loss, which includes classification loss and bounding box regression loss. The preset regularization weights represent the style correction loss. This represents the joint total loss used for backpropagation to update network parameters during the training phase.

[0111] It should be noted that classification loss measures the prediction bias of the target category, while bounding box regression loss measures the prediction bias of the target location; due to the preset regularization weight coefficients... By taking a smaller positive number, the style correction loss accounts for a low proportion of the joint total loss; therefore, the style correction loss is called a weak regularization term. Through the above settings, the target detection loss is prioritized to play a dominant role, and the network prioritizes learning the target category and location detection capabilities. The style correction loss only plays an auxiliary constraint role, used to reduce the style statistical bias between the cross-domain synthetic sample subset and the real target domain sample subset, avoiding excessive intervention by the style term that could lead to a decline in detection performance.

[0112] The processor performs backpropagation using the joint total loss as the loss function. Backpropagation propagates the error back from the network output to the input, using chain rule differentiation to calculate the gradient of each layer's parameters with respect to the joint total loss. During this process, the frozen backbone network parameters do not participate in gradient updates; even if the gradient is calculated, the weights for those parameters will not be modified. Only trainable parameters are updated using the gradient.

[0113] In this embodiment, trainable parameters include parameters indicating that the backbone network is not frozen, parameters of the neck network, and parameters of the detection head.

[0114] After the parameter update is completed, the current iteration ends. The next batch of mixed fine-tuning is read, and the iterative training process is repeated to enter the next round of iterative training.

[0115] In step 107, when the iteration stopping condition is met, the final target detection network with the trainable parameters converged and updated is obtained. The target domain image to be detected in the real scene is input into the final target detection network, and the target category and target location corresponding to the target domain image to be detected are output.

[0116] The network has undergone multiple rounds of iterative training. Each iteration completes forward propagation, loss calculation, and backpropagation to update the trainable parameters of the object detection network for the hybrid fine-tuning batch. The processor continuously reads each hybrid fine-tuning batch, repeatedly executes the training process of forward propagation, loss calculation, and backpropagation to update the trainable parameters, and determines in real time whether the preset iteration stopping condition has been met.

[0117] In this embodiment, the iteration stopping condition can be at least one of the following: the preset training iteration round has been reached, the validation set detection index meets the preset threshold, etc. When the iteration stopping condition is not met, the next batch of mixed fine-tuning is read and the iteration training continues. When the iteration stopping condition is met, the target domain fine-tuning training process is terminated, and the final target detection network with converged and updated trainable parameters is obtained.

[0118] After training terminates, the processor disables the style correction branch. This branch contains all the computational logic used during training to extract the original feature maps of the preset feature layers, calculate the batch-normalized Gram matrix, and calculate the style correction loss. During inference, this branch's operations are no longer executed, thus not increasing the computational overhead of the inference phase. At this point, only the backbone network, neck network, and detection head of the final object detection network are retained for inference operations.

[0119] In practical applications, the target domain image to be detected in a real scene is input into the final target detection network. The target domain image to be detected is processed by the backbone network of the final target detection network for feature extraction. The feature map output by the backbone network is sent to the neck network to complete multi-scale feature fusion. The fused features output by the neck network are sent to the detection head. The detection head performs prediction calculations and outputs the target category and target location corresponding to the target domain image to be detected, thus completing the target detection task.

[0120] Example

[0121] This embodiment is used to verify the actual processing effect of the present invention in the quality inspection scenario of manufacturing workpiece defects. The verification is conducted on three typical industrial workpiece defects: welds, scratches, and cracks. The present invention organizes the detection of small-sample cross-domain industrial targets into three consecutive stages: the source domain pre-training stage, the cross-domain target sample recombination stage, and the target domain fine-tuning stage including cross-domain style correction. All the image processing, network training, and feature statistical calculation processes described below are executed by the processor of the computer equipment on the workpiece image data acquired by the industrial camera. Ultimately, this serves the automatic identification of workpiece defects on industrial production lines, alleviating the problem of missed or false detections of targets caused by the scarcity of available samples in the target domain image set labeled with the first target to be inspected after production line process changes.

[0122] In the source domain pre-training stage, a source domain image set with the first target to be inspected is acquired, and the target detection network including the backbone network, neck network and detection head is trained using the source domain image set with the first target to be inspected. The source domain pre-training enables the target detection network to acquire basic target localization and classification capabilities. In this stage, the basic morphological features of the target to be inspected corresponding to industrial workpieces such as welds, scratches, and cracks are learned.

[0123] The target domain image set labeled with the first target to be inspected is used for subsequent fine-tuning with fewer samples; a single original target domain image can contain one or more instances of the target to be inspected. The target domain samples provide the morphology of the target to be inspected consistent with the actual deployment conditions, but their coverage of background, lighting, and imaging conditions is limited. In actual industrial production lines, this situation corresponds to the reality that after changing workstations, light sources, and cameras, the number of manually annotated target images such as welds, scratches, and cracks in the target domain image set labeled with the first target to be inspected is insufficient.

[0124] In the cross-domain target sample recombination stage, a source domain target mask is first generated based on the first target annotation in the source domain image. Image restoration and removal of the original target from the source domain are then performed on the mask-covered area to create a cleaned source background image. Next, real target patches are cropped from the original target domain image based on the first target annotation and placed into the effective placement area of ​​the cleaned source background image. After local photometric alignment and fusion using a continuous soft mask matrix, a fused image is formed. Simultaneously, a second target annotation is automatically generated based on the placed target category, center coordinates, width, and height, resulting in a cross-domain synthetic sample. This process corresponds to the cross-domain target sample recombination algorithm flow. This step expands the training sample size by generating cross-domain synthetic samples, compensating for the insufficient number of labeled images in the target domain image set with the first target annotation, while suppressing stitching artifacts caused by patch pasting and preventing the network from learning false stitching traces that lead to misjudgment of the target.

[0125] In the target domain fine-tuning stage, semantic preservation enhancement is performed on the original target domain image to obtain enhanced real target domain samples, which together with the cross-domain synthetic samples generated by the cross-domain target sample recombination algorithm constitute a hybrid fine-tuning dataset.

[0126] The target detection network is initialized using source domain pre-trained parameters, and some shallow network parameters of the target detection network can be frozen to reduce the risk of overfitting under few sample conditions. This prevents the network from overfitting to a small number of target samples corresponding to welds, scratches, and cracks, and ensures the generalization ability of the target detection network on real industrial workpiece images.

[0127] Since the background of cross-domain synthetic samples originates from the source domain, their features may still contain source domain textures, colors, or imaging styles. This invention sets up a cross-domain style correction algorithm during fine-tuning. It uses the original feature maps corresponding to preset feature layers to calculate the batch-level normalized Gram matrix of the real target domain sample subset and the cross-domain synthetic sample subset within the mixed fine-tuning batch, respectively. The batch-level features of the real target domain sample subset are used to statistically correct the residual source background style of the cross-domain synthetic sample subset. The cross-domain style correction algorithm only serves as a weak regularization constraint during training and does not modify the detection head. After the iteration stopping condition is met, the style correction branch is disabled, and only the final target detection network, updated with convergent trainable parameters, is used to complete deployment and inference. This process eliminates interference from the residual background style of cross-domain synthetic samples, making the target detection network more adaptable to the imaging of real workpieces corresponding to the target domain image set labeled with the first target to be detected, thus improving the recognition accuracy of welds, scratches, and cracks.

[0128] Specifically, the cross-domain target sample recombination algorithm process includes extraction of real target image patches, source background purification, determination of effective placement area and scale, local photometric alignment, continuous soft mask fusion, and simultaneous generation of annotations for the second target.

[0129] Cropping the actual target patch based on the first target annotation in the original target domain image:

[0130] (9)

[0131] in, Indicates the first A real target image patch obtained by cropping from the original target domain image. Represents the original target domain image. This represents the image cropping operator. This indicates the first [item] in the original target domain image. The first target to be inspected is marked with a bounding box. The actual target image obtained here is a local image of the target to be inspected, such as a weld, scratch, or crack, in an industrial workpiece.

[0132] Generate a source domain target mask based on one or more first target annotations in the source domain image, and perform image inpainting on the area covered by the source domain target mask:

[0133] (10)

[0134] in, This indicates the background image of the purified source region, after removing the original target from the source region. Represents the image restoration operator. Represents the source domain image. This represents the binary mask of the target to be inspected in the source domain, and marks the area of ​​the original target to be inspected in the source domain image that needs to be repaired.

[0135] After the original target to be inspected in the source domain is removed, the cleaned source background image continues to retain the material texture, lighting variations, and imaging context of the source domain, while avoiding overlap between the original target to be inspected in the source domain and the migrated target to be inspected. The cleaned source background image retains the real texture and lighting noise of the industrial workpiece surface, making the generated cross-domain synthetic sample closer to the workpiece image collected from the real production line.

[0136] Determine the effective placement area within the cleaned source background image. The effective placement area should ensure that the center of the actual target image patch is located in the central effective area of ​​the image and far from the image boundaries. If the actual target image patch is too large relative to the cleaned source background image, scale it proportionally while maintaining the aspect ratio.

[0137] Before placement, local photometric alignment is performed based on the average brightness of the actual target block to be inspected and the effective placement area to obtain the actual target block to be inspected as described in formula (1).

[0138] A continuous soft mask matrix with a higher weight in the central region and a gradually decreasing weight in the edge region is constructed and fused to obtain a local synthetic region as described in formula (2).

[0139] Let the center of the target to be inspected after fusion be... The width and height of the actual target image after transformation are respectively and Then the synthesized bounding box of the target to be detected can be represented as:

[0140] (11)

[0141] in, This represents the composite bounding box of the second target to be inspected. This represents the x-coordinate of the center of the actual target patch after merging and placement. This represents the ordinate of the center of the actual target patch after merging and placement. This represents the width of the actual target tile after scaling transformation. This represents the height of the actual target image block after scaling transformation.

[0142] By normalizing the center coordinates and width / height according to the dimensions of the fused image and writing them along with the category into the second target annotation, cross-domain synthetic samples can be obtained. Therefore, cross-domain synthetic samples do not require additional manual annotation. Cross-domain synthetic samples containing targets such as welds, scratches, and cracks can be generated in large quantities without manual annotation, reducing the labor cost of sample annotation in industrial quality inspection scenarios.

[0143] Specifically, during target domain fine-tuning, enhanced real target domain samples and cross-domain synthesized samples are jointly input into the target detection network. The cross-domain style correction algorithm obtains original feature maps from one or more shallow to mid-layers before multi-scale feature fusion in the backbone network and neck network. These original feature maps preserve local contrast, edge continuity, texture distribution, and background-target transition information, making them suitable for constraining residual source background styles in cross-domain synthesized samples.

[0144] The aforementioned feature information corresponds to the edge and texture features of the target to be inspected on the surface of the industrial workpiece, ensuring that the contour information of the target to be inspected, such as welds, scratches, and cracks, is not destroyed by the correction operation.

[0145] For the For each preset feature layer, the original feature map and its flattened two-dimensional feature matrix respectively satisfy:

[0146] (12)

[0147] (13)

[0148] in, Indicates sample In the The original feature map output by each preset feature layer has a three-dimensional dimension, with the dimension size being the number of channels. Feature map height Feature map width , Represents the three-dimensional feature map tensor space. Indicates the first The number of channels in a preset feature layer Indicates the feature map height. Indicates the width of the feature map. Indicates that the sample In the The two-dimensional feature matrix obtained by flattening the original feature maps output by the preset feature layers in terms of spatial dimensions is a two-dimensional matrix with a dimension of . OK, List, Represents the flattened two-dimensional matrix space. This represents a single input sample.

[0149] The normalized Gram matrix of the sample in this layer is determined by formula (4). The samples in the mixed fine-tuning batch are divided into a subset of real target domain samples and a subset of cross-domain synthetic samples. Within the subset of real target domain samples and the subset of cross-domain synthetic samples, the arithmetic mean of the single-sample normalized Gram matrix of all samples in the subset of real target domain samples and the arithmetic mean of the single-sample normalized Gram matrix of all samples in the subset of cross-domain synthetic samples are calculated according to the sample dimension. The result is obtained when the subset of real target domain samples is in the first layer. The batch mean and cross-domain synthetic sample subset of the hierarchical batch normalized Gram matrix at the th level The batch mean of the batch-level normalized Gram matrix is ​​shown in formulas (5) and (6).

[0150] Traverse the preset feature layer set All internal feature layers are preset. For each layer, the square of the Frobenius norm between the mean of the batch-normalized Gram matrix of the cross-domain synthetic sample subset and the mean of the batch-normalized Gram matrix of the real target domain sample subset is calculated. The values ​​are accumulated layer by layer to obtain the style correction loss as described in formula (7).

[0151] Multiply the style correction loss by the preset regularization weight coefficient and add it to the target detection loss to obtain the joint total loss as shown in formula (8).

[0152] The cross-domain style correction algorithm aligns the batch-normalized Gram matrix statistics of the cross-domain synthetic sample subset and the real target domain sample subset, without requiring a one-to-one correspondence between the samples for the target category, location, or spatial structure. The target detection loss continues to provide classification and localization supervision, thus preserving the target's boundary, shape, and localization cues while reducing the influence of residual source background style. The style correction branch is deactivated upon reaching the iteration stopping condition, and the inference network remains consistent with the basic target detection network. This approach avoids smoothing out the key contour features of weld seams, scratches, and cracks during the correction process, ensuring the localization accuracy of industrial workpieces.

[0153] Cross-scenario verification:

[0154] This section validates workpiece images acquired entirely by industrial cameras, conducting tests on three types of manufacturing workpiece defects: steel plate welds, hot-rolled steel strip scratches, and wind turbine blade cracks. Three sets of cross-domain industrial datasets are used for validation: steel plate weld images to ship weld images (approximately 400 source domain images with first-target annotations, and 258 target domain images with first-target annotations); mobile phone screen scratch images to hot-rolled steel strip scratch images (approximately 110 target domain images with first-target annotations); and wind turbine blade crack images to KolektorSDD dataset images (approximately 1308 source domain images with first-target annotations, and 52 target domain images with first-target annotations). These three sets of data respectively demonstrate cross-domain variations in background and lighting, material reflection and texture, and material contrast and crack geometry, corresponding to imaging differences caused by changes in production lines, workpiece materials, and lighting conditions in industrial production.

[0155] Industrial cross-domain validation includes the following three sets of tasks: transferring a source domain image set labeled with a first target object to a target domain image set labeled with the first target object:

[0156] Table 1 Industrial Cross-Domain Validation Task Settings

[0157]

[0158] Data from the target domain image set labeled with the first target to be inspected is divided into non-overlapping training pools, validation sets, and test sets. For each industrial task, a corresponding number of labeled images are re-sampled from the target training pool under 1-shot, 2-shot, and 5-shot settings. These extracted images serve three purposes: first, to generate enhanced real target domain samples; second, to extract real target patches for the cross-domain target sample recombination algorithm; and third, to construct real target features for the cross-domain style correction algorithm. The cleaned source background images are taken only from the training set corresponding to the source domain image set labeled with the first target to be inspected. The validation set is used to select the optimal checkpoint during the target domain fine-tuning process, and the independent test set is used only for the final evaluation of this checkpoint. The test set consists entirely of real industrial workpiece images to simulate the inference and detection conditions of target domain images in actual production lines.

[0159] Each task and each small sample setting underwent three independent trials, each using newly extracted images from a target domain image set labeled with the first target to be detected and an independent random seed; data partitioning and sampling results within the same trial remained consistent across all comparison methods and ablation variants. The following results are the arithmetic mean of the three trials. Detection performance was evaluated using precision (P), recall (R), mAP (mean Average Precision) 50, and mAP 50-95.

[0160] The above evaluation indicators directly reflect the detection and location capabilities of welds, scratches, and cracks, and are used to evaluate the practical performance of this method in the context of defect inspection of workpieces in the manufacturing industry.

[0161] To further compare the target detection accuracy of the method of the present invention with that of existing methods under different sample sizes of target domain image sets with the first target to be detected, comparative tests were conducted under 5-shot, 2-shot and 1-shot settings respectively.

[0162] Table 2. Quantitative Comparison Results of Cross-Domain Weld Inspection Tasks

[0163]

[0164] Experimental analysis of weld seam tasks: Under a 5-shot setting, the P, R, mAP50, and mAP50-95 values ​​of the method of this invention are 0.987, 0.996, 0.997, and 0.693, respectively, all of which are the optimal values ​​among the listed methods. When the number of available images in the target domain image set with the first target annotation is reduced from 5 to 2, mAP50 decreases by only 0.003, and mAP50-95 decreases by only 0.011; under a 1-shot setting, P, R, and mAP50 still reach 0.947, 0.985, and 0.991, respectively. Although the mAP50-95 of FT-FSOD under a 1-shot setting is 0.670, slightly higher than the 0.648 of the method of this invention, the P, R, and mAP50 of the method of this invention are higher under this setting. These results indicate that the method of this invention has good target detection coverage and recognition stability under conditions of a small number of available images in the target domain and non-uniform illumination.

[0165] Table 3. Quantitative comparison results of cross-domain scratch detection tasks

[0166]

[0167] Scratch task experiment analysis: Under the 5-shot setting, the method of this invention achieved the best results in all four indicators, with mAP50 at 0.877, an improvement of 0.077 compared to FSDA-DETR's 0.800, and mAP50-95 at 0.496, higher than FT-FSOD's 0.480. Under the 2-shot setting, FT-FSOD's P and mAP50-95 were 0.857 and 0.479, respectively, slightly higher than the method of this invention; the method of this invention achieved R and mAP50 at 0.823 and 0.844, respectively, the highest values ​​among the listed methods. Under the 1-shot setting, the method of this invention achieved the best results in all four indicators, indicating that the method of this invention has good adaptability to weak contrast, fine scratches, and metallic texture interference.

[0168] Table 4. Quantitative Comparison Results of Crack Cross-Domain Detection Tasks

[0169]

[0170] Crack detection task experimental analysis: Under 5-shot and 2-shot settings, the P, R, and mAP50 values ​​of the method of this invention are the highest among the listed methods. The mAP50-95 of FT-FSOD are 0.482 and 0.410, respectively, slightly higher than that of the method of this invention. Under the 1-shot setting, all four indicators of the method of this invention achieve optimal values, with P, R, mAP50, and mAP50-95 being 0.946, 0.682, 0.804, and 0.382, respectively. The above experimental results show that when the number of labeled images corresponding to crack-type targets is extremely small, the method of this invention can still maintain relatively stable target detection coverage and recognition capabilities. Different methods differ in strict localization indicators, which also indicates that target detection capability and high-precision boundary localization capability belong to different optimization focuses.

[0171] Under a 5-shot setup, ablation tests were performed on schemes including basic fine-tuning, incorporating cross-domain target sample recombination algorithms, and simultaneously incorporating both cross-domain target sample recombination and cross-domain style correction algorithms. The results are as follows:

[0172] Table 5 Results of 5-shot ablation experiments

[0173]

[0174] The above results demonstrate that, under the stated dataset, detector, and training settings, the cross-domain target sample reconstruction algorithm improves the mAP50 and mAP50-95 scores for all three industrial tasks. Here, CTSR represents the cross-domain target sample reconstruction algorithm, and CDSC represents the cross-domain style correction algorithm. Introducing the cross-domain style correction algorithm into the cross-domain target sample reconstruction algorithm further improves both evaluation metrics. This proves that sample reconstruction and style correction can synergistically improve target detection accuracy under cross-domain limited-sample conditions. The performance improvement varies under different tasks, different limited-sample quantities, and different evaluation metrics. These experimental results do not guarantee that this invention will be absolutely superior across all datasets or all evaluation metrics.

[0175] Based on the same inventive concept, this invention provides a cross-domain target detection device based on sample recombination and style correction. Since the principle of this device in solving the technical problem is similar to that of the cross-domain target detection method based on sample recombination and style correction, the implementation of this device can refer to the implementation of the method, and the repeated parts will not be described again.

[0176] like Figure 3 As shown, the device includes a first obtaining unit 201, a second obtaining unit 202, an initialization unit 203, a selection unit 204, a construction unit 205, an update unit 206, and an output unit 207.

[0177] The first obtaining unit 201 is used to obtain a source domain image set with a first target annotation and a target domain image set with a first target annotation, train a target detection network using the source domain image set with the first target annotation to obtain source domain pre-training parameters; generate a source domain target mask based on the first target annotation of the source domain image, and perform image restoration on the area covered by the source domain target mask to obtain a purified source background image.

[0178] The second obtaining unit 202 is used to extract real target patches based on the first target label of the original target domain image, implant the real target patches into the background image of the purification source and determine the effective placement area, perform geometric constraints, local photometric alignment and continuous soft mask fusion on the real target patches, and generate a second target label based on the fused placement position, transformation size and target category to obtain a cross-domain synthetic sample.

[0179] Initialization unit 203 is used to perform semantic preservation enhancement on the original target domain image to obtain enhanced real target domain samples, combine the enhanced real target domain samples with cross-domain synthetic samples to form a hybrid fine-tuning dataset and divide it into hybrid fine-tuning batches, and initialize the target detection network using source domain pre-trained parameters.

[0180] The selection unit 204 is used to select one or more fixed network layers from the shallow to middle layers of the backbone network of the initialized target detection network as preset feature layers before the training iteration. All the preset feature layers are summarized to form a preset feature layer set.

[0181] The construction unit 205 is used to input the mixed fine-tuning batch into the initialized target detection network during the iteration process, calculate the batch-level normalized Gram matrix on the original feature map corresponding to the preset feature layer for the real target domain sample subset and the cross-domain synthetic sample subset within the mixed fine-tuning batch, respectively, and construct the style correction loss based on the difference between the batch-level normalized Gram matrices of the two types of sample subsets.

[0182] The update unit 206 is used to combine the style correction loss as a weak regularization term with the target detection loss to obtain a joint total loss, and to update the trainable parameters of the target detection network by backpropagation based on the joint total loss. The trainable parameters include the unfrozen parameters of the backbone network, the neck network parameters, and the detection head parameters.

[0183] The output unit 207 is used to obtain the final target detection network with converged update of trainable parameters when the iteration stopping condition is reached. The target domain image to be detected in the real scene is input into the final target detection network, and the target category and target location corresponding to the target domain image to be detected are output.

[0184] It should be understood that the units included in the above-described cross-domain target detection device based on sample recombination and style correction are only logically divided according to the functions implemented by the device. In practical applications, the above units can be superimposed or split. Furthermore, the functions implemented by the cross-domain target detection device based on sample recombination and style correction provided in this embodiment correspond one-to-one with the cross-domain target detection method based on sample recombination and style correction provided in the above embodiments. The more detailed processing flow implemented by this device has been described in detail in the first embodiment of the above method, and will not be described in detail here.

[0185] Another embodiment of the present invention provides a computer device, the computer device including: a processor and a memory; the memory is used to store computer program code, the computer program code including computer instructions; when the processor executes the computer instructions, the computer device performs each step of the cross-domain target detection method based on sample recombination and style correction shown in the above method embodiment.

[0186] Another embodiment of the present invention provides a computer-readable storage medium storing computer instructions that, when executed on a computer device, cause the computer device to perform the various steps of the cross-domain target detection method based on sample recombination and style correction as described in the above method embodiments.

[0187] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A cross-domain target detection method based on sample recombination and style correction, characterized in that, include: A source domain image set with a first target annotation and a target domain image set with a first target annotation are obtained. The target detection network is trained using the source domain image set with the first target annotation to obtain source domain pre-training parameters. A source domain target mask is generated based on the first target annotation of the source domain images. Image restoration is performed on the area covered by the source domain target mask to obtain a purified source background image. Based on the first target annotation of the original target domain image, extract the real target patch, embed the real target patch into the background image of the purification source and determine the effective placement area, perform geometric constraints, local photometric alignment and continuous soft mask fusion on the real target patch, and generate the second target annotation based on the fused placement position, transformation size and target category to obtain the cross-domain synthetic sample. The original target domain image is semantically preserved and enhanced to obtain enhanced real target domain samples. The enhanced real target domain samples and cross-domain synthetic samples are combined to form a hybrid fine-tuning dataset and divided into hybrid fine-tuning batches. The target detection network is initialized using source domain pre-trained parameters. Before training iteration, one or more fixed network layers are selected from the shallow to middle layers of the backbone network of the initialized target detection network as preset feature layers, and all the preset feature layers are summarized to form a preset feature layer set. During the iteration process, the hybrid fine-tuning batch is input into the initialized target detection network. For the real target domain sample subset and the cross-domain synthetic sample subset within the hybrid fine-tuning batch, the batch-level normalized Gram matrix is ​​calculated on the original feature map corresponding to the preset feature layer. The style correction loss is constructed based on the difference between the batch-level normalized Gram matrices of the two types of sample subsets. The style correction loss is combined with the target detection loss as a weak regularization term to obtain the joint total loss. The trainable parameters of the target detection network are updated by backpropagation based on the joint total loss. The trainable parameters include the unfrozen parameters of the backbone network, the neck network parameters, and the detection head parameters. When the iteration stopping condition is met, the final target detection network with converged and updated trainable parameters is obtained. The target domain image to be detected in the real scene is input into the final target detection network, and the target category and target location corresponding to the target domain image are output.

2. The method as described in claim 1, characterized in that, The process involves extracting real target patches from the first target annotation of the original target domain image, embedding the real target patches into the background image of the purification source and determining the effective placement area, performing geometric constraints, local photometric alignment, and continuous soft mask fusion on the real target patches, and generating a second target annotation based on the fused placement position, transformed size, and target category to obtain a cross-domain synthetic sample. Specifically, this includes: Based on the first target label corresponding to a single original target domain image of the target domain image set with the first target label, the real target image patch is cropped and extracted; an effective placement area is determined within the background image of the purification source. Align the center of the actual target image block to be inspected with the center of the effective placement area. When the size of the actual target image block to be inspected is larger than the size of the background image of the purification source, scale the actual target image block to be inspected proportionally so that the actual target image block to be inspected is placed within the effective placement area. The average brightness of the real target image block to be inspected and the effective placement area inside the background image of the purification source are determined. The brightness difference between the real target image block to be inspected and the effective placement area is eliminated by photometric alignment. All pixels are constrained to the legal pixel range by the pixel truncation function to obtain the real target image block to be inspected after local photometric alignment. Using a continuous soft mask matrix, the real target image patch after local photometric alignment is fused into the effective placement area of ​​the clean source background image to obtain a local composite area. The pixels of the synthesized region are then backfilled into the effective placement area of ​​the purified source background image to obtain the fused image. A second target label is generated based on the final placement state of the real target image patch in the effective placement area. Based on the fused image and the second target label, a cross-domain synthetic sample for target domain network fine-tuning is generated.

3. The method as described in claim 2, characterized in that, The actual target image patch after local photometric alignment is determined by the following formula: ； The synthetic region is determined by the following formula: ； in, This represents the actual target image patch after local photometric alignment. Indicates the first A real target image patch extracted from the original target domain image. This function represents the average brightness of an image region. This indicates the effective placement area in the background image of the purification source. This represents a pixel truncation function that restricts the output pixels to the range of valid pixel values ​​in the image. This represents the composite region obtained by fusing the real target image patch with the background image of the clean source. This indicates that the size of the target image patch is the same as the actual target image patch, and the value range is... A continuous soft mask matrix is ​​used, and the fusion weights decrease continuously from the center of the patch to the edge. This represents the element-wise multiplication operation of matrices.

4. The method as described in claim 1, characterized in that, The hybrid fine-tuning dataset is determined by the following formula: ； in, This indicates that a semantically preserving and enhancing operator is performed on the original target domain image. This represents the target domain image set labeled with the first target to be detected. This represents a sample set consisting of several cross-domain synthesized samples. This represents a hybrid fine-tuning dataset used for fine-tuning the target domain. This represents the union operation of sets.

5. The method as described in claim 1, characterized in that, The original feature map corresponding to the preset feature layer is the unfused original feature map output by the target detection network backbone network after feature extraction is completed and before multi-scale feature fusion is performed by the neck network. The step of calculating the batch-level normalized Gram matrix on the original feature map corresponding to the preset feature layer specifically includes: For each sample in the hybrid fine-tuning batch, the original feature map output by each preset feature layer of a single sample is flattened into a two-dimensional feature matrix, and the normalized Gram matrix of a single sample is determined based on the two-dimensional feature matrix. ； in, Indicates the layer number of the preset feature layer. This represents a single input sample. Indicates that the sample In the The two-dimensional feature matrix is ​​obtained by flattening the original feature map output by each preset feature layer in terms of spatial dimensions. This represents the matrix transpose operation. Indicates the first The number of channels in a preset feature layer Indicates the first The height of the output feature map of each preset feature layer Indicates the first The width of the output feature map of each preset feature layer. Indicates sample In the The normalized Gram matrix calculated on each preset feature layer represents the second-order statistical correlation between feature channels.

6. The method as described in claim 1, characterized in that, The two types of sample subset batch-normalized Gram matrices include the cross-domain synthesized sample subsets at the [number]th [time]. The batch mean of the hierarchical batch normalized Gram matrix and the true target domain sample subset at the th level The batch mean of the batch-level normalized Gram matrix is ​​determined by the following formula: ； ； in, Indicates the cross-domain synthetic sample subset in the th domain. The batch mean of the batch-level normalized Gram matrix. This represents the subset of cross-domain synthesized samples within the current hybrid fine-tuning batch. This indicates the number of samples within the cross-domain synthetic sample subset. Represents the first sample within a cross-domain synthetic sample subset. One sample, Represents a subset of the real target domain samples in the th case. The batch mean of the batch-level normalized Gram matrix. This represents a subset of the true target domain samples within the current hybrid fine-tuning batch. This represents the number of samples within the subset of the true target domain samples. Represents the first sample in the true target domain subset. One sample, Indicates the layer number of the preset feature layer. Indicates the first The cross-domain synthetic sample in the first The normalized Gram matrix calculated on each preset feature layer Indicates the first The real target domain sample at the th The normalized Gram matrix calculated on each preset feature layer For the summation operator, it represents traversing a cross-domain synthetic sample subset. Perform a summation operation on all samples. For the summation operator, it represents traversing a subset of samples from the real target domain. Perform a summation operation on all samples.

7. The method as described in claim 1, characterized in that, The style correction loss is determined by the following formula: ； The combined total loss is determined by the following formula: ； in, Indicates style correction loss, Indicates the cross-domain synthetic sample subset in the th domain. The batch mean of the batch-level normalized Gram matrix. Represents a subset of the real target domain samples in the th case. The batch mean of the batch-level normalized Gram matrix. The square of the Frobenius norm of the matrix difference is represented by... Indicates the layer number of the preset feature layer. This represents the set of preset feature layers used for style statistics. This represents the object detection loss, which includes classification loss and bounding box regression loss. The preset regularization weights represent the style correction loss. This represents the joint total loss used for backpropagation to update network parameters during the training phase. For the summation operator, it represents traversing the preset feature layer set. All layer numbers Perform a summation operation.

8. A cross-domain target detection device based on sample recombination and style correction, characterized in that, include: The first obtaining unit is used to acquire a source domain image set with a first target annotation and a target domain image set with a first target annotation, train a target detection network using the source domain image set with the first target annotation to obtain source domain pre-training parameters; generate a source domain target mask based on the first target annotation of the source domain image, and perform image restoration on the area covered by the source domain target mask to obtain a purified source background image. The second obtaining unit is used to extract real target patches based on the first target label of the original target domain image, implant the real target patches into the background image of the purification source and determine the effective placement area, perform geometric constraints, local photometric alignment and continuous soft mask fusion on the real target patches, and generate a second target label based on the fused placement position, transformation size and target category to obtain a cross-domain synthetic sample. An initialization unit is used to perform semantic preservation enhancement on the original target domain image to obtain enhanced real target domain samples. The enhanced real target domain samples and cross-domain synthetic samples are combined to form a hybrid fine-tuning dataset and divided into hybrid fine-tuning batches. The target detection network is initialized using source domain pre-trained parameters. The selection unit is used to select one or more fixed network layers from the shallow to middle layers of the backbone network of the initialized target detection network as preset feature layers before the training iteration. All the preset feature layers are summarized to form a preset feature layer set. The construction unit is used to input the mixed fine-tuning batch into the initialized target detection network during the iteration process, calculate the batch-level normalized Gram matrix on the original feature map corresponding to the preset feature layer for the real target domain sample subset and the cross-domain synthetic sample subset within the mixed fine-tuning batch, respectively, and construct the style correction loss based on the difference between the batch-level normalized Gram matrices of the two types of sample subsets. The update unit is used to combine the style correction loss as a weak regularization term with the target detection loss to obtain a joint total loss, and to update the trainable parameters of the target detection network by backpropagation based on the joint total loss. The trainable parameters include the unfrozen parameters of the backbone network, the neck network parameters, and the detection head parameters. The output unit is used to obtain the final target detection network with converged update of trainable parameters when the iteration stopping condition is met. The target domain image to be detected in the real scene is input into the final target detection network, and the target category and target location corresponding to the target domain image are output.

9. A computer-readable storage medium, characterized in that, The system contains a computer program that, when executed by a processor, causes the processor to perform the cross-domain target detection method based on sample recombination and style correction as described in any one of claims 1-7.

10. A computer device, characterized in that, The computer device includes a memory and a processor. The memory stores a computer program, which, when executed by the processor, causes the processor to perform the cross-domain target detection method based on sample recombination and style correction as described in any one of claims 1-7.

Images