An anchor-free based domain adaptation target detection device, target detection method and system

Abstract

The application discloses an anchor-free based domain adaptive target detection model, comprising a foreground domain adaptive structure based on a double prediction layer and a background domain adaptive structure based on sample allocation; the application also discloses a detection method, comprising: constructing an anchor-free based domain adaptive target detection model; taking a labeled target domain simulation target data set as a domain adaptive target detection model training input, and obtaining a pre-training domain adaptive target detection model used for detecting unlabeled target domain real target images; the labeled target domain simulation target data set and the unlabeled target domain real target images have domain differences; obtaining a target detection result based on inputting the unlabeled target domain real target images into the pre-training domain adaptive fruit detection model; processing the unlabeled target domain real target images and the target detection result, generating and outputting a labeled target domain real target data set. The application also discloses corresponding systems, electronic devices and computer readable storage media.

Description

Technical Field

[0001] This invention relates to the fields of image processing and intelligent information extraction technology, and in particular to an anchor-free domain-adaptive target detection device, target detection method and system. Background Technology

[0002] High-precision fruit detection technology is a prerequisite and key to realizing intelligent information technology construction and automated orchard management in the fruit industry. With the improvement of agricultural automation and the development of deep learning, more and more deep learning-based detection models are being used for fruit detection. Deep learning-based fruit detection technology can extract various features from images and structured data collected in orchards and effectively combine them with agricultural machinery to be better applied to many intelligent orchard tasks such as fruit positioning, fruit sorting, fruit yield prediction, and automatic fruit harvesting.

[0003] At present, deep learning-based fruit detection technology has been widely used. However, it relies on a large number of labeled fruit datasets to support the training and learning of the detection model, which is costly due to manual labeling. Furthermore, due to the poor generalization performance of current deep learning models, new fruit datasets need to be created and new detection model structures need to be designed for different scenarios, environments, shooting methods, and fruit types. The whole process is time-consuming and labor-intensive.

[0004] Given this problem, automatic annotation methods for datasets are receiving increasing attention. The automatic annotation work for fruit datasets aims to automatically generate target bounding boxes from a set of image data of real-world target scenes, while achieving a certain level of annotation accuracy to meet various practical applications in smart industry construction. Therefore, there is an urgent need to design a method that can improve the automatic annotation accuracy of different categories of target datasets.

[0005] The applicant's previous research, EasyDAMv1 and EasyDAMv2, proposed an automatic annotation method for fruit datasets. This method can automatically annotate fruit images using a pre-annotated fruit image dataset as the source domain and various unannotated fruit categories as the target domain, while achieving a certain level of annotation accuracy, thus saving the manual cost required for fruit dataset annotation. The method mainly includes the following steps: First, using a set of pre-annotated source domain fruit images, a generative adversarial network (GAN) is used to obtain simulated images with target domain fruit features; then, these simulated images are used as training data and input into the applicant's previously proposed Orange-Yolo fruit detection network to learn the target domain fruit features; finally, the target domain fruit images to be annotated are input into the pre-trained Orange-Yolo fruit detection model to obtain pseudo-label data for the target domain fruit images, and a pseudo-label update strategy is used to further improve the pseudo-label accuracy, thereby completing the automatic annotation of the target domain fruit dataset.

[0006] However, because the training data used to train the fruit detection network in this process consists of source domain fruit images generated by GANs to simulate target domain fruit images, while the images to be labeled are real target domain fruit images from actual scenes, there is a significant domain difference between the training data of the fruit detection model and the actual application data. Whether this domain difference problem can be solved directly affects the final labeling accuracy. Specifically, this domain difference problem can be further divided into foreground domain difference and background domain difference. Figure 1 This demonstrates an example of the domain difference problem in the EasyDAM method:

[0007] (1) Foreground domain difference problem (e.g.) Figure 1 (b) The problem of mismatch between the fruit training label box and the fruit target scale in (c): Due to the various target domain simulation fruit images in the EasyDAM automatic annotation method (such as...) Figure 1 In (b)), the fruit foreground target is derived from the source domain fruit image (e.g., ...). Figure 1 In (a)), the foreground fruit target is derived through a generative adversarial network, and its supervision information still retains the scale features of the source domain fruit. However, due to differences in shooting angle, shooting distance, and the type of fruit in the target domain, the real fruit in the target domain (such as...) is not accurately represented. Figure 1 The scale features of the fruit in the source domain are diverse, and the scale feature information of the fruit in the target domain cannot be completely matched with the actual scale of the real fruit. The scale difference is large, which causes the foreground domain difference problem and affects the quality of the pseudo-labels generated by the fruit detection model.

[0008] (2) Background domain differences (e.g.) Figure 1(b) The fruit background in the image is inconsistent with the fruit background scene in (c): This is because the target domain simulates the fruit image in the EasyDAM automatic annotation method (e.g., Figure 1 (b) is composed of source domain fruit images (e.g.) Figure 1 When performing the transformation in (a)), only the foreground target of the fruit is transformed and generated, while the background of the fruit image in the source domain is still retained, but the real fruit image in the target domain (such as...) is transformed. Figure 1 In (c) the background of the fruit in the actual scene, the difference in background domain caused by the fruit detection model further affected the quality of the pseudo-labels generated.

[0009] See Figure 1 Taking citrus as the source domain and dragon fruit as the target domain as an example (the boxes in the figure represent label information). Figure 1 (a) represents a source domain citrus fruit image. Figure 1 (b) represents the target domain, a simulated dragon fruit image (fruit detection model training data). Figure 1 (c) represents the actual dragon fruit image in the target domain (data used in the fruit detection model). Figure 1 (b) by Figure 1 (a) It is derived from a generative adversarial network, thus causing Figure 1 (b) and Figure 1 (c) There is a significant domain difference between the foreground fruit target and the background.

[0010] To address the above issues, this paper analyzes existing technologies in two aspects: the ability of object detection networks to learn scale information of foreground objects and the learning methods of foreground-background features in images.

[0011] (1) Existing technologies for foreground target scale feature information: Currently, most methods for detecting fruits such as apples, mangoes, peaches, and pears employ anchor-based detectors. These detectors introduce prior knowledge of the fruit target scale through anchor box settings to adapt to the generation of detection boxes at different fruit target scales. Some methods have further designed multi-scale detection structures to detect fruit targets of different sizes. These methods primarily rely on the training data and actual application data conforming to the same scale information distribution for model training, thus ensuring model performance on test data. However, when the scale information distributions of the source and target domains differ, the anchor box parameters in the anchor-based model used in the original EasyDAM method mainly come from the fruit scale information in the source domain, making it difficult to effectively apply to fruit detection tasks under different scale conditions in the target domain scene.

[0012] Therefore, efforts began to be made to design anchor-free detectors. These networks do not require pre-defined anchor boxes and are not limited by the scale features of the targets in the training set when generating detection boxes, thus avoiding the lack of flexibility in detection box generation caused by pre-defined anchor boxes. Currently, some anchor-free detection models have been used for fruit detection. For example, the TomatoDet model avoids the problems of complex hyperparameter design and low detection efficiency caused by anchor-based detectors that require exhaustive enumeration of anchor boxes and classification operations. However, the circular bounding boxes predicted by this network are only suitable for tomatoes or other round fruits with an aspect ratio close to 1:1, and are not suitable for detecting other fruit categories with significant length and width differences. Another example is the ShuffleNetV2-YOLOX model for apple detection. Its anchorless structure avoids the computational burden caused by anchor boxes, achieving a balance between speed and accuracy. However, this network is only designed for the characteristics of apples and cannot be effectively applied to the detection of multiple fruit categories. Third, a D2D model was used to detect green persimmons and green apples. The anchorless detection structure avoided excessive computation and storage costs. However, its performance was poor for fruits of other colors besides green. Fourth, the anchorless detector CenterNet was used for target detection on four fruit datasets: apples, bananas, oranges, and pears. This avoided the drawbacks of complex parameter tuning and high computational cost associated with anchor-box detectors. However, this network is designed for detecting fruits that are picked and placed on a table, and whose relative size in the image is relatively large, making it difficult to apply to complex real-world field scenarios.

[0013] Therefore, current research on fruit detection using anchor-free detection models focuses on avoiding the hyperparameters and computational burden caused by anchor frames, thus achieving a balance between speed and accuracy. Furthermore, these models perform well only for single fruit species or fixed shooting methods and scenes, failing to generalize or adapt to the complex fruit detection tasks of different species and scales in real-world field scenarios. Therefore, the first problem this invention aims to solve is how to leverage the characteristic of anchor-free detection models—that they are not limited by prior anchor frame scale information—to further design detection structures that cater to the scale diversity of various target domains in real-world field scenarios, thereby addressing the problem of scale domain differences in fruit foreground targets.

[0014] (2) Existing technologies for foreground-background feature learning in images: Research has been conducted on how to avoid interference from background features and improve the ability of detection models to extract foreground features. Existing methods, for vehicle and pedestrian detection tasks under different street scenes, extract domain-invariant features between training data and actual application data by adding a foreground-background feature classifier during the training process of the detection model, thereby improving the model's ability to extract foreground target features under different backgrounds. However, such methods require prior knowledge of the foreground-background data distribution under the application scenario to train the detection model, which is not convenient for application in actual engineering. In addition, in the design of general detection models, existing methods adjust the model's learning ability for foreground-background features by formulating a positive and negative sample label allocation strategy during model training. For example, the PAA method is proposed, which uses the confidence score of the classification branch and the IoU (Intersection over Union) between the anchor box and the artificial label box in the regression branch to calculate the score of the anchor box, and establishes a probability distribution function based on the scores of all anchor boxes to determine positive and negative samples, avoiding the problem of insufficient foreground sample feature learning in complex backgrounds caused by methods that only determine positive and negative samples based on IoU values. Furthermore, the ATSS method is proposed, which can automatically assign positive and negative sample labels adaptively to the dataset, enabling the detection model to fully learn from both foreground and background features in the training data. Existing technologies also propose the OTA method, which uses a "supply-demand" relationship for global pairing, allowing each manual label box to obtain the optimal number of positive and negative samples for training from a global perspective, thereby constructing an optimal learning path for the model to learn from foreground and background features.

[0015] Therefore, by formulating a positive and negative sample label allocation strategy in the detection model, the model's attention to the foreground and background and its feature extraction ability can be adjusted. However, the current focus of related technologies is on how to balance the positive and negative sample allocation so that the detection model can fully learn both foreground and background features. This makes it difficult for the model to focus more on the extraction of foreground features and reduce the influence of background features. Therefore, how to design a positive and negative sample allocation strategy for the detection model to enhance the model's ability to extract foreground features and thus reduce the feature interference between different backgrounds between training data and application data provides a possibility for solving the background domain difference problem. This is the second technical problem that this invention aims to solve. Summary of the Invention

[0016] To address the problems existing in the prior art, the first aspect of this invention provides an anchor-free domain-adaptive target detection model, and the second aspect provides a target detection method. The third aspect provides a target detection system. The fourth aspect provides an electronic device including a processor and a memory, the memory storing a plurality of instructions, and the processor being configured to read the instructions and execute the method as described in the second aspect. The fifth aspect provides a computer-readable storage medium storing a plurality of instructions that can be read by a processor and executed as described in the second aspect.

[0017] The target detection method, system, electronic device, and computer-readable storage medium based on the anchor-free domain adaptation target detection model provided by this invention have the following beneficial technical effects: (1) It solves the domain difference problem between training data and application data faced by the target detection model in the EasyDAM fruit automatic labeling method. (2) In response to the foreground scale domain difference problem of target scale feature mismatch between training data and application data, this invention proposes a foreground domain adaptation structure based on dual prediction layers: an anchorless prediction method for generating detection boxes based on center point positioning of targets, and further designs dual prediction layers with different resolutions to detect targets of different scales respectively, realizing the adaptive generation of bounding boxes of targets of various target domains under domain differences and multi-scale detection of targets of different sizes, thereby effectively overcoming foreground target domain differences and further improving the accuracy of pseudo-label generation. (3) To address the background domain difference between training data and application data, which are in different target scenarios, this invention proposes a background domain adaptation strategy based on sample allocation. By designing a positive and negative sample allocation strategy in the detection network, the network's ability to learn foreground feature information is enhanced, interference from different background information is avoided, and the domain difference problem of inconsistent background scenarios between the source domain target and the target domain target is overcome, thereby further improving the label generation accuracy. By using an improved heatmap-based positive sample point expansion strategy and a continuous label value loss function design to guide the network training process, the model can enhance its feature extraction ability for foreground targets and avoid interference from different background domains. Attached Figure Description

[0018] Figure 1 This is a schematic diagram illustrating the domain difference problem in the existing EasyDAM method.

[0019] Figure 2 This is a flowchart of the target detection method of the present invention.

[0020] Figure 3This is a flowchart illustrating how the DomAda-FruitDet-based domain-adaptive fruit detection network improves the accuracy of the EasyDAM automatic fruit labeling method, as described in this invention.

[0021] Figure 4 This is a schematic diagram comparing the original network neck and the improved network neck described in this invention. Figure 4 (a) indicates the neck of the original CenterNet network; Figure 4 (b) represents the improved design of the foreground domain adaptation structure based on a dual prediction layer.

[0022] Figure 5 This is a schematic diagram illustrating the dimensional characteristics of citrus fruits in the source domain and the target domain, as described in this invention. Figure 5 (a) is a schematic diagram of the citrus scale characteristics of the source region; Figure 5 (b) and Figure 5 (c) is a schematic diagram of the dragon fruit scale characteristics of the target domain; Figure 5 (d) is a schematic diagram of the apple-scale characteristics of the target domain.

[0023] Figure 6 This is a schematic diagram illustrating the principle of the prediction and supervision method for learning the foreground and background features of the target using center point heatmaps in the existing CenterNet technology; wherein... Figure 6 (a) is a schematic diagram of the input fruit image; Figure 6 (b) is Figure 6 (a) True value heatmap H of the target center point after encoding; Figure 6 (c) is a schematic diagram of the Gaussian kernel encoded region of a fruit target in the image.

[0024] Figure 7 This is a schematic diagram comparing the original CenterNet positive and negative sample label allocation strategy and the improved method described in this invention.

[0025] Figure 8 This diagram illustrates the comparison of the supervision effects between the original loss function (improved Focal loss) and the improved continuous label value loss function described in this invention.

[0026] Figure 9 This is a diagram of the target detection system architecture based on the anchor-free domain adaptive target detection model described in this invention.

[0027] Figure 10 This is a schematic diagram of the electronic device structure described in this invention. Detailed Implementation

[0028] Example 1

[0029] This embodiment provides an anchor-free domain-adaptive target detection model, including: a foreground domain adaptation structure based on dual prediction layers and a background domain adaptation structure based on sample allocation; wherein: the foreground domain adaptation structure based on dual prediction layers is constructed based on anchor-free detection methods and multi-scale detection, used to extract the scale features of targets in the image and adaptively generate detection boxes to overcome foreground domain differences, the dual prediction layers include a deep prediction layer and a shallow prediction layer; the background domain adaptation structure based on sample allocation adjusts the positive and negative sample allocation in the model to overcome background domain differences through a background domain adaptation strategy based on sample allocation, improves the model's focus on foreground target features, and reduces background feature interference to overcome background domain differences; the anchor-free detection method is executed by an anchor-free detector, including locating the target based on the center point to generate a detection box for each target.

[0030] Example 2

[0031] like Figure 2As shown, this embodiment provides a target detection method implemented using the anchor-free domain adaptation target detection model of Embodiment 1, including: S1, constructing an anchor-free (anchor-free) domain adaptation target detection model; the domain adaptation target detection model includes a foreground domain adaptation structure based on dual prediction layers and a background domain adaptation structure based on sample allocation; wherein, the foreground domain adaptation structure based on dual prediction layers is constructed based on the anchor-free detection method and multi-scale detection, used to extract the scale features of the target in the image and adaptively generate detection boxes to overcome foreground domain differences, the anchor-free detection method is executed by the anchor-free detector, including locating the target based on the center point to generate a detection box for each target, the dual prediction layers include a deep prediction layer and a shallow prediction layer; the background domain adaptation structure based on sample allocation adjusts the model through a background domain adaptation strategy based on sample allocation. The model allocates positive and negative samples to bridge background domain differences, enhancing the model's focus on foreground target features and reducing background feature interference. S2, the labeled target domain simulated target dataset is used as the training input for the domain-adaptive target detection model, resulting in a pre-trained model for detecting unlabeled target domain real target images. The labeled target domain simulated target dataset and the unlabeled target domain real target images exhibit domain differences. These domain differences include a first domain difference and a second domain difference, where the first domain difference represents the difference between the source domain target label and the target domain target foreground, and the second domain difference represents the difference between the source domain target background and the target domain target background. S3, based on inputting the unlabeled target domain real target images into the pre-trained domain-adaptive fruit detection model, target detection results are obtained. S4, the unlabeled target domain real target images and the target detection results are processed to generate and output a labeled target domain real target dataset. In this embodiment, the processing includes adding pseudo-labels to the unlabeled target domain real fruit images after target detection result data conversion. Of course, those skilled in the art can perform other appropriate label data processing, all within the scope of this invention.

[0032] In a preferred embodiment, the anchor-free detection method includes locating the target based on a center point to generate a detection box for each target.

[0033] This embodiment is applied to the automatic fruit labeling scenario, thus proposing an anchor-free domain-adaptive fruit detection network for the fruit detection model design module in the EasyDAM automatic fruit labeling method. The overall solution process is as follows: Figure 3 As shown, Figure 3In this process, the labeled target domain simulated fruit dataset is taken as input, and the obtained labeled target domain real fruit dataset is taken as output as the final output result, where fruit is used as the target definition.

[0034] To address the domain discrepancy issue in fruit label generation, this embodiment uses the anchor-free target detection model CenterNet as the baseline model to construct a domain-adaptive fruit detection network, and applies it to the EasyDAM platform, such as... Figure 3 The section on anchor-free domain-adaptive fruit detection network design in this embodiment names the detection model DomAda-FruitDet. Through DomAda-FruitDet's domain-adaptive fruit detection capabilities, high-precision label data generation of fruit images in the target domain can be achieved even when domain differences exist.

[0035] DomAda-FruitDet comprises two main architectural components: (1) To address the foreground domain discrepancy problem, this invention designs a foreground domain adaptation structure based on a dual prediction layer (e.g., ...). Figure 3 As shown in ①), the method of locating the fruit by center point and detecting targets of different scales by different depth prediction layers can effectively adapt to the scale of the fruit target to generate the detection box; (2) In view of the background domain difference problem, the present invention further designs a background domain adaptation structure based on sample allocation corresponding to the background domain adaptation strategy (such as Figure 3 As shown in Figure ②, by enhancing the model's ability to extract foreground target features, the influence of different background scenes on fruit detection can be effectively avoided.

[0036] In a preferred embodiment, the foreground domain adaptation structure based on dual prediction layers is used to solve the foreground domain difference problem in the EasyDAM automatic fruit annotation method. Establishing the foreground domain adaptation structure based on dual prediction layers includes:

[0037] (1) Establish the CenterNet baseline model;

[0038] (2) To improve the network neck of the CenterNet baseline model for the target scale characteristics, a first adaptive structure is obtained, including: modifying the input feature map of the CenterNet baseline model network neck; adding the deep prediction layer and the shallow prediction layer to the predicted feature map output by the CenterNet baseline model network neck to form an improved internal feature fusion structure and neck prediction layer output, thereby using two corresponding predicted feature maps output by the deep prediction layer and the shallow prediction layer as the first adaptive structure. The first adaptive structure is used for the detection of targets at different scales. The original CenterNet, by utilizing the detection principle based on the center key point, can make the generation of detection boxes to a certain extent unaffected by the scale information of the training data, but its network neck only has a single prediction layer output, making it difficult to accurately detect targets at different scales and unable to solve the problem of foreground domain differences. Therefore, in order to achieve foreground scale domain transfer, this embodiment improves the network neck of the original model based on the fruit target scale characteristics under domain differences and designs a foreground domain adaptive structure based on a dual prediction layer. This structure mainly includes two parts: a dual prediction layer design for fruit scale characteristics and a pruning design for fruit scale feature generalization. Original network neck vs. improved network neck, for example Figure 4 (a) and Figure 4 As shown in (b). Wherein, Figure 4 (a) represents a predicted feature map output through the neck of the CenterNet network, where the predicted feature map is used for the detection of targets at various scales; Figure 4 (b) represents the improved design of the foreground domain adaptation structure based on a dual prediction layer, which outputs two prediction feature maps to specifically detect fruit targets of different sizes. Figure 4 (a) and Figure 4 The numbers in (b) represent the downsampling factor of the feature map, which are used to improve three parts to construct the foreground adaptation structure: the feature map of the network neck input, the feature fusion structure inside the neck, and the output of the neck prediction layer.

[0039] (3) The first adaptation structure is pruned based on target scale feature generalization to form a pruned network neck structure, and the foreground domain adaptation structure based on the pruned network neck structure is formed. The pruning based on target scale feature generalization includes: pruning the shallow to deep fusion part, retaining the iterative aggregation structure of the deep to shallow fusion part, constructing the network neck and the prediction feature map output by the network neck, thereby forming the foreground domain adaptation structure based on the dual prediction layer.

[0040] In this embodiment, in the application scenario of fruit labeling and detection, establishing the foreground domain adaptation structure based on the dual prediction layer includes:

[0041] (1) Establish the CenterNet baseline model;

[0042] (2) Design of a dual prediction layer based on the CenterNet baseline model to target fruit scale characteristics:

[0043] Analysis reveals that, based on the dataset used and the relevant target scale definition standards, fruits from different target domains exhibit varying scale characteristics in the images. The dataset comprising images of fruits from all target domains shows a wide range of fruit scales, with significant differences compared to the source domain fruits. Furthermore, some images contain extremely small target fruits (pixel area <32x32 pixels or the square root of their relative area percentage <3%). The limited availability of features for these small fruits is a key factor affecting detection accuracy. Specific examples include... Figure 5 (a)- Figure 5 As shown in (d).

[0044] Based on the comprehensive analysis of the relationship between the depth of the network convolutional layers, the resolution of the output feature maps, and the scale of the target to be detected, it can be seen that the receptive field scale of the feature maps output by shallow networks is smaller, focusing on the color and texture details of the image, and is more suitable for detecting smaller targets; while the receptive field scale of the feature maps output by deep networks is larger and contains rich semantic features, and is more suitable for detecting larger targets. Therefore, in order to build the model's multi-scale detection capability and achieve good detection results for small targets such as fruits, this invention, based on the above analysis and principles, modifies the input feature map of the network neck and designs two prediction layers in the predicted feature map output by the network neck for detecting fruit targets of different scales, as described in detail below:

[0045] A. Modify the input feature map of the CenterNet baseline model's neck region, including: adding feature maps from a shallower layer of the backbone network to the feature map input to the neck region, and deleting the feature map from the deepest layer. This ensures that while fully preserving the fruit's detailed information, the predicted feature map output by the neck region does not suffer feature loss due to upsampling operations. Figure 4 The comparison of “input” in (a) and (b) is shown.

[0046] B. In the prediction layer, the predicted feature map is output to the detection head. Two different resolution predicted feature maps are used to perform targeted detection of fruit targets of different sizes, including: using... Figure 4(b) The prediction feature map output by prediction layer 1 with a 2x downsampling is used to detect smaller fruit targets. This is because the 2x downsampling prediction feature map integrates features from different depths of the backbone network. It ensures the detection capability of small targets by leveraging its high resolution and retaining rich fruit detail features, without losing the semantic information of the fruit in the deeper features. At the same time, since the semantic features of the fruit are relatively simple, it is not necessary to design a deep prediction layer to locate large-scale fruit targets. Therefore, this embodiment uses the prediction feature map output by prediction layer 2 with a 4x downsampling to detect fruit targets of other scales. This is because the 4x downsampling prediction feature map integrates features from feature maps other than the shallowest layer of the backbone network. It has a moderate resolution and avoids interference from too many detail features in the shallowest layer, thus ensuring the detection capability of fruit targets of other scales except for small targets.

[0047] Ultimately, based on the characteristics of fruit scale distribution applicable to practical applications, targeted detection of fruit targets of different sizes can be performed using two predicted feature maps with different resolutions, which can effectively solve the problem of scale domain differences in fruit foreground targets.

[0048] Figure 5 The diagram shows the fruit-scale characteristics of the source domain and the target domain citrus fruit. Figure 5 (a) is a schematic diagram of the scale characteristics of citrus fruits in the source region, where the target scale of each citrus fruit is similar; Figure 5 (b) and Figure 5 (c) is a schematic diagram of the scale characteristics of dragon fruit in the target domain. Different scenes and shooting methods will result in significant differences in the scale of the same fruit in the image, and compared to... Figure 5 (a) and Figure 5 (d) Different types of fruits will have significantly different scales in the image; Figure 5 (d) is a schematic diagram of the target domain apple scale characteristics, as shown in the circle, where there are some extremely small target apple fruits.

[0049] (3) Pruning design based on fruit-scale characteristics:

[0050] When the network neck is responsible for feature fusion, excessive fusion of deep features is a key reason for model overfitting and reduced generalization. Since the training data of the fruit detection model is derived from fruits of the same source domain with a uniform scale distribution, while the fruit scales in real-world applications are diverse, it is necessary to avoid operations that affect the model's generalization in the design of the network neck structure.

[0051] However, the original Centernet's feature fusion method for the network neck consists of two parts: deep-to-shallow fusion and shallow-to-deep fusion. The latter leads to over-fusion of deep features. Therefore, this invention prunes the shallow-to-deep fusion part, retaining the iterative aggregation structure of the deep-to-shallow fusion part, to construct the predicted feature map of the network neck and its output, such as... Figure 4 The “internal structure” is shown in (a) and (b). The predicted feature map of the original structure is output from the fusion part from shallow to deep, as shown in Equation (1).

[0052]

[0053] Where F represents the feature map in the shallow-to-deep fusion part; n represents the number of fusion times to obtain the feature map (the predicted feature map is output when n=3); m represents the feature map downsampling factor; DC represents deformable convolution operation; DU represents deformable convolution and upsampling operation; and O represents the feature map output from the deep-to-shallow fusion part.

[0054] As can be seen from Equation (1), in the process of fusing from shallow to deep and outputting the predicted feature map, each fusion operation will introduce a deeper feature map. The successive fusion will cause the proportion of information contained in the shallow feature map in the final output predicted feature map to decrease one by one. This will cause the network to focus too much on specific fruit shape and structure, while ignoring the diversity of fruit in different perspectives, sizes, and postures.

[0055] The predicted feature map output after cropping is output from the fusion part from deep to shallow, as shown in Equation (2).

[0056]

[0057] Where O represents the feature map in the deep-to-shallow fusion part; n represents the number of fusions of feature maps of the same dimension when acquiring this feature map; m represents the feature map downsampling factor (where m = 2, 4, 8, 16, when m = 2: n = 1, 2, 3; when m = 4: n = 1, 2; when m = 8: n = 1; when m = 16: n = 0. In this embodiment, the predicted feature map is preferably output when m = 2, n = 3 and m = 4, n = 2); DC represents deformable convolution operation; DU represents deformable convolution and upsampling operation.

[0058] As shown in Equation (2), after cropping the shallow-to-deep fusion part, the predicted feature map is directly output through the deep-to-shallow fusion part. Each feature fusion operation introduces a shallower feature map. The final output predicted feature map not only retains the rich fruit detail information contained in the shallow feature map, but also avoids the interference of excessive fusion of deep features. In addition, directly outputting the predicted feature map from the deep-to-shallow fusion part makes the resolution of the predicted feature map more flexible, and is not limited by the limitation that the shallow-to-deep fusion part can only be output at a 4x downsampling resolution.

[0059] By using the above method, the pruned network neck structure avoids operations that affect the model's generalization ability and maximizes the fruit feature information extracted from the backbone network to construct subsequent prediction layers, further ensuring the network's ability to detect fruits at multiple scales under domain differences.

[0060] In a preferred embodiment, the background domain adaptation structure based on sample allocation is used to solve the background domain difference problem in the EasyDAM automatic fruit annotation method. Establishing the background domain adaptation structure based on sample allocation includes:

[0061] (1) Set an improved positive and negative sample label allocation strategy in the heatmap branch; and learn the foreground and background features of the target based on the improved positive and negative sample label allocation strategy; wherein the improved positive and negative sample label allocation strategy is a positive sample point expansion strategy based on the heatmap.

[0062] (2) Set a loss function corresponding to the positive sample point expansion strategy based on the heatmap, wherein the loss function is a continuous label value loss function.

[0063] Foreground-invariant features refer to the common features of foreground targets across different background domains. Encouraging the model to focus more on extracting foreground target features is an effective way to achieve cross-background domain detection. However, the original CenterNet training method suffers from a severe imbalance in positive and negative sample label allocation; the number of positive samples used to learn foreground features is far less than the number of negative samples used to learn background features. This results in insufficient ability of the model to extract foreground target features and an inability to address the background domain differences. This invention analyzes the center point heatmap used by the original CenterNet for learning foreground-background features for target localization and develops an improved sample allocation strategy accordingly.

[0064] The original CenterNet used center point heatmaps to learn the foreground and background features of the target. Its prediction and supervision methods are as follows: Let the input fruit image be I∈R. W×H (like Figure 6(a), W and H represent the width and height of the image, respectively. The Heatmap branch will generate a predicted heatmap of the target center point of I. Meanwhile, by encoding each fruit target in I using the Gaussian kernel generation formula, a true heatmap of the target center point is generated. (like Figure 6 (b)) used for Supervision will be conducted. Among these,

[0065] The Gaussian kernel generation formula is shown in formula (3):

[0066]

[0067] Where m represents the index of the m-th target label box in I; K m (x,y) represents the Gaussian encoded value at pixel position (x,y) on the actual heatmap H; and This represents scaling I to the feature map scale. At time σ, the x and y coordinates of the center point of the m-th label box; p This is the offset parameter.

[0068] In H, the pixel corresponding to the center point of the fruit target has a value of 1, and the value of the corresponding pixel decreases in a Gaussian distribution as it gets farther away from the center point (e.g., ...). Figure 6 (c) The encoded values ​​of pixels with values ​​less than a set threshold and the corresponding pixels in the background are 0, thus distributing the Gaussian non-zero encoded values ​​near the pixel corresponding to the center of the fruit. Therefore, it can be seen that H pairs in the Heatmap prediction branch... The supervision method reflects how the model learns foreground target features and image background features, which in turn determines the model's ability to learn foreground-background features. This supervision method is implemented by formulating the positive and negative sample label assignment strategy of the Heatmap branch.

[0069] This invention improves the positive and negative sample label allocation strategy in the Heatmap branch and designs a corresponding loss function (modified from the improved Focal loss function to a continuous label value loss function). This includes two parts: a heatmap-based positive sample point expansion strategy and the design of the continuous label value loss function. A comparison between the original strategy and the improved method is provided below. Figure 7 As shown. Figure 7 A comparison of the original CenterNet positive and negative sample label assignment strategy with the improved method.

[0070] As a preferred embodiment, the heatmap-based positive sample point augmentation strategy includes:

[0071] Points with non-zero values ​​on the real heatmap in the center point heatmap branch that correspond to the same pixel position on the predicted heatmap are considered as positive sample points of the heatmap branch. They are supervised by positive sample labels and used to learn the foreground features of the target.

[0072] Points with a value of 0 on the actual heatmap in the center point heatmap branch, corresponding to sample points at the same pixel position on the predicted heatmap, are considered as negative sample points of the heatmap branch. They are supervised by negative sample labels and used to learn the background features of the image where the target is located.

[0073] In this embodiment, by Figure 7 As shown in (a), the original CenterNet encodes the center point heatmap H on the training image, where H... xy The point (x,y)∈H with a value of 1 (a sample point on the true branch heatmap H has a value of 1) corresponds to the branch prediction heatmap. pixel position (Branch Prediction Heatmap) The value of a certain sample point on the [database name] is [value]. Points with a value of 1 are considered positive samples in this branch and are supervised by positive samples with a label value of 1, used to learn foreground features; while the remaining points are supervised by negative samples with a label value of 0, used to learn background features in the image. Wherein, H... xy Points with a value of 1 (a sample point on the branch-based heatmap H) are encoded solely by the center point of each fruit target bounding box. However, the number of fruits in the image is much lower than the total number of pixels in the feature map. Therefore, in H... xy The number of pixels ≠1 (where the value of a sample point on the true heatmap H is not 1) is far greater than H. xy =1 (the value of a sample point on the branch's true heatmap H is 1). Therefore, the original CenterNet model has a serious problem of imbalance between positive and negative sample labels. The number of positive sample points used to learn foreground object features is extremely small, and it cannot focus on learning the foreground object to solve the background domain difference problem.

[0074] Therefore, to address the problem of insufficient foreground feature learning ability caused by the imbalance in the original positive and negative sample allocation, this invention proposes a positive sample point expansion strategy based on heatmaps. Figure 7 As shown in (b), the specific description is as follows: H on H xy ≠0 corresponds to pixel position All are considered positive sample points in the Heatmap branch, supervised by positive sample labels, and used to learn the foreground features of the fruit target; while H on H xy =0 corresponds to pixel position All points are considered negative samples in the Heatmap branch, supervised by negative sample labels, and used to learn image background features. The improved policy algorithm is shown in Algorithm 1.

[0075]

[0076]

[0077] In the original positive and negative sample allocation, each fruit had only one positive sample point. Using the positive and negative sample label allocation method in Algorithm 1, the number of positive sample points for each fruit becomes the number of all non-zero points in the Gaussian distribution region encoded based on the fruit's center point. This number is determined by the fruit's size and the Gaussian threshold during encoding, and is often greater than one. This method significantly expands the number of positive samples, providing the model with ample training samples of foreground targets, thereby improving the detection model's ability to extract foreground target features.

[0078] As a preferred embodiment, the continuous label value loss function is shown in formula (4):

[0079]

[0080] Among them, L H The calculated loss value is N, where N is the total number of pixels in the feature map; Predicting heatmaps for branches The value of a sample point on H xy This represents the pixel value at the corresponding location in the actual heatmap H; the default setting is β = 2.

[0081] After establishing the rules for assigning positive and negative sample labels, it is necessary to use the corresponding loss function to supervise the training of positive and negative samples. The original CenterNet uses the improved Focal loss function to supervise the Heatmap branch, as shown in Equation (5).

[0082]

[0083] From equation (5), where L H The calculated loss value is given by N, where N is the total number of pixels in the feature map. Predict heatmaps for the Heatmap branch The value of a sample point on H xyThis represents the pixel value corresponding to the actual heatmap H, with default settings of α=2 and β=4. It's evident that the supervision label values ​​in the original CenterNet heatmap branch loss function are discrete; that is, label value 1 is used to supervise positive samples, and label value 0 is used to supervise negative samples. This supervision value setting is not suitable for the improved design's positive and negative sample label assignment strategy. Specifically, if the original loss function is used, since all non-zero values ​​on the center point heatmap H correspond to heatmap branches... All pixel positions are considered positive sample points, and all newly added positive sample points are supervised by the label value 1. Therefore, the positive sample points at other positions outside the center point will be indistinguishable from the center point, and the effectiveness of the supervision for model training will be the same. This will cause the model to have difficulty locating the exact position of the center point. In order to solve the above problem, this invention uses a loss function that makes the label value continuous for the improved positive and negative sample label allocation strategy, which is called the continuous label value loss function, as shown in equation (4) above.

[0084] The label values ​​for the continuous label loss function are no longer limited to the positive sample supervision label value of 1 and the negative sample supervision label value of 0; any decimal between 0 and 1 can be used as the label, and H is used as the label. xy direct as The label values ​​of the sample points (x, y). To intuitively represent the supervision effect of the loss function, the original loss function (Improved Focal Loss) and the improved loss function (Continuous label loss) are compared at different H... xy The loss value of the predicted value at the value is Figure 8 It will be displayed in the middle. Figure 8 This is a comparison chart of the supervision effects of the original loss function (improved Focal loss) and the improved continuous label loss function. The original loss function can only use 0 or 1 as labels for supervision, while the improved loss function can use any continuous value between 0 and 1 as labels for supervision.

[0085] Figure 8 In each unit coordinate system, the horizontal coordinate is Predicted value of the upper pixel (x,y) The vertical axis represents the pixel point on the central heatmap H corresponding to that point. xy These are the loss values ​​obtained for different values ​​(1, 0.8, 0.5, 0.2, and 0 are shown in the figure). Different H values... xy The analysis under the value is as follows:

[0086] ①When H xy =1 (or H) xyWhen (x,y) = 0, (x,y) is the sample point corresponding to the target center point (or background). The loss value calculated by Continuous label loss is consistent with the original improved Focal Loss, that is, (x,y) is regarded as a positive sample point (or negative sample point) for supervision.

[0087] ②When H xy =0.8, H xy =0.5 and H xy When = 0.2, (x, y) are the sample points corresponding to the positions relatively close, moderate, and far from the target center point, respectively. Since the original improved FocalLoss uses discrete label values, although the sample points closer to the center point are subject to weaker supervision by the 0 value, it still uses the negative sample label value 0 to supervise (x, y), and cannot achieve the expansion of positive sample points; at the same time, if the piecewise condition of equation (3) is changed so that (x, y) are all subject to H in equation (3), xy Under the condition of H=1, the supervised loss function cannot distinguish between newly added positive sample points and the center positive sample points. The improved continuous label value loss function, however, uses continuous label values, which not only allows for the expansion of positive samples by treating (x,y) as positive sample points, but also allows for direct differentiation using different H values. xy The value is used to supervise the newly added positive sample points. Meanwhile, due to H... xy The values ​​follow a Gaussian distribution encoded by the target foreground. Therefore, the positive sample points corresponding to the target center point will be subject to stricter positive sample supervision than the positive sample points near it, while the supervision intensity of the positive samples near the center point gradually decreases according to the Gaussian distribution as the distance from the center point increases.

[0088] Therefore, by using a continuous label loss function to improve the positive and negative sample allocation strategy, the positive samples at the center point can be distinguished from other positive samples, so that positive sample points at different locations have different influence on model updates, thereby making the supervision settings of newly added positive samples more reasonable.

[0089] The target detection method proposed in this embodiment is applied to the automatic labeling of fruits. It is implemented based on the anchor-free domain-adapted fruit detection network DomAda-FruitDet, and in a specific implementation, it is applied to the automatic fruit labeling method to replace the original fruit detection network, thereby further improving the label generation accuracy. Of course, those skilled in the art should know that the domain-adapted target detection model used can also be applied to other scenarios for target detection, all of which are within the protection scope of this invention.

[0090] DomAda-FruitDet primarily comprises two design aspects: ① It utilizes a foreground domain adaptation structure based on a dual prediction layer, employing an anchorless detection method and multi-scale detection capabilities to extract target fruit scale features from images and adaptively generate label boxes to overcome foreground domain differences; ② It employs a background domain adaptation strategy based on sample allocation, utilizing adjustments to the positive and negative sample allocation within the model to enhance the model's focus on foreground target fruit features and reduce background feature interference to overcome background domain differences. These two design aspects effectively address the domain difference problem faced by fruit detection networks in automatic fruit labeling methods and obtain high-precision fruit label data. In network effectiveness verification, this invention, based on a set of labeled source domain citrus fruit data, achieves automatic labeling of target domain real apple, tomato, dragon fruit, and mango datasets. The final tested mAP accuracy values ​​of the fruit detection model reached 90.9%, 90.8%, 88.3%, and 94.0%, respectively, verifying the effectiveness of the proposed network. Therefore, the network used in the method and system of the present invention can solve the problem of domain differences in data in different domains, effectively improve the accuracy of target label generation, and provide more accurate and effective labeling data for target automatic labeling tasks.

[0091] Example 3

[0092] like Figure 9 As shown, this embodiment provides a target detection system, including: a detection model construction module 101, used to construct an anchor-free domain-adaptive target detection model; a pre-trained model acquisition module 102, used to take a labeled target domain simulation target dataset as the training input of the domain-adaptive target detection model to obtain a pre-trained domain-adaptive target detection model for detecting unlabeled target domain real target images; the labeled target domain simulation target dataset and the unlabeled target domain real target images have domain differences; wherein, the domain differences include first domain differences and second domain differences; a target detection result acquisition module 103, used to obtain target detection results based on inputting the unlabeled target domain real target images into the pre-trained domain-adaptive target detection model; and a target label acquisition module 104, used to process the unlabeled target domain real target images and the target detection results to generate and output a labeled target domain real target dataset.

[0093] The present invention also provides a memory that stores a plurality of instructions for implementing the method as described in Embodiment 1.

[0094] like Figure 10As shown, the present invention also provides an electronic device, including a processor 301 and a memory 302 connected to the processor 301. The memory 302 stores a plurality of instructions, which can be loaded and executed by the processor to enable the processor to perform the method as described in Embodiment 1.

[0095] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention. Clearly, those skilled in the art can make various alterations and modifications to the invention without departing from its spirit and scope. Thus, if these modifications and modifications of the invention fall within the scope of the claims and their equivalents, the invention is also intended to include these modifications and modifications.

Claims

1. An anchor-free based domain adaptation object detection apparatus, characterized in that, Comprise: A foreground domain adaptation structure based on double prediction layers and a background domain adaptation structure based on sample allocation; wherein: The foreground domain adaptation structure based on double prediction layers is constructed based on an anchor-free detection method and a multi-scale detection, and is used for extracting scale features of targets in an image and adaptively generating detection boxes to span foreground domain differences, the double prediction layers comprising a deep prediction layer and a shallow prediction layer; the anchor-free detection method is performed by an anchor-free detector, and comprises positioning a target based on a center point to generate a detection box for each target; the foreground domain adaptation structure based on double prediction layers comprises: (1) establishing a CenterNet baseline model; (2) modifying a network neck of the CenterNet baseline model to obtain a first adaptation structure according to scale characteristics of the target, comprising: modifying input feature maps of the network neck of the CenterNet baseline model, comprising: adding a feature map from a shallower layer of a backbone network to the network neck input feature maps of the neck part, and deleting a deepest layer feature map; adding the deep prediction layer and the shallow prediction layer to the prediction feature maps output by the network neck of the CenterNet baseline model to form an improved neck internal feature fusion structure and a neck prediction layer output, so as to output two corresponding prediction feature maps of the deep prediction layer and the shallow prediction layer as the first adaptation structure, which is used for detecting targets of different scales; (3) implementing a cutting based on target scale feature generalization on the first adaptation structure to form a cut network neck structure, and forming the foreground domain adaptation structure based on double prediction layers based on the cut network neck structure, wherein the cutting based on target scale feature generalization comprises: cutting a shallow-to-deep fusion part, retaining an iterative aggregation structure of a deep-to-shallow fusion part, constructing a network neck and a prediction feature map output by the network neck, so as to form the foreground domain adaptation structure based on double prediction layers; The background domain adaptation structure based on sample allocation adjusts positive and negative sample allocation in the device to span background domain differences through a background domain adaptation strategy based on sample allocation; The background domain adaptation structure based on sample allocation comprises: (1) setting an improved positive and negative sample label allocation strategy in the improved heat map branch, and learning pre-background features of the target based on the improved positive and negative sample label allocation strategy; wherein the improved positive and negative sample label allocation strategy is a positive sample point expansion strategy based on a heat map; (2) setting a loss function corresponding to the positive sample point expansion strategy based on the heat map, the loss function being a continuous label value loss function.

2. A target detection method, implemented based on the anchor-free based domain adaptation target detection device of claim 1. Comprise: S1, constructing an anchor-free domain adaptation target detection model; S2, inputting the labeled target domain simulation target data set as the domain adaptation target detection model training input, obtaining a pre-trained domain adaptation target detection model for detecting unlabeled target domain real target images; the labeled target domain simulation target data set and the unlabeled target domain real target images have domain differences; wherein the domain differences include first domain differences and second domain differences; the first domain differences represent the differences between source domain target labels and target domain target foregrounds, and the second domain differences represent the differences between source domain target backgrounds and target domain target backgrounds; S3, obtaining a target detection result based on inputting the unlabeled target domain real target image into the pre-trained domain adaptation target detection model; S4, processing the unlabeled target domain real target image and the target detection result to generate and output a labeled target domain real target data set.

3. The target detection method of claim 2, wherein the two corresponding predicted feature maps are two predicted feature maps of different resolutions, and the detection head part outputs the predicted feature maps of different resolutions to detect fruit targets of different sizes respectively; wherein the two predicted feature maps of different resolutions are a 2x down-sampled predicted feature map for detecting smaller targets and a 4x down-sampled predicted feature map for detecting other scale targets. The cropping of the target scale feature generalization-based cropping forms a cropped network neck structure, which includes the output of the predicted feature map after cropping from the deep-to-shallow fusion part, and the process is as shown in formula (2):

4. The object detection method of claim 3, wherein, Wherein, O represents the feature map in the deep-to-shallow fusion part; n represents the fusion times of the feature map of the same dimension when the feature map is obtained; m represents the down-sampling multiple of the feature map; DC represents the deformable convolution operation; and DU represents the deformable convolution and up-sampling operation. The continuous label value loss function is as shown in formula (4):

5. The object detection method of claim 4, wherein, The positive sample point expansion strategy based on the heat map includes: wherein L H is a calculated loss value, N is the total number of pixel points on the feature map; is the value of a sample point on the branch prediction heat map H xy is the value of the pixel point at the corresponding position of the real heat map H; the default setting is β = 2.

6. The object detection method of claim 5, wherein, The sample points corresponding to the points with a value of 0 on the real heat map in the center point heat map branch to the same pixel position of the predicted heat map are regarded as positive sample points of the heat map branch, which are supervised by the positive sample labels and used to learn the foreground features of the target; The sample points corresponding to the points with a value of 0 on the real heat map in the center point heat map branch to the same pixel position of the predicted heat map are regarded as negative sample points of the heat map branch, which are supervised by the negative sample labels and used to learn the background features of the image where the target is located. It includes:

7. A target detection system for implementing the target detection method according to any one of claims 2 to 6, characterized in that A detection model construction module (101) for constructing an anchor-free domain adaptation target detection model; A pre-trained model acquisition module (102) for inputting a labeled target domain simulation target data set as the domain adaptation target detection model training input, obtaining a pre-trained domain adaptation target detection model for detecting unlabeled target domain real target images; the labeled target domain simulation target data set and the unlabeled target domain real target images have domain differences; wherein the domain differences include first domain differences and second domain differences; ​ The target detection result acquisition module (103) is configured to acquire a target detection result based on inputting the unlabeled target domain real target image into the pre-trained domain adaptive target detection model. The target label acquisition module (104) is configured to process the unlabeled target domain real target image and the target detection result, and generate and output a labeled target domain real target data set.

8. An electronic device, comprising a processor and a memory, the memory storing a plurality of instructions, wherein, The processor is configured to read the instructions and perform the target detection method according to any one of claims 2-6. 9.A computer readable storage medium, storing a plurality of instructions, wherein, The plurality of instructions can be read by the processor and executed to perform the target detection method according to any one of claims 2-6.

Images