Image detection model training method, device and image detection method

Abstract

An embodiment of the present invention provides an image detection model training method, device and image detection method, wherein the image detection model training includes: extracting features from a sample image through a feature extraction network to obtain a first feature map; inputting the first feature map into an image-level domain classifier, and updating the first feature map according to the attention weights of each region in the first feature map to obtain a second feature map; inputting the second feature map into an instance detection network to obtain the location area and category classification results of each instance; generating a third feature map based on the second feature map; inputting the third feature map into a target instance-level domain classifier corresponding to the target instance category label to obtain a domain category prediction result output by the target instance-level domain classifier; training the detection network according to the domain category prediction result, the category classification result and the regression parameter to obtain a target detection model for performing instance detection on the image to be detected in the target domain.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to an image detection model training method, device and image detection method. Background Art

[0002] The crown is a crucial component of trees. Accurately identifying it can play a positive role in monitoring tree growth, preventing pests and diseases, and predicting tree biomass and small-lot volume. Furthermore, the crown is a crucial factor in building forest stand models. Identifying crown width can effectively predict stand density and estimate tree competition.

[0003] At present, deep learning methods are commonly used to detect tree crowns. However, current deep learning methods have high requirements on the shooting conditions of training samples and test samples, and are only applicable to tree crown detection in specific areas. For sample images from different regions or different shooting sources, the detection effect is poor. Summary of the Invention

[0004] The embodiments of the present invention provide an image detection model training method, device and image detection method to improve the detection effect of the image detection model.

[0005] In a first aspect, an embodiment of the present invention provides an image detection model training method, which is applied to a detection network. The detection network includes an object detection model, an image-level domain classifier, and instance-level domain classifiers corresponding to different instance categories. The method includes:

[0006] Obtaining a sample image and the target detection model to be trained, wherein the sample image includes a source domain image and a target domain image, the source domain image has an instance category label, the target domain image does not have an instance category label, and both the source domain image and the target domain image have domain category labels, the target detection model is pre-trained using the source domain image, and the target detection model includes a feature extraction network and an instance detection network;

[0007] Performing feature extraction on the sample image through the feature extraction network to obtain a first feature map;

[0008] Inputting the first feature map into the image-level domain classifier to obtain a domain category prediction result output by the image-level domain classifier, determining an attention weight for each region in the first feature map based on the domain category prediction result, and updating the first feature map according to the attention weight for each region in the first feature map to obtain a second feature map;

[0009] Inputting the second feature map into the instance detection network to obtain a location region and category classification result of each instance;

[0010] Determining the instance category label corresponding to the sample image according to the category classification results of each instance and the domain category label corresponding to the sample image;

[0011] generating a third feature map based on the second feature map, wherein the third feature map is marked with a location area of ​​each instance corresponding to a target instance category label, and the target instance category label is any one of the instance category labels corresponding to the sample image;

[0012] Inputting the third feature map into a target instance-level domain classifier corresponding to the target instance category label to obtain a domain category prediction result output by the target instance-level domain classifier;

[0013] The detection network is trained based on the domain category prediction results output by the image-level domain classifier, the domain category prediction results output by the target instance-level domain classifier, and the category classification results and regression parameters of each instance output by the instance detection network to obtain a target detection model for performing instance detection on the image to be detected in the target domain.

[0014] In a second aspect, an embodiment of the present invention provides an image detection model training device, which is applied to a detection network. The detection network includes an object detection model, an image-level domain classifier, and instance-level domain classifiers corresponding to different instance categories. The device includes:

[0015] an acquisition module, configured to acquire a sample image and the target detection model to be trained, wherein the sample image includes a source domain image and a target domain image, the source domain image has an instance category label, the target domain image does not have an instance category label, and both the source domain image and the target domain image have domain category labels; the target detection model is pre-trained using the source domain image, and includes a feature extraction network and an instance detection network;

[0016] an extraction module, configured to extract features from the sample image using the feature extraction network to obtain a first feature map;

[0017] an updating module, configured to input the first feature map into the image-level domain classifier to obtain a domain category prediction result output by the image-level domain classifier, determine an attention weight for each region in the first feature map based on the domain category prediction result, and update the first feature map according to the attention weight for each region in the first feature map to obtain a second feature map;

[0018] a processing module, configured to input the second feature map into the instance detection network to obtain a location region and category classification result of each instance;

[0019] a determination module, configured to determine the instance category label corresponding to the sample image based on the category classification results of the instances and the domain category label corresponding to the sample image;

[0020] a generating module, configured to generate a third feature map based on the second feature map, wherein the third feature map is marked with a location area of ​​each instance corresponding to a target instance category label, and the target instance category label is any one of the instance category labels corresponding to the sample image;

[0021] The processing module is further configured to input the third feature map into a target instance-level domain classifier corresponding to the target instance category label to obtain a domain category prediction result output by the target instance-level domain classifier;

[0022] A training module is used to train the detection network based on the domain category prediction results output by the image-level domain classifier, the domain category prediction results output by the target instance-level domain classifier, and the category classification results and regression parameters of each instance output by the instance detection network, so as to obtain a target detection model for instance detection of the image to be detected in the target domain.

[0023] In a third aspect, an embodiment of the present invention provides an electronic device comprising: a memory and a processor; wherein the memory stores executable code, and when the executable code is executed by the processor, the processor executes the image detection model training method as described in the first aspect.

[0024] In a fourth aspect, an embodiment of the present invention provides a non-transitory machine-readable storage medium, on which executable code is stored. When the executable code is executed by a processor of an electronic device, the processor executes the image detection model training method described in the first aspect.

[0025] In a fifth aspect, an embodiment of the present invention provides an image detection method, comprising:

[0026] Obtain the image to be detected and the target detection model;

[0027] Detecting the model to be detected by the target detection model to obtain instance regions and category classification results corresponding to each instance in the image to be detected;

[0028] The target detection model is trained using the image detection model training method described in the first aspect.

[0029] In a sixth aspect, an embodiment of the present invention provides an image detection device, including:

[0030] The acquisition module is used to obtain the image to be detected and the target detection model;

[0031] A detection module, configured to detect the model to be detected using the target detection model to obtain instance regions and category classification results corresponding to each instance in the image to be detected;

[0032] The target detection model is trained using the image detection model training method described in the first aspect.

[0033] In the seventh aspect, an embodiment of the present invention provides an electronic device, comprising: a memory and a processor; wherein the memory stores executable code, and when the executable code is executed by the processor, the processor executes the image detection method as described in the fifth aspect.

[0034] In an eighth aspect, an embodiment of the present invention provides a non-transitory machine-readable storage medium having executable code stored thereon. When the executable code is executed by a processor of an electronic device, the processor executes the image detection method as described in the fifth aspect.

[0035] In an embodiment of the present invention, it is assumed that a detection network including a target detection model, an image-level domain classifier, and an instance-level domain classifier corresponding to different instance categories is to be trained to improve the image detection effect of the target detection network in the detection network. First, after extracting the first feature map of the sample image through the feature extraction network, the source domain image and the target domain image are aligned at the picture level by the image-level domain classifier to obtain a second feature map, thereby highlighting the transferable areas in the first feature map and suppressing the negative transfer of each area. Then, the instance-level features in the second feature map are extracted by the instance detection network to obtain a third feature map. The third feature map is aligned at the instance level by the instance-level domain classifier to avoid misalignment by using the instance category label of the instance, thereby further improving the image alignment effect. Finally, the detection network is trained based on the results output by the target detection model, the image-level domain classifier, and the instance-level domain classifier, thereby improving the image detection effect of the target detection network. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following is a brief introduction to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0037] Figure 1 A flowchart of an image detection model training method provided by an embodiment of the present invention.

[0038] Figure 2A schematic diagram of the structure of a detection network provided by an embodiment of the present invention.

[0039] Figure 3 The present invention provides a flowchart of an image detection method.

[0040] Figure 4 A schematic structural diagram of an image detection model training device provided by an embodiment of the present invention.

[0041] Figure 5 A schematic structural diagram of an electronic device provided by an embodiment of the present invention.

[0042] Figure 6 A schematic structural diagram of another electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0043] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0044] In addition, the step sequence in the following method embodiments is only an example and not a strict limitation.

[0045] The image classification method provided in the embodiments of the present invention can be performed by an electronic device, which can be a terminal device such as a PC or a laptop, or a server. The server can be a physical server or a virtual server. The server can be a physical or virtual server on the user side, or a cloud server.

[0046] The solutions provided by the embodiments of the present invention can be used to implement image classification and detection. Simply put, after acquiring different images, users (referred to herein as researchers with image classification and detection needs) can implement the solutions provided by the embodiments of the present invention to achieve the aforementioned image classification and detection, thereby improving the accuracy and efficiency of image classification.

[0047] Taking the example of tree crown detection, rapid and accurate crown detection is crucial for forestry management and precision forestry. The crown is the primary site of photosynthesis and a vital component of trees. Accurately identifying crown information can be instrumental in monitoring tree growth, preventing tree pests and diseases, and predicting tree biomass and small-lot stock. Furthermore, crown is a crucial factor in building forest stand models. Identifying crown width can effectively predict stand density and estimate tree competition. Therefore, accurate and real-time crown detection across large areas is crucial for both production and research. However, the vast spatial scale and data variability of crown detection make traditional deep learning methods difficult to apply to cross-regional crown detection scenarios.

[0048] Current deep learning methods have high requirements on the shooting conditions of training samples and test samples, and are only applicable to tree crown detection in specific areas. However, the detection effect is poor for sample images from different areas or different shooting sources.

[0049] It should be noted that the image detection model training method and image detection method provided by the present invention can be applied to tree crown detection scenarios, and can also be applied to image detection scenarios in other fields.

[0050] The training process of the image detection model training method provided by the present invention is described in detail below in conjunction with the following embodiments.

[0051] Figure 1 This is a flowchart of a method for training an image detection model provided by an embodiment of the present invention. The training method can be applied to a detection network comprising an object detection model, an image-level domain classifier, and instance-level domain classifiers corresponding to different instance categories. The method comprises the following steps:

[0052] S101, obtaining a sample image and an object detection model to be trained.

[0053] S102: Extract features from the sample image through a feature extraction network to obtain a first feature map.

[0054] S103: Input the first feature map into the image-level domain classifier to obtain the domain category prediction result output by the image-level domain classifier, determine the attention weights of each region in the first feature map based on the domain category prediction result, and update the first feature map according to the attention weights of each region in the first feature map to obtain the second feature map.

[0055] S104: Input the second feature map into the instance detection network to obtain the location area and category classification results of each instance.

[0056] S105 , determining the instance category label corresponding to the sample image according to the category classification result of each instance and the domain category label corresponding to the sample image.

[0057] S106: Generate a third feature map based on the second feature map.

[0058] S107 , inputting the third feature map into a target instance-level domain classifier corresponding to the target instance category label to obtain a domain category prediction result output by the target instance-level domain classifier.

[0059] S108, training the detection network based on the domain category prediction results output by the image-level domain classifier, the domain category prediction results output by the target instance-level domain classifier, and the category classification results and regression parameters of each instance output by the instance detection network to obtain a target detection model for instance detection of the image to be detected in the target domain.

[0060] To improve the detection performance of the object detection model in the target domain, we can gradually align the feature distributions of the source and target domains by aligning the source and target domain images at the image and instance levels. Specifically, we can use the object detection model as a baseline model and add two domain classifier structures for feature alignment, such as an image-level domain classifier and an instance-level domain classifier, to this baseline model to achieve the image-level and instance-level alignment between the source and target domain images described above.

[0061] Specifically, the number of instance-level domain classifiers can be determined based on the number of instance categories in the sample image. For example, if the sample image is a tree crown image in a tree crown detection scenario, the instance categories in the tree crown image include tree crown instance categories and background instance categories. Therefore, two corresponding instance-level domain classifiers can be provided, one corresponding to the tree crown instance category and the other corresponding to the background instance category.

[0062] First, a sample image and a target detection model to be trained are obtained. In this embodiment, the sample image includes a source domain image and a target domain image.

[0063] The source domain image and the target domain image are images containing instances of the same category and having similar shooting perspectives, image textures or backgrounds. Furthermore, the source domain image corresponds to an instance category label, while the target domain image does not have an instance category label. In addition, the source domain image and the target domain image each correspond to a domain category label, that is, the source domain image corresponds to a source domain label, and the target domain image corresponds to a target domain label. In this embodiment, both the source domain image and the target domain image contain tree crown information. Accordingly, the instance category label can include the foreground (tree crown) category label and the background category label in the image, as well as the region coordinate information corresponding to each category label.

[0064] The object detection model is pre-trained using source domain images and includes a feature extraction network and an instance detection network. The instance detection network can include a Region Proposal Network (RPN), which generates region coordinate information for each instance. In this embodiment, the object detection model is Faster-RCNN.

[0065] Then, the feature extraction network extracts features from the sample image to obtain a first feature map. After obtaining the first feature map, the first feature map is input into the image-level domain classifier to obtain the domain category prediction result output by the image-level domain classifier. Based on the domain category prediction result, the attention weights of each region in the first feature map are determined. The first feature map is updated according to the attention weights of each region in the first feature map to obtain a second feature map.

[0066] For a certain area in the source domain image or the target image, based on the domain category prediction results output by the image-level domain classifier, it can be determined whether the image-level domain classifier can successfully distinguish the area. If the image-level domain classifier can successfully distinguish the area, it indicates that the area is a representative feature of the source domain and target domain foreground images. Therefore, a larger attention weight can be given to the area, so as to achieve the effect of enhancing areas with high transferability and suppressing areas with poor transferability.

[0067] The second feature map is then fed into the instance detection network to obtain the location region and category classification results for each instance. In this embodiment, the location region of each instance can be determined using the region coordinate information generated by the region generation network. Simultaneously, the instance category label corresponding to the sample image can be determined based on the category classification results for each instance and the domain category label corresponding to the sample image.

[0068] Next, a third feature map is generated based on the second feature map. In this embodiment, the third feature map is labeled with the location regions of each instance corresponding to the target instance category label, where the target instance category label is any instance category label corresponding to the sample image. Specifically, after inputting the second feature map into the instance detection network to obtain the location regions of each instance, the obtained location regions of each instance can be mapped into the second feature map to generate the third feature map.

[0069] Since source domain images have corresponding instance category labels, while target images do not, sample images of different domain categories have corresponding instance category labels that are different. In this embodiment, if the sample image is a source domain image, the instance category label of the source domain image is directly used; if the sample image is a target domain image, the category classification results of each instance are used as the instance category label of the target domain image.

[0070] After determining the instance category label corresponding to each instance in the third feature map, the third feature map can be input into a target instance-level domain classifier corresponding to the target instance category label to obtain a domain category prediction result output by the target instance-level domain classifier.

[0071] Finally, based on the domain category prediction results output by the image-level domain classifier, the domain category prediction results output by the target instance-level domain classifier, and the category classification results of each instance output by the instance detection network, the detection network is trained to obtain a target detection model for instance detection of the target domain's images to be detected.

[0072] In another embodiment, the object detection model may also include a Strong Weak Faster-RCNN. In this embodiment, the object detection model adds two domain classifier branches attached to the feature extraction network based on Faster-RCNN to achieve image-level alignment between the source domain image and the target domain image. Simultaneously, an instance-level domain classifier is added to the fully connected layer to achieve instance-level alignment between the source domain image and the target domain image. Based on the structures described above, a detection network is constructed.

[0073] It should be noted that when the target detection model is Strong Weak Faster-RCNN, since Strong Weak Faster-RCNN contains a domain classifier branch and has the same function as the image-level domain classifier described above, the domain classifier contained in the Strong Weak Faster-RCNN is the image-level domain classifier described above.

[0074] The following is combined with Figure 2 The training process of the detection network and image detection model training method provided by the present invention is described in detail.

[0075] Figure 2 A schematic diagram of the structure of a detection network provided by an embodiment of the present invention. Figure 2 As shown, the detection network includes a target detection model, a multi-level transferable attention module, and a multi-adversarial instance-level alignment module. The multi-level transferable attention module is used to determine the attention weights of each region in the sample image based on the output of the image-level domain classifier, and weight the feature maps of the corresponding layers of the sample image based on the attention weights to obtain feature maps with attention. It should be noted that in this embodiment, the target detection model is Strong Weak Faster-RCNN.

[0076] In this embodiment, the feature extraction network of the object detection model includes a first feature extraction network and a second feature extraction network. The sampling receptive field corresponding to the first feature extraction network is smaller than the sampling receptive field corresponding to the second feature extraction network. Accordingly, the image-level domain classifier includes a first image-level domain classifier corresponding to the first feature extraction network and a second image-level domain classifier corresponding to the second feature extraction network.

[0077] The detection network also includes a first gradient reversal layer located between the feature extraction network and the image-level domain classifier, which is used to implement adversarial training of the feature extraction network and the image-level domain classifier. Accordingly, the detection network also includes a second gradient reversal layer located between the instance detection network and the target instance-level domain classifier, which is used to implement adversarial training of the instance detection network and the target instance-level domain classifier. In this embodiment, the instance-level detection network includes a region generation network and a fully connected layer.

[0078] When training the detection network, the source domain image and the target domain image are first input into the first feature extraction network, which extracts features from the source and target domain images and outputs a first feature map. The first feature map is then input into the first image-level domain classifier through the first gradient reversal layer. The first image-level domain classifier performs domain classification on each pixel in the first feature map and outputs a first classification feature map with the same width and height as the first feature map. The value of each pixel in the first classification feature map represents the probability that the corresponding position in the corresponding first feature map belongs to the source domain.

[0079] Based on the first classification feature map, a local attention value for each pixel in the first feature map can be determined. The local attention value can be calculated using the following formula (1).

[0080]

[0081] in, represents the local attention value of region k in image i; represents the probability that region k in image i belongs to the source domain; For about The entropy function of , where:

[0082]

[0083] Where j = 1 or 0, j = 1 means that the probability of region k in image i belonging to the source domain is less than the probability of belonging to the target domain, j = 0 means that the probability of region k in image i belonging to the source domain is greater than the probability of belonging to the target domain.

[0084] For a region in a sample image, if the first-level domain classifier can distinguish the source domain from the target domain, it means that this region is a representative feature of the foreground image of the source and target domains, and accordingly, a larger attention value should be assigned to this region. However, incorrect attention values ​​will have a certain negative impact on the domain adaptation task. To reduce this impact, the deep residual attention structure can be used to enhance the robustness of the detection network to incorrect attention values.

[0085] The weighted calculation of the first feature map can be implemented based on the following formula (2).

[0086]

[0087] in, Represents the feature The updated features obtained after weighting, Represents region k in the first feature map corresponding to image i.

[0088] After updating the first feature map, the updated first feature map is input into the second feature extraction network, which extracts features from the first feature map and outputs a high-level feature map. The updated first feature map is input into the second image-level domain classifier through the first gradient reversal layer. The second image-level domain classifier performs domain classification on each pixel in the updated first feature map and outputs a second classified feature map with the same width and height as the updated first feature map. The value of each pixel in the second classified feature map represents the probability that the corresponding position in the corresponding first feature map belongs to the source domain.

[0089] Based on the second classification feature map, a global attention value for each pixel in the high-level feature map can be determined. Then, the high-level feature map is updated based on the local attention value of each pixel in the high-level feature map to obtain a second feature map. It should be noted that the processing method for updating the high-level feature map is the same as the processing method for updating the first feature map, and the present invention will not be repeated here.

[0090] It should be noted that the network structure of the second domain classifier is different from that of the first domain classifier. This may lead to a size mismatch between the feature image output by the second domain classifier and the high-level feature map, making it impossible to directly perform weighted updates on the high-level feature map. Therefore, an upsampling layer can be added to the second domain classifier to ensure that the size of the feature image output by the second domain classifier is consistent with that of the high-level feature map, allowing for subsequent weighted updates.

[0091] After obtaining the second feature map, it is input into the region generation network in the object detection model. The region generation network sorts and selects the candidate regions with the highest scores. After the Roi-Align operation, the third feature map is obtained. The third feature map is marked with the location area of ​​each instance feature corresponding to the target instance category label. After the third feature map passes through the fully connected layer of the object detection model, the corresponding category classification result of each instance is output.

[0092] In the above training process, the image-level domain classifier is able to mix the sample image features from the source domain and the target domain, but in order to further improve the detection performance of the detection model, it is necessary to further focus on the feature distribution of local instances.

[0093] To avoid incorrectly aligning instances from different domains to different instance classes, which could lead to false or missed detections and consequently, poor model performance, in this embodiment, in the tree crown detection scenario, if the foreground tree crown features in the source domain are aligned with the background features in the target domain, false or missed detections could occur, leading to poor model performance.

[0094] Therefore, in this embodiment, the instance-level domain classifiers may include multiple ones. Specifically, taking the sample image as a tree crown image in a tree crown detection scene as an example, the instance categories in the tree crown image include foreground instance categories and background instance categories, then two corresponding instance-level domain classifiers may be provided, including an instance-level domain classifier corresponding to the foreground instance category and an instance-level domain classifier corresponding to the background instance category.

[0095] Before inputting instance-level features into the instance-level domain classifier, it is necessary to first determine the instance category label corresponding to each sample image. Specifically, if the sample image is a source domain image, the instance category label of the source domain image is directly used. If the sample image is a target domain image, the category classification result of each instance is used as the instance category label of the target domain image.

[0096] Afterwards, the third feature map is input into the target instance-level domain classifier corresponding to the target instance category label to obtain the domain category prediction result output by the target instance-level domain classifier.

[0097] Finally, based on the domain category prediction results output by the image-level domain classifier, the domain category prediction results output by the target instance-level domain classifier, and the category classification results of each instance output by the instance detection network, the detection network is trained to obtain a target detection model for instance detection of the target domain's images to be detected.

[0098] In this embodiment, the image-level alignment loss corresponding to the image-level domain classifier is determined based on the domain category prediction results output by the image-level domain classifier and the domain category labels corresponding to each sample image. The instance-level alignment loss corresponding to the target instance-level domain classifier is determined based on the domain category prediction results output by the target instance-level domain classifier and the domain category labels corresponding to each instance. The entropy regularization loss and detection loss corresponding to the instance detection network are determined based on the category classification results of each instance output by the instance detection network and the instance category labels corresponding to the source domain image. The detection network is trained based on the image-level alignment loss, instance-level alignment loss, entropy regularization loss, and detection loss.

[0099] Specifically, the image-level alignment loss includes a first image-level alignment loss corresponding to the first image-level domain classifier and a second image-level alignment loss corresponding to the second image-level domain classifier.

[0100] Among them, the first image-level alignment loss can be calculated by the following formula (3).

[0101]

[0102] Among them, L loc (F l ,D l ) represents the first image-level alignment loss; L locs represents the first image-level alignment loss corresponding to the source domain; L loct represents the first image-level alignment loss corresponding to the target domain; F l represents the first feature extraction network; D l represents the first image-level domain classifier.

[0103] The first image-level alignment loss corresponding to the source domain can be calculated by the following formula (4).

[0104]

[0105] Among them, n s represents the number of source domain images; W represents the width of the first feature map; H represents the height of the first feature map; F l represents the first feature extraction network; D l represents the first image-level domain classifier; represents the image i in the source domain.

[0106] The first image-level alignment loss corresponding to the target domain can be calculated by the following formula (5).

[0107]

[0108] Among them, n trepresents the number of target domain images; W represents the width of the first feature map; H represents the height of the first feature map; F l represents the first feature extraction network; D l represents the first image-level domain classifier; represents an image i in the target domain.

[0109] The second image-level alignment loss can be calculated using the following formula (6).

[0110]

[0111] Among them, L global (F g ,D g ) represents the second image-level alignment loss, L globals represents the second image-level alignment loss corresponding to the source domain; L globalt represents the second image-level alignment loss corresponding to the target domain; F g represents the second feature extraction network; D g represents the second image-level domain classifier.

[0112] The second image-level alignment loss corresponding to the source domain can be calculated by the following formula (7).

[0113]

[0114] Among them, n s represents the number of source domain images; F g represents the second feature extraction network; D g represents the second image-level domain classifier; represents the updated first feature map corresponding to the image i in the target domain; γ (γ≥0) represents the increase of the focus parameter.

[0115] The second image-level alignment loss corresponding to the target domain can be calculated by the following formula (8).

[0116]

[0117] Among them, n t represents the number of target domain images; F g represents the second feature extraction network; D g represents the second image-level domain classifier; represents the updated first feature map corresponding to the image i in the target domain; γ (γ≥0) represents the increase of the focus parameter.

[0118] Furthermore, adversarial domain adaptation aims to align sample features from the source and target domains, preventing the domain classifier from correctly determining the domain category of the current feature. To achieve this, the model must focus on hard-to-confuse samples that are far from the domain classification boundary and prioritize them towards the boundary. Therefore, hard-to-confuse samples are captured using the domain category predictions output by the target instance-level domain classifier and weighted to give them a higher weight. It should be noted that a higher domain classification probability indicates a hard-to-confuse sample, while a lower probability indicates an easy-to-confuse sample.

[0119] Among them, the weight value of instance-level alignment loss can be calculated by the following formula (9).

[0120] W i,j,c =-plog(1-p) (9)

[0121] Among them, W i,j,c Represents instance feature m (i,j) The corresponding weight value;

[0122] d represents the domain label of the sample image;

[0123] 1 represents the source domain label, 0 represents the target domain label, p i,j,c Represents instance feature m (i,j) The probability of belonging to the source domain, m (i,j) represents the jth instance feature in image i.

[0124] The weighted instance-level alignment loss can be calculated by the following formula (10).

[0125] L multi-ins-w =L inss-w +L inst-w (10)

[0126] Among them, L multi-ins-w represents instance-level alignment loss; L inss-w represents the instance-level alignment loss corresponding to the source domain; L inst-w represents the instance-level alignment loss corresponding to the target domain.

[0127] The instance-level alignment loss corresponding to the source domain can be calculated by the following formula (11).

[0128]

[0129] Among them, n s represents the number of source domain images; represents the category classification result corresponding to the source domain image; p i,j,c Represents instance feature m (i,j) The probability of belonging to the source domain, m (i,j)represents the jth instance feature in image i, W i,j,c represents the domain classification probability of the jth instance in image i output by the instance-level domain classifier corresponding to category c.

[0130] The instance-level alignment loss corresponding to the target domain can be calculated using the following formula (12).

[0131]

[0132] Among them, n t Indicates the number of target domain images; represents the category classification result corresponding to the target domain image; p i,j,c Represents instance feature m (i,j) The probability of belonging to the target domain.

[0133] In this embodiment, the entropy regularization loss can be determined based on the category classification result corresponding to the target domain image output by the instance detection network. The detection loss is determined based on the category classification result corresponding to the source domain image output by the instance detection network and the instance category label corresponding to the source domain image. It should be noted that the detection loss includes the classification loss determined by the category classification result corresponding to the source domain image output by the instance detection network, the regression loss determined based on the regression parameters and the true bounding box coordinates of the instance features in the source domain image, and the classification loss and regression loss generated by the region generation network.

[0134] The entropy regularization loss can be calculated by the following formula (13).

[0135]

[0136] In this embodiment, the detection loss corresponding to the instance detection network can be calculated by referring to the detection loss in the related art, and the present invention will not elaborate on it here.

[0137] According to an embodiment of the present invention, first, after extracting the first feature map of the sample image through the feature extraction network, the source domain image and the target domain image are aligned at the picture level through the image-level domain classifier to obtain a second feature map, thereby highlighting the transferable area in the first feature map and suppressing the negative transfer of each area. Then, the instance-level features in the second feature map are extracted through the instance detection network to obtain a third feature map. The third feature map is aligned at the instance level through the instance-level domain classifier to avoid misalignment through the instance category label of the instance, thereby further improving the image alignment effect. Finally, the detection network is trained based on the results output by the target detection model, the image-level domain classifier, and the instance-level domain classifier, thereby improving the image detection effect of the target detection network.

[0138] The image detection method provided by the present invention is described in detail below with reference to the following embodiments.

[0139] Figure 3 A flowchart of an image detection method provided by an embodiment of the present invention. The method includes the following steps:

[0140] S301, obtaining an image to be detected and a target detection model.

[0141] S302: Detect the model to be detected by the target detection model to obtain the instance region corresponding to each instance in the image to be detected and the category classification result. The target detection model is trained using the image detection model training method in the above embodiment.

[0142] Figure 4 A schematic diagram of the structure of an image detection model training device provided by an embodiment of the present invention is shown in FIG. Figure 4 As shown, the device is applied to a detection network, which includes an object detection model, an image-level domain classifier, and instance-level domain classifiers corresponding to different instance categories. The device includes: an acquisition module 401, an extraction module 402, an update module 403, a processing module 404, a determination module 405, a generation module 406, and a training module 407.

[0143] An acquisition module 401 is used to acquire sample images and a target detection model to be trained. The sample images include source domain images and target domain images. The source domain images have instance category labels, and the target domain images do not have instance category labels. Both the source domain images and the target domain images have domain category labels. The target detection model is pre-trained using the source domain images, and the target detection model includes a feature extraction network and an instance detection network.

[0144] The extraction module 402 is configured to extract features from the sample image through a feature extraction network to obtain a first feature map.

[0145] The updating module 403 is used to input the first feature map into the image-level domain classifier to obtain the domain category prediction result output by the image-level domain classifier, determine the attention weight of each region in the first feature map based on the domain category prediction result, and update the first feature map according to the attention weight of each region in the first feature map to obtain the second feature map.

[0146] The processing module 404 is configured to input the second feature map into the instance detection network to obtain a location region and category classification result of each instance.

[0147] The determination module 405 is configured to determine the instance category label corresponding to the sample image according to the category classification result of each instance and the domain category label corresponding to the sample image.

[0148] The generation module 406 is used to generate a third feature map based on the second feature map, wherein the third feature map is marked with a location area of ​​each instance corresponding to the target instance category label, and the target instance category label is any one of the instance category labels corresponding to the sample image.

[0149] The processing module 404 is further configured to input the third feature map into a target instance-level domain classifier corresponding to the target instance category label to obtain a domain category prediction result output by the target instance-level domain classifier.

[0150] The training module 407 is used to train the detection network based on the domain category prediction results output by the image-level domain classifier, the domain category prediction results output by the target instance-level domain classifier, and the category classification results and regression parameters of each instance output by the instance detection network, so as to obtain a target detection model for instance detection of the image to be detected in the target domain.

[0151] According to an embodiment of the present invention, if the sample image is a source domain image, the instance category label of the source domain image is directly used; if the sample image is a target domain image, the category classification result of each instance is used as the instance category label of the target domain image.

[0152] According to an embodiment of the present invention, the feature extraction network includes a first feature extraction network and a second feature extraction network, and the sampling receptive field corresponding to the first feature extraction network is smaller than the sampling receptive field corresponding to the second feature extraction network; the image-level domain classifier includes a first image-level domain classifier corresponding to the first feature extraction network, and a second image-level domain classifier corresponding to the second feature extraction network.

[0153] According to an embodiment of the present invention, the detection network also includes a first gradient reversal layer located between the feature extraction network and the image-level domain classifier, which is used to realize adversarial training of the feature extraction network and the image-level domain classifier; the detection network also includes a second gradient reversal layer located between the instance detection network and the target instance-level domain classifier, which is used to realize adversarial training of the instance detection network and the target instance-level domain classifier.

[0154] According to an embodiment of the present invention, the training module 407 is further used to determine the image-level alignment loss corresponding to the image-level domain classifier based on the domain category prediction results output by the image-level domain classifier and the domain category labels corresponding to each sample image; determine the instance-level alignment loss corresponding to the target instance-level domain classifier based on the domain category prediction results output by the target instance-level domain classifier and the domain category labels corresponding to each instance; determine the entropy regularization loss and detection loss corresponding to the instance detection network based on the category classification results of each instance output by the instance detection network and the instance category labels corresponding to the source domain image; and train the detection network based on the image-level alignment loss, instance-level alignment loss, entropy regularization loss and detection loss.

[0155] According to an embodiment of the present invention, the training module 407 is further used to determine the entropy regularization loss based on the category classification result corresponding to the target domain image output by the instance detection network; and determine the detection loss based on the category classification result corresponding to the source domain image output by the instance detection network and the instance category label corresponding to the source domain image.

[0156] In one possible design, the above Figure 4 The structure of the image classification device shown can be implemented as an electronic device. Figure 5 As shown, the electronic device 500 may include: a processor 501 and a memory 502. The memory 502 stores executable code, and when the executable code is executed by the processor 501, the processor 501 can at least implement the aforementioned Figure 1 The image detection model training method provided in the illustrated embodiment.

[0157] The structure of the control device may further include a communication interface 503 for communicating with other devices.

[0158] Figure 6 A schematic diagram of the structure of another electronic device provided by an embodiment of the present invention, such as Figure 6 As shown, the electronic device 600 may include one or more of the following components: a processing component 602 , a memory 604 , a power component 606 , a multimedia component 608 , an audio component 610 , an input / output (I / O) interface 612 , a sensor component 614 , and a communication component 616 .

[0159] The processing component 602 generally controls the overall operation of the electronic device 600, such as operations associated with display, phone calls, data communications, camera operation, and recording operations. The processing component 602 may include one or more processors 620 to execute instructions to perform all or part of the steps of the above-described method S101-S108. In addition, the processing component 602 may include one or more modules to facilitate interaction between the processing component 602 and other components. For example, the processing component 602 may include a multimedia module to facilitate interaction between the multimedia component 608 and the processing component 602.

[0160] The memory 604 is configured to store various types of data to support operations on the electronic device 600. Examples of such data include instructions for any application or method operating on the electronic device 600, contact data, phone book data, messages, pictures, videos, etc. The memory 604 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0161] The power supply assembly 606 provides power to the various components of the electronic device 600. The power supply assembly 606 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 600.

[0162] The multimedia component 608 includes a screen that provides an output interface between the electronic device 600 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, slides, and gestures on the touch panel. The touch sensor can not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 608 includes a front camera and / or a rear camera. When the electronic device 600 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each front camera and rear camera can be a fixed optical lens system or have a focal length and optical zoom capability.

[0163] The audio component 610 is configured to output and / or input audio signals. For example, the audio component 610 includes a microphone (MIC), which is configured to receive external audio signals when the electronic device 600 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in the memory 604 or transmitted via the communication component 616. In some embodiments, the audio component 610 also includes a speaker for outputting audio signals.

[0164] The input / output interface 612 provides an interface between the processing component 602 and peripheral interface modules, such as a keyboard, a click wheel, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0165] The sensor assembly 614 includes one or more sensors for providing various aspects of status assessment for the electronic device 600. For example, the sensor assembly 614 can detect the open / closed state of the electronic device 600, the relative positioning of components, such as the display and keypad of the electronic device 600. The sensor assembly 614 can also detect changes in the position of the electronic device 600 or a component of the electronic device 600, the presence or absence of user contact with the electronic device 600, the orientation or acceleration / deceleration of the electronic device 600, and temperature changes of the electronic device 600. The sensor assembly 614 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 614 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 614 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0166] The communication component 616 is configured to facilitate wired or wireless communication between the electronic device 600 and other devices. The electronic device 600 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G or 4G or a combination thereof. In an exemplary embodiment, the communication component 616 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 616 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0167] In an exemplary embodiment, the electronic device 600 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0168] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is further provided, such as a memory 604 including instructions, and the instructions can be executed by the processor 620 of the electronic device 600 to perform the above method. For example, the non-transitory computer-readable storage medium can be implemented by any type of volatile or non-volatile memory device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0169] In addition, an embodiment of the present invention provides a non-transitory machine-readable storage medium, wherein the non-transitory machine-readable storage medium stores executable code, and when the executable code is executed by a processor of an electronic device, the processor executes the aforementioned Figure 1 The image detection model training method provided in the illustrated embodiment.

[0170] The device embodiments described above are merely illustrative, wherein the modules described as separate components may or may not be physically separate. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0171] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by adding a necessary general hardware platform, and of course can also be implemented by a combination of hardware and software. Based on this understanding, the above technical solution, or the part that contributes to the present invention, can be embodied in the form of a computer product. The present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0172] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for training an image detection model, characterized in that: Applied to a detection network, the detection network includes an object detection model, an image-level domain classifier, and instance-level domain classifiers corresponding to different instance categories, the method includes: Obtaining a sample image and the target detection model to be trained, wherein the sample image includes a source domain image and a target domain image, the source domain image has an instance category label, the target domain image does not have an instance category label, and both the source domain image and the target domain image have domain category labels, the target detection model is pre-trained using the source domain image, and the target detection model includes a feature extraction network and an instance detection network; Performing feature extraction on the sample image through the feature extraction network to obtain a first feature map; Inputting the first feature map into the image-level domain classifier to obtain a domain category prediction result output by the image-level domain classifier, determining an attention weight for each region in the first feature map based on the domain category prediction result, and updating the first feature map according to the attention weight for each region in the first feature map to obtain a second feature map; Inputting the second feature map into the instance detection network to obtain a location region and category classification result of each instance; Determining the instance category label corresponding to the sample image according to the category classification results of each instance and the domain category label corresponding to the sample image; generating a third feature map based on the second feature map, wherein the third feature map is marked with a location area of ​​each instance corresponding to a target instance category label, and the target instance category label is any one of the instance category labels corresponding to the sample image; Inputting the third feature map into a target instance-level domain classifier corresponding to the target instance category label to obtain a domain category prediction result output by the target instance-level domain classifier; The detection network is trained based on the domain category prediction results output by the image-level domain classifier, the domain category prediction results output by the target instance-level domain classifier, and the category classification results and regression parameters of each instance output by the instance detection network to obtain a target detection model for performing instance detection on the image to be detected in the target domain.

2. The method according to claim 1, characterized in that Determining the instance category label corresponding to the sample image according to the category classification results of each instance and the domain category label corresponding to the sample image includes: If the sample image is a source domain image, the instance category label of the source domain image is directly used; If the sample image is a target domain image, the category classification results of the instances are used as instance category labels of the target domain image.

3. The method according to claim 1, characterized in that The feature extraction network includes a first feature extraction network and a second feature extraction network, and the sampling receptive field corresponding to the first feature extraction network is smaller than the sampling receptive field corresponding to the second feature extraction network; The image-level domain classifier includes a first image-level domain classifier corresponding to the first feature extraction network and a second image-level domain classifier corresponding to the second feature extraction network.

4. The method according to claim 1, wherein The detection network further includes a first gradient reversal layer located between the feature extraction network and the image-level domain classifier, for implementing adversarial training of the feature extraction network and the image-level domain classifier; The detection network also includes a second gradient reversal layer located between the instance detection network and the target instance-level domain classifier, for implementing adversarial training of the instance detection network and the target instance-level domain classifier.

5. The method according to claim 1, characterized in that The step of training the detection network according to the domain category prediction result output by the image-level domain classifier, the domain category prediction result output by the target instance-level domain classifier, and the category classification result of each instance output by the instance detection network comprises: Determining an image-level alignment loss corresponding to the image-level domain classifier according to the domain category prediction result output by the image-level domain classifier and the domain category labels corresponding to each of the sample images; Determining the instance-level alignment loss corresponding to the target instance-level domain classifier according to the domain category prediction result output by the target instance-level domain classifier and the domain category label corresponding to each instance; and Determining an entropy regularization loss and a detection loss corresponding to the instance detection network according to a category classification result of each instance output by the instance detection network and the instance category label corresponding to the source domain image; The detection network is trained based on the image-level alignment loss, the instance-level alignment loss, the entropy regularization loss, and the detection loss.

6. The method according to claim 5, characterized in that Determining the entropy regularization loss and the detection loss corresponding to the instance detection network according to the category classification result of each instance output by the instance detection network and the instance category label corresponding to the source domain image includes: determining the entropy regularization loss according to the category classification result corresponding to the target domain image output by the instance detection network; The detection loss is determined according to the category classification result corresponding to the source domain image output by the instance detection network and the instance category label corresponding to the source domain image.

7. An image detection method, characterized in that: include: Obtain the image to be detected and the target detection model; Detecting the image to be detected using the target detection model to obtain instance regions and category classification results corresponding to each instance in the image to be detected; The target detection model is trained using the image detection model training method described in any one of claims 1 to 6.

8. An image detection model training device, characterized in that: Applied to a detection network, the detection network includes a target detection model, an image-level domain classifier, and instance-level domain classifiers corresponding to different instance categories, the device includes: an acquisition module, configured to acquire a sample image and the target detection model to be trained, wherein the sample image includes a source domain image and a target domain image, the source domain image has an instance category label, the target domain image does not have an instance category label, and both the source domain image and the target domain image have domain category labels; the target detection model is pre-trained using the source domain image, and includes a feature extraction network and an instance detection network; an extraction module, configured to extract features from the sample image using the feature extraction network to obtain a first feature map; an updating module, configured to input the first feature map into the image-level domain classifier to obtain a domain category prediction result output by the image-level domain classifier, determine an attention weight for each region in the first feature map based on the domain category prediction result, and update the first feature map according to the attention weight for each region in the first feature map to obtain a second feature map; a processing module, configured to input the second feature map into the instance detection network to obtain a location region and category classification result of each instance; a determination module, configured to determine the instance category label corresponding to the sample image based on the category classification results of the instances and the domain category label corresponding to the sample image; a generating module, configured to generate a third feature map based on the second feature map, wherein the third feature map is marked with a location area of ​​each instance corresponding to a target instance category label, and the target instance category label is any one of the instance category labels corresponding to the sample image; The processing module is further configured to input the third feature map into a target instance-level domain classifier corresponding to the target instance category label to obtain a domain category prediction result output by the target instance-level domain classifier; A training module is used to train the detection network based on the domain category prediction results output by the image-level domain classifier, the domain category prediction results output by the target instance-level domain classifier, and the category classification results and regression parameters of each instance output by the instance detection network, so as to obtain a target detection model for instance detection of the image to be detected in the target domain.

9. An electronic device, characterized in that: include: A memory and a processor; wherein the memory stores executable code, and when the executable code is executed by the processor, the processor executes the image detection model training method according to any one of claims 1 to 6.

10. A non-transitory machine-readable storage medium, characterized in that The non-transitory machine-readable storage medium stores executable code, and when the executable code is executed by a processor of an electronic device, the processor executes the image detection model training method as described in any one of claims 1 to 6.

Images