Target detection method, device, intelligent terminal and computer-readable storage medium

Abstract

The present invention discloses a target detection method, device, intelligent terminal and computer-readable storage medium, wherein the target detection method comprises: obtaining a source domain image and a target domain image; performing feature extraction on the source domain image and the target domain image to obtain image features; obtaining domain-invariant features based on the image features; obtaining domain-invariant region features based on the domain-invariant features; and performing target detection based on the domain-invariant region features. The domain-invariant region features are domain-invariant instance features. Thus, compared with the prior art, the scheme of the present invention takes into account the instance-level representation that is very important for target detection, and can therefore be used for target detection tasks that focus on the instance feature level, and is beneficial to improving the accuracy and migration capability of the target detection method.

Description

Technical Field

[0001] The present invention relates to the field of domain adaptive target detection technology, and in particular to a target detection method, device, intelligent terminal and computer-readable storage medium. Background Art

[0002] Domain adaptive object detection is a branch of domain adaptive learning, and is the embodiment of domain adaptive learning methods in image object detection tasks. Its main research content is how to effectively use the object detection model trained on the source domain data on the target domain data when there is a domain deviation between the training data and the test data, for example, the lighting and weather of the training and test scenes are different. Labeling the target domain data is a time-consuming and laborious task, so unsupervised domain adaptation is a common scenario. At present, the common method to solve the problem of unsupervised domain adaptation is feature alignment.

[0003] In the prior art, in order to solve the problem of domain adaptive object detection, various unsupervised domain adaptive object detection methods usually reduce domain deviation by adjusting the feature level or pixel level distribution of the source domain and the target domain, thereby achieving feature alignment. The problem with the prior art is that these feature alignment methods only consider the overall representation at the image level, while ignoring the instance level representation that is very important for object detection. Therefore, they are not applicable to object detection tasks that focus on the instance feature level, and affect the accuracy and migration ability of the object detection method.

[0004] Therefore, the existing technology still needs to be improved and developed. Summary of the invention

[0005] The main purpose of the present invention is to provide a target detection method, device, intelligent terminal and computer-readable storage medium, aiming to solve the problem that the feature alignment method of the target detection method in the prior art only considers the overall representation at the image level, cannot be applied to target detection tasks focusing on the instance feature level, and is not conducive to improving the accuracy and migration ability of the target detection method.

[0006] In order to achieve the above object, the first aspect of the present invention provides a target detection method, wherein the method comprises:

[0007] Obtain source domain images and target domain images;

[0008] Perform feature extraction on the above-mentioned source domain image and target domain image to obtain image features;

[0009] Based on the above image features, domain invariant features are obtained;

[0010] Obtaining domain-invariant region features based on the above domain-invariant features;

[0011] Object detection is performed based on the above domain-invariant region features.

[0012] Optionally, the extracting features of the source domain image and the target domain image to obtain image features includes:

[0013] Extracting features of the source domain image and the target domain image using a feature extractor;

[0014] The extracted features are used as image features.

[0015] Optionally, the acquiring of domain invariant features based on the image features includes:

[0016] The above image features are input into an extractor for extracting domain-invariant features, and the domain-invariant features are extracted and output by the extractor for extracting domain-invariant features.

[0017] Optionally, after obtaining the domain invariant features based on the image features, the method further includes:

[0018] The domain-invariant features are subtracted from the image features to obtain domain-specific features.

[0019] Optionally, after subtracting the domain-invariant feature from the image feature to obtain the domain-specific feature, the object detection method further includes:

[0020] The above domain-specific features are restricted to be orthogonal to the above domain-invariant features.

[0021] Optionally, the obtaining of the domain-invariant region feature based on the domain-invariant feature includes:

[0022] By using a candidate region generation network, a domain-invariant candidate region is obtained based on the above domain-invariant feature extraction;

[0023] Based on the above domain-invariant candidate regions, feature alignment is performed to obtain domain-invariant region features.

[0024] Optionally, the target detection based on the domain-invariant region features includes:

[0025] An object detection task is performed on the domain-invariant region features, wherein the object detection task includes instance classification and / or detection box regression.

[0026] A second aspect of the present invention provides a target detection device, wherein the device comprises:

[0027] An image acquisition module, used to acquire source domain images and target domain images;

[0028] An image feature extraction module is used to extract features from the source domain image and the target domain image to obtain image features;

[0029] A domain invariant feature acquisition module, used to acquire domain invariant features based on the above image features;

[0030] A domain invariant region feature acquisition module, used to acquire domain invariant region features based on the above domain invariant features;

[0031] The detection module is used to detect the target based on the above domain invariant region features.

[0032] A third aspect of the present invention provides an intelligent terminal, comprising a memory, a processor, and a target detection program stored in the memory and executable on the processor, wherein the target detection program implements any one of the steps of the target detection method when executed by the processor.

[0033] A fourth aspect of the present invention provides a computer-readable storage medium, on which a target detection program is stored. When the target detection program is executed by a processor, the step of any one of the above-mentioned target detection methods is implemented.

[0034] As can be seen from the above, the scheme of the present invention obtains source domain images and target domain images; performs feature extraction on the above source domain images and target domain images to obtain image features; obtains domain invariant features based on the above image features; obtains domain invariant region features based on the above domain invariant features; and performs target detection based on the above domain invariant region features. Among them, the above domain invariant region features are domain invariant instance features. In this way, compared with the prior art, the scheme of the present invention takes into account the instance-level representation that is very important for target detection, and can therefore be used for target detection tasks that focus on the instance feature level, and is beneficial to improving the accuracy and migration capabilities of target detection methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0036] Figure 1 It is a schematic diagram of a feature alignment method based on adversarial game in the prior art provided by an embodiment of the present invention;

[0037] Figure 2 It is a schematic diagram of a strong and weak distribution alignment method in the prior art provided by an embodiment of the present invention;

[0038] Figure 3 is a flow chart of a target detection method provided by an embodiment of the present invention;

[0039] Figure 4 is a schematic diagram of a domain adaptive target detection method in a specific application scenario provided by an embodiment of the present invention;

[0040] Figure 5 Embodiment of the present invention Figure 3 Specific flow diagram of step S200;

[0041] Figure 6 is a schematic diagram of vector decomposition provided by an embodiment of the present invention;

[0042] Figure 7 is a schematic diagram of a traditional decoupling method provided by an embodiment of the present invention;

[0043] Figure 8 is a schematic diagram of a feature decoupling method based on vector decomposition provided by an embodiment of the present invention;

[0044] Fig. 9 Embodiment of the present invention Figure 3 Specific flow diagram of step S400;

[0045] Fig.10 is a schematic diagram of a model training algorithm provided by an embodiment of the present invention;

[0046] Fig.11 is a schematic diagram of the comparison results of the detection performance of each target detection method provided by the embodiment of the present invention when migrating from daytime to foggy scene;

[0047] Fig.12 is a schematic structural diagram of a target detection device provided by an embodiment of the present invention;

[0048] Fig.13 It is a block diagram of the internal structure principle of an intelligent terminal provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0049] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present invention.

[0050] It should be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0051] It should also be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0052] It should be further understood that the term "and / or" used in the present description and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0053] As used in this specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0054] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0056] Traditional machine learning assumes that test samples and training samples come from the same data distribution, but in many current machine learning scenarios, training samples and test samples may come from different data distributions. Domain adaptive learning can effectively solve the problem of inconsistent data distributions of training samples and test samples. As a recent research in the field of machine learning, it has received widespread attention in recent years.

[0057] In many actual machine learning scenarios, there are situations where the data distribution of training samples and test samples is different. For example, how to make the model trained in sunny scenes also show good enough performance in foggy or dark scenes. The training data here is collected in sunny scenes, while the test data is collected in foggy or dark scenes, and the data distribution of the two is inconsistent. This deviation between domains will make it difficult to migrate models between the two domains, and direct migration will cause a significant decrease in the performance of the model. How to learn in this case where the data distribution of the source domain and the target domain is inconsistent is the problem of domain adaptive learning. In general, the focus of domain adaptive learning is to overcome the problem of inconsistent distribution of the source domain and the target domain and achieve learning tasks in the target domain.

[0058] Domain adaptive object detection is a branch of domain adaptive learning, and is the embodiment of domain adaptive learning methods in image object detection tasks. Its main research content is how to effectively use the object detection model trained on the source domain data on the target domain data when there is a domain deviation between the training data and the test data, for example, the lighting and weather of the training and test scenes are different. In the object detection methods of the prior art, when the domain deviation between the training set and the test set is large, it is difficult for the model to achieve good performance on the test set. For example, it is difficult for a model trained in the source domain (daytime) to achieve good results in the target domain (foggy or night). Therefore, how to achieve all-weather cross-domain object detection has become an urgent problem to be solved.

[0059] In the prior art, in order to solve the problem of domain adaptive object detection, various unsupervised domain adaptive object detection methods usually reduce domain deviation by adjusting the feature level or pixel level distribution of the source domain and the target domain, thereby achieving feature alignment. The problem with the prior art is that these feature alignment methods only consider the overall representation at the image level, while ignoring the instance level representation that is very important for object detection. Therefore, they are not applicable to object detection tasks that focus on the instance feature level, and affect the accuracy and migration ability of the object detection method.

[0060] Figure 1 FIG. 1 is a schematic diagram of a feature alignment method based on adversarial game in the prior art provided by an embodiment of the present invention. Figure 1As shown in the figure, in the back-propagation-based training, a domain classifier is connected to the feature extractor through a gradient reversal layer (i.e., multiplying the gradient by a negative constant). During the training process, the model will maximize the domain classification loss to narrow the gap between the source domain and the target domain. Specifically, this method mainly uses a feature extractor and two classifiers. The feature extractor is used to extract the features of the image, one classifier is used for the target task, and the other is used to distinguish the source domain from the target domain. This method cleverly designs a gradient reversal layer (GRL) to ensure that the feature distributions of the source domain and the target domain are similar (as indistinguishable as possible for the domain classifier), thereby obtaining domain-invariant features. Since the domain classifier can clearly distinguish the source domain from the target domain, the gap between the source domain and the target domain can be narrowed by maximizing the domain classification loss. With the help of adversarial training, the difference between domains can be significantly narrowed, thereby improving the performance of the model. However, methods based on feature alignment usually only consider feature-level alignment and ignore instance-level alignment, which is very important for object detection.

[0061] Figure 2 is a schematic diagram of a strong and weak distribution alignment (SW) method in the prior art provided by an embodiment of the present invention, which implements strong local alignment through a local domain classifier network and weak global alignment through a global domain classifier. Specifically, Figure 2 As shown in the figure, domain adaptation is performed at the low-level features and high-level features respectively. Since each element of the low-level features reflects the regional information of the image, aligning each element of the low-level features helps to reduce the domain distance between image regions. Aligning high-level features helps to reduce the domain distance between overall images. In this way, considering both local and global alignment can solve the problem of domain adaptive object detection. However, this method does not take into account the importance of domain invariant features for domain adaptive object detection, and also ignores the instance-level representation that is very important for object detection. Therefore, it cannot be applied to object detection tasks that focus on instance feature levels, and its accuracy and migration capabilities are limited.

[0062] In order to solve the problems of the prior art, an embodiment of the present invention provides a target detection method. In the embodiment of the present invention, a source domain image and a target domain image are obtained; feature extraction is performed on the source domain image and the target domain image to obtain image features; domain invariant features are obtained based on the image features; domain invariant region features are obtained based on the domain invariant features; and target detection is performed based on the domain invariant region features. The domain invariant region features are domain invariant instance features. In this way, compared with the prior art, the scheme of the present invention takes into account the instance-level representation that is very important for target detection, and can therefore be used for target detection tasks that focus on the instance feature level, and is beneficial to improving the accuracy and migration capabilities of target detection methods.

[0063] Exemplary Methods

[0064] like Figure 3 As shown, an embodiment of the present invention provides a target detection method. Specifically, the method includes the following steps:

[0065] Step S100: acquiring a source domain image and a target domain image.

[0066] It is necessary to perform domain adaptive object detection on the target domain image based on the source domain image and the target domain image. Specifically, in this embodiment, there may be domain deviation between the source domain image and the target domain image. For example, the source domain image may be a daytime image, and the target domain image may correspond to a night image or a foggy image, which is not specifically limited here.

[0067] Step S200: extract features from the source domain image and the target domain image to obtain image features.

[0068] Optionally, feature extraction may be performed on the source domain image and the target domain image respectively to obtain image features corresponding to the source domain image and image features corresponding to the target domain image respectively.

[0069] Step S300: obtaining domain invariant features based on the above image features.

[0070] Among them, the above-mentioned domain invariant features refer to features related to the attributes of the detection target, and are independent of attributes such as image style, background, and lighting.

[0071] Step S400: obtaining domain-invariant region features based on the domain-invariant features.

[0072] Step S500: Detecting an object based on the domain-invariant region features.

[0073] The domain-invariant region feature is a feature of a region corresponding to the domain-invariant feature. For example, the domain-invariant region may be a region where the detection target is located. The domain-invariant region feature is a feature of the region, that is, a domain-invariant instance feature. Target detection based on the domain-invariant region feature is target detection based on the domain-invariant instance feature.

[0074] As can be seen from the above, the target detection method provided by the embodiment of the present invention obtains a source domain image and a target domain image; performs feature extraction on the above source domain image and target domain image to obtain image features; obtains domain invariant features based on the above image features; obtains domain invariant region features based on the above domain invariant features; and performs target detection based on the above domain invariant region features. Among them, the above domain invariant region features are domain invariant instance features. In this way, compared with the prior art, the scheme of the present invention takes into account the instance-level representation that is very important for target detection, and can therefore be used for target detection tasks that focus on the instance feature level, and is beneficial to improving the accuracy and migration capabilities of the target detection method.

[0075] Figure 4 It is a schematic diagram of a domain adaptive target detection method in a specific application scenario provided by an embodiment of the present invention, wherein E1 and E2 are two feature extractors, GRL is a gradient reversal layer, RPN is a candidate region generation network (RegionProposal Network), Domain Classifier is a domain classifier, RA represents an ROI alignment operation (RA, ROI-Alignment), FC represents a fully connected network, Orthogonal represents the calculation of feature orthogonal loss, and Class and Bbox represent the output results of classification and bounding box regression using a fully connected network, respectively. Figure 4 This embodiment is only an example application scenario provided by the present invention. There may be other specific methods in other specific application scenarios, which are not specifically limited here.

[0076] Specifically, in this embodiment, Figure 5 As shown, the above step S200 includes:

[0077] Step S201: extracting features of the source domain image and the target domain image using a feature extractor.

[0078] Step S202: taking the extracted features as image features.

[0079] Optionally, feature extraction may be performed on the source domain image and the target domain image respectively to obtain image features corresponding to the source domain image and image features corresponding to the target domain image respectively.

[0080] exist Figure 4 In the application scenario shown, the input source domain image x is obtained s , label y s , bounding box b s And the target domain image x t For the source domain image x s and the target domain image x tFirst, the feature extractors E1 and E2 composed of two layers of networks are used to extract the image features F of the source domain and the target domain images. b Among them, the image feature F b The size of F is c1*h1*w1, where c1, h1, and w1 represent the number of channels, height, and width of the feature, respectively. l It is the feature extracted by feature extractor E1. After being processed by gradient reversal layer and domain classifier, it can be used to reduce the domain deviation between source domain and target domain. b =E2(F l ).

[0081] Specifically, in this embodiment, the step S300 includes: inputting the image features into an extractor for extracting domain-invariant features, and extracting and outputting the domain-invariant features through the extractor for extracting domain-invariant features.

[0082] Among them, the extractor for extracting domain-invariant features can be a feature extractor based on a multi-layer convolutional network. By inputting image features into the feature extractor, domain-invariant features can be extracted.

[0083] exist Figure 4 In the application scenario shown, the image feature F b Input to the extractor (multi-layer convolutional network) E for extracting domain-invariant features DIR In the example, the output is the domain invariant feature F di , where the domain invariant feature F di The size of F is c1*h1*w1, di =E DIR (F b ).

[0084] For domain adaptive object detection, it is very important to obtain domain invariant representation, which not only can alleviate the impact of domain bias, but also helps to obtain domain invariant object representation. Therefore, this scheme focuses on extracting domain invariant instance-level features, which helps to improve the migration ability of detection methods.

[0085] Specifically, in this embodiment, after the step S300, the method further includes: subtracting the domain-invariant feature from the image feature to obtain a domain-specific feature.

[0086] Optionally, there may be other methods for obtaining domain-specific features, such as directly extracting domain-specific features, etc., which are not specifically limited here. The above-mentioned domain-specific features refer to features such as image style and background that are irrelevant to the detection target. Removing these features can improve the generalization performance of the model and improve the accuracy and migration ability of the target detection method.

[0087] exist Figure 4 In the application scenario shown, the image feature F b Subtract the domain invariant feature F di Get domain-specific features F ds , where the domain-specific feature F ds The size of F is c1*h1*w1, ds =F b -F di .

[0088] Optionally, a domain classifier can be used to ensure that domain-specific features F ds contains more domain-specific information. Furthermore, in order to reduce domain deviation and improve decomposition ability, image features F can be classified based on domain classifiers. b and domain invariant feature F di The model is trained to maximize the domain classification loss, thereby narrowing the gap between the source domain and the target domain. Figure 4 In the application scenario shown, the feature F l After the gradient reversal layer, the domain classifier is used to perform domain classification and the image feature F b After the gradient reversal layer, the domain classifier is used for domain classification. l The size is c0*h0*w0, where c0, h0, and w0 represent the number of channels, height, and width of the feature, respectively.

[0089] Specifically, in this embodiment, after subtracting the domain-invariant feature from the image feature to obtain the domain-specific feature, the method further includes: restricting the domain-specific feature and the domain-invariant feature to be orthogonal.

[0090] A key step in feature separation is to ensure the independence of domain-invariant features and domain-specific features. Optionally, orthogonal loss can be set to ensure their independence by limiting the orthogonality between domain-invariant features and domain-specific features. Figure 4 In the application scenario shown, the restricted domain invariant feature F di and domain-specific features F ds Orthogonal to each other.

[0091] Optionally, in an embodiment of the present invention, feature decomposition is used to decompose instance features, and feature decomposition is achieved based on vector decomposition, and the decomposed vectors (i.e., corresponding features) are restricted to be orthogonal. The purpose of feature decoupling is to obtain a set of independent representations that can fully express the information of the original features. In an embodiment of the present invention, feature decoupling is converted into a process of vector decomposition.

[0092] Figure 6 is a schematic diagram of a vector decomposition provided by an embodiment of the present invention, such as Figure 6 As shown in FIG. 1 , vector decomposition is a general process of decomposing a vector into two or more component vectors, and the sum of these component vectors is the original vector. Obviously, the idea of ​​vector decomposition is similar to that of feature decoupling. Therefore, the vector decomposition idea is used in the embodiment of the present invention to obtain a decoupled representation.

[0093] Specifically, for the case of two components, given an input representation I, an extractor F is designed to decompose the first component V1 from I. Then, we take the difference V2 between I and V1 as the second component and ensure that they remain orthogonal, as shown in the following formula:

[0094] V1=F(I),V2=I-V1,V1⊥V2 (1)

[0095] In addition, for the case of three components, extractors F and G are designed to decompose the first component V1 and the second component V2, and ensure that they remain orthogonal, as shown in the following formula:

[0096] V1=F(I), V2=G(I), V3=I-V1-V2 (2)

[0097] In the above formula (2), V1, V2 and V3 remain orthogonal. In this way, in the embodiment of the present invention, the domain-invariant features and domain-specific features in the features can be decoupled based on the vector decomposition method, thereby improving the migration capability of the target detection method.

[0098] Figure 7 is a schematic diagram of a traditional decoupling method provided by an embodiment of the present invention, Figure 8 Schematic diagram of a feature decoupling method based on vector decomposition provided by an embodiment of the present invention. Figure 7 As shown in , given an input I, traditional decoupling usually uses two extractors F and G to decouple V1 and V2. V1 and V2 should remain independent. At the same time, in order to ensure that V1 and V2 contain all the information of I, reconstruction operations are usually used. Figure 8As shown, the feature decoupling method based on vector decomposition provided by the embodiment of the present invention uses only one extractor to decompose V1. V2 is represented by the difference between I and V1. At the same time, V1 and V2 remain orthogonal. The decoupling of vector decomposition does not require the use of reconstruction operations to ensure that V1 and V2 contain all the information of I. Specifically, it does not rely on reconstruction operations to ensure that the decomposed components contain all the information of the input. Vector decomposition only uses the difference I-V1 between the original input I and the decomposed component V1 as the second decoupled component, thereby reducing parameters and calculations. In addition, by performing differential operations, all components can be added together to contain all the information of the input without relying on reconstruction operations. Specifically, the embodiment of the present invention designs a feature extractor including multiple convolutional layers, decomposes domain-invariant features from a given input representation, and then obtains a complementary second decoupled component through component differential operations. At the same time, the embodiment of the present invention ensures their independence through orthogonal constraints of the two decomposed components. After obtaining the domain-invariant features, they are used for target detection. Since the domain-invariant features do not contain domain-specific information, they can better solve the migration problem between the two domains.

[0099] Specifically, in this embodiment, Fig. 9 As shown, the above step S400 includes:

[0100] Step S401: obtaining a domain-invariant candidate region based on the domain-invariant feature extraction through a candidate region generation network.

[0101] Step S402: performing feature alignment based on the domain-invariant candidate regions to obtain domain-invariant region features.

[0102] Among them, the above-mentioned candidate region generation network (RPN, Region Proposal Network) is a pre-set and trained network for generating candidate regions, and the above-mentioned domain-invariant candidate regions are candidate regions corresponding to domain-invariant features generated by the candidate region generation network, which can be candidate regions corresponding to detection targets.

[0103] exist Figure 4 In the application scenario shown, the domain invariant feature F di In the input target detector, the candidate region generation network (RPN) generates domain-invariant features F di The domain-invariant candidate region is extracted from the image, and then the ROI alignment operation (RA, ROI-Alignment) is used to obtain the domain-invariant region feature. The domain-invariant region feature is the feature of the region corresponding to the domain-invariant feature. For example, the domain-invariant region can be the region where the detection target is located, and the domain-invariant region feature is the feature of the region, that is, the domain-invariant instance feature.

[0104] Specifically, in this embodiment, the step S500 includes: performing a target detection task on the domain-invariant region features, wherein the target detection task includes instance classification and / or detection box regression.

[0105] Optionally, the above target detection task may also include other specific target detection tasks, which are not specifically limited here. Figure 4 In the application scenario shown, the related tasks of target detection, such as instance classification and detection box regression, are performed on the above domain-invariant region features. For a picture from the source domain, the detection loss can be expressed as: Among them, n s represents the number of candidate regions, A j is the result of the ROI alignment operation on the jth candidate region. The detection network R contains classification and regression networks. det Contains classification loss and detection box regression loss.

[0106] Optionally, before the above step S100, the above target detection method further includes: training a preset target detection model based on a two-stage deep network training mechanism. The above target detection model is a pre-set model for performing target detection based on the above target detection method. Specifically, the above two-stage deep network training mechanism includes training in the feature decomposition stage and training in the feature orthogonalization stage.

[0107] In the first stage, i.e., the feature decomposition stage, the main purpose is to facilitate our model to decompose the input features into two different parts. Specifically, Figure 4 In the application scenario shown, the domain invariant feature F di We first use the candidate region generation network RPN from F di Extract the set of candidate targets O from 1 In order to ensure F b and F di Containing the same content at the same position, based on the set O of candidate targets extracted 1 , in F b and F di Perform ROI alignment operations on each of them to obtain candidate region features and Then set up two networks R b and R di to perform classification and bounding box regression. Finally, for images from the source domain, the detection loss is calculated: in and They represent the classification detection loss and bounding box regression detection loss respectively. In order to ensure that F ds Contains more domain-specific information and uses adversarial training mechanism to design a domain classification network C ds Come to F ds The domain label of the source domain is set to 0, and the domain label of the target domain is set to 1. In addition, in order to reduce the domain deviation and improve the decomposition ability, two domain classifiers C b and C l F b and F l By performing domain classification, the gap between the source domain and the target domain can be narrowed by maximizing the domain classification loss. Finally, the loss of the first stage can be expressed as follows:

[0108]

[0109]

[0110] in, and are the objective functions on the source domain and the target domain, respectively. and represents the cross entropy loss for domain classification. The total loss of the first stage training is and Through the optimization of the first step, the model can decompose the input features into two different parts, and then proceed to the second step of training.

[0111] In the second stage, i.e., the feature orthogonalization stage, the main purpose is to keep the decomposed components orthogonal. Specifically, in this stage, the feature extractors E1 and E2 are first fixed. Then, the feature extractors E2 and E DIR To get F b and F di Next, we use the candidate region generation network RPN and ROI alignment operation to respectively di and F ds Extracting domain-invariant regional features and The key idea of ​​decoupling is to keep the decoupled components independent. The embodiment of the present invention is based on the vector decomposition theory and tries to ensure that the decomposed components are orthogonal, which is equivalent to independent operation. Therefore, the orthogonal loss function L is designed. ⊥ , and there are:

[0112]

[0113]

[0114]

[0115]

[0116] Among them, n, c, h and w represent the number of candidate boxes, the number of channels, height and width of the image respectively. avgpool represents global average pooling, ⊙ represents the multiplication of corresponding elements, and by minimizing the orthogonal loss, the separated F di and F ds is independent. ds contains more domain-specific information, then this loss function will encourage F di Contains more domain invariant information. In addition, the candidate region features are used in the embodiment of the present invention and To calculate the orthogonal loss, rather than the entire feature F di and F ds , which not only reduces the amount of calculation, but also enables the model to focus more on the target area.

[0117] Finally, the loss of the second stage can be expressed as follows:

[0118]

[0119]

[0120] in, Indicates based on A di The detection loss, and denote the orthogonal losses on the source domain and the target domain respectively. The total loss of the second stage training is and After the second stage of training, the decomposed domain-invariant features and domain-specific features remain independent, which enhances the decoupling ability of the model.

[0121] Fig.10 is a schematic diagram of a model training algorithm provided by an embodiment of the present invention. In the embodiment of the present invention, the above model can be trained end-to-end. The training details are as follows: Fig.10 As shown. Among them, Fig.10 In the algorithm shown, Eq. (6) represents the above formula (3) and formula (4), and Eq. (8) represents the above formula (9) and formula (10).

[0122] Thus, the target detection method provided in the embodiment of the present invention performs feature decoupling based on vector decomposition. Specifically, an operation of vector orthogonal decomposition is designed to decouple image features into representations of domain-invariant features and domain-specific features. Compared with the traditional decoupling method, the scheme in the embodiment of the present invention does not rely on reconstruction operations to ensure that the decomposed components contain all the input information. And only one feature extractor is defined to extract domain-invariant features, and then a complementary second decoupled component is obtained as a domain-specific feature through a differential operation, which reduces parameters and calculations.

[0123] At the same time, the solution of the present invention applies feature decoupling to the domain adaptive target detection method. The traditional distribution alignment method is performed on the overall feature representation of the image, and therefore cannot be applied to target detection tasks that focus on the instance feature level. Domain adaptive target detection needs to narrow the gap between the source domain and the target domain at the instance feature level. The solution of the present invention focuses on feature extraction at the instance level of domain invariance, which is very helpful for improving the migration ability of the detection model. After decoupling the domain invariant features, the subsequent modules complete the detection task on the image based on the domain invariant features. Since the domain invariant features do not contain domain-specific information, they can better solve the migration problem between the two domains.

[0124] Furthermore, the target detection method proposed in the solution of the present invention is suitable for all-weather cross-domain target detection. It can achieve accurate target recognition and positioning under various weather and lighting conditions. Fig.11 This is a schematic diagram of the detection performance comparison results of each target detection method provided by an embodiment of the present invention when migrating from daytime to foggy scenes, where "SourceOnly" means that the model is trained using only data from the source domain, and does not use data from the target domain. "DAF" represents the domain adaptive target detection method in the prior art, and "SW(B)" represents the strong and weak distribution alignment method. It can be seen that the method corresponding to the solution of the present invention achieves the best performance.

[0125] Exemplary Devices

[0126] like Fig.12 As shown in , corresponding to the above target detection method, an embodiment of the present invention further provides a target detection device, and the above target detection device includes:

[0127] The image acquisition module 610 is used to acquire a source domain image and a target domain image.

[0128] It is necessary to perform domain adaptive object detection on the target domain image based on the source domain image and the target domain image. Specifically, in this embodiment, there may be domain deviation between the source domain image and the target domain image. For example, the source domain image may be a daytime image, and the target domain image may correspond to a night image or a foggy image, which is not specifically limited here.

[0129] The image feature extraction module 620 is used to extract features from the source domain image and the target domain image to obtain image features.

[0130] Optionally, feature extraction may be performed on the source domain image and the target domain image respectively to obtain image features corresponding to the source domain image and image features corresponding to the target domain image respectively.

[0131] The domain invariant feature acquisition module 630 is used to acquire domain invariant features based on the above image features.

[0132] Among them, the above-mentioned domain invariant features refer to features related to the attributes of the detection target, and are independent of attributes such as image style, background, and lighting.

[0133] The domain invariant region feature acquisition module 640 is used to acquire the domain invariant region feature based on the above domain invariant feature.

[0134] The detection module 650 is used to perform target detection based on the domain-invariant region features.

[0135] The domain-invariant region feature is a feature of a region corresponding to the domain-invariant feature. For example, the domain-invariant region may be a region where the detection target is located. The domain-invariant region feature is a feature of the region, that is, a domain-invariant instance feature. Target detection based on the domain-invariant region feature is target detection based on the domain-invariant instance feature.

[0136] As can be seen from the above, the target detection device provided by the embodiment of the present invention acquires the source domain image and the target domain image through the image acquisition module 610; extracts features from the above source domain image and the target domain image through the image feature extraction module 620 to acquire image features; acquires domain invariant features based on the above image features through the domain invariant feature acquisition module 630; acquires domain invariant region features based on the above domain invariant features through the domain invariant region feature acquisition module 640; and performs target detection based on the above domain invariant region features through the detection module 650. Among them, the above domain invariant region features are domain invariant instance features. In this way, compared with the prior art, the scheme of the present invention takes into account the instance-level representation that is very important for target detection, and can therefore be used for target detection tasks that focus on the instance feature level, and is beneficial to improving the accuracy and migration capability of the target detection method.

[0137] Specifically, in this embodiment, the image feature extraction module 620 is specifically used to: extract features of the source domain image and the target domain image using a feature extractor; and use the extracted features as image features.

[0138] Optionally, feature extraction may be performed on the source domain image and the target domain image respectively to obtain image features corresponding to the source domain image and image features corresponding to the target domain image respectively.

[0139] exist Figure 4 In the application scenario shown, the input source domain image x is obtained s , label y s , bounding box b s And the target domain image x t For the source domain image x s and the target domain image x t First, the feature extractors E1 and E2 composed of two layers of networks are used to extract the image features F of the source domain and the target domain images. b Among them, the image feature F b The size of F is c1*h1*w1, where c1, h1, and w1 represent the number of channels, height, and width of the feature, respectively. l It is the feature extracted by feature extractor E1. After being processed by gradient reversal layer and domain classifier, it can be used to reduce the domain deviation between source domain and target domain. b =E2(F l ).

[0140] Specifically, in this embodiment, the domain-invariant feature acquisition module 630 is specifically used to: input the image features into an extractor for extracting domain-invariant features, and extract and output the domain-invariant features through the extractor for extracting domain-invariant features.

[0141] Among them, the extractor for extracting domain-invariant features can be a feature extractor based on a multi-layer convolutional network. By inputting image features into the feature extractor, domain-invariant features can be extracted.

[0142] exist Figure 4 In the application scenario shown, the image feature F b Input to the extractor (multi-layer convolutional network) E for extracting domain-invariant features DIR In the example, the output is the domain invariant feature F di , where the domain invariant feature F di The size of F is c1*h1*w1, di =E DIR (F b ).

[0143] For domain adaptive object detection, it is very important to obtain domain invariant representation, which not only can alleviate the impact of domain bias, but also helps to obtain domain invariant object representation. Therefore, this scheme focuses on extracting domain invariant instance-level features, which helps to improve the migration ability of detection methods.

[0144] Specifically, in this embodiment, the target detection device is further used to: subtract the domain-invariant feature from the image feature to obtain the domain-specific feature.

[0145] Optionally, there may be other methods for obtaining domain-specific features, such as directly extracting domain-specific features, etc., which are not specifically limited here. The above-mentioned domain-specific features refer to features such as image style and background that are irrelevant to the detection target. Removing these features can improve the generalization performance of the model and improve the accuracy and migration ability of the target detection method.

[0146] exist Figure 4 In the application scenario shown, the image feature F b Subtract the domain invariant feature F di Get domain-specific features F ds , where the domain-specific feature F ds The size of F is c1*h1*w1, ds =F b -F di .

[0147] Optionally, a domain classifier can be used to ensure that domain-specific features F ds contains more domain-specific information. Furthermore, in order to reduce domain deviation and improve decomposition ability, image features F can be classified based on domain classifiers. b and domain invariant feature F di The model is trained to maximize the domain classification loss, thereby narrowing the gap between the source domain and the target domain. Figure 4 In the application scenario shown, the feature F l After the gradient reversal layer, the domain classifier is used to perform domain classification and the image feature F b After the gradient reversal layer, the domain classifier is used for domain classification. l The size is c0*h0*w0, where c0, h0, and w0 represent the number of channels, height, and width of the feature, respectively.

[0148] Specifically, in this embodiment, the target detection device is further used to: limit the orthogonality of the domain-specific features and the domain-invariant features.

[0149] A key step in feature separation is to ensure the independence of domain-invariant features and domain-specific features. Optionally, orthogonal loss can be set to ensure their independence by limiting the orthogonality between domain-invariant features and domain-specific features. Figure 4 In the application scenario shown, the restricted domain invariant feature F di and domain-specific features F ds Orthogonal to each other.

[0150] Optionally, in an embodiment of the present invention, instance features are decomposed by feature decoupling, and feature decoupling is achieved based on vector decomposition, and the decomposed vectors (i.e., corresponding features) are restricted to be orthogonal. The purpose of feature decoupling is to obtain a set of independent representations that can fully express the information of the original features. In an embodiment of the present invention, feature decoupling is converted into a vector decomposition process. The specific feature decoupling process is shown above and will not be repeated here.

[0151] Specifically, in this embodiment, the domain-invariant region feature acquisition module 640 is specifically used to: obtain a domain-invariant candidate region based on the domain-invariant feature extraction through a candidate region generation network; perform feature alignment based on the domain-invariant candidate region to obtain domain-invariant region features.

[0152] Among them, the above-mentioned candidate region generation network (RPN, Region Proposal Network) is a pre-set and trained network for generating candidate regions, and the above-mentioned domain-invariant candidate regions are candidate regions corresponding to domain-invariant features generated by the candidate region generation network, which can be candidate regions corresponding to detection targets.

[0153] exist Figure 4 In the application scenario shown, the domain invariant feature F di In the input target detector, the candidate region generation network (RPN) generates domain-invariant features F di The domain-invariant candidate region is extracted from the image, and then the ROI alignment operation (RA, ROI-Alignment) is used to obtain the domain-invariant region feature. The domain-invariant region feature is the feature of the region corresponding to the domain-invariant feature. For example, the domain-invariant region can be the region where the detection target is located, and the domain-invariant region feature is the feature of the region, that is, the domain-invariant instance feature.

[0154] Specifically, in this embodiment, the detection module 650 is specifically used to perform a target detection task on the domain-invariant region features, wherein the target detection task includes instance classification and / or detection box regression.

[0155] Optionally, the above target detection task may also include other specific target detection tasks, which are not specifically limited here. Figure 4 In the application scenario shown, the related tasks of target detection, such as instance classification and detection box regression, are performed on the above domain-invariant region features. For a picture from the source domain, the detection loss can be expressed as: Among them, n s represents the number of candidate regions, A j is the result of the ROI alignment operation on the jth candidate region. The detection network R contains classification and regression networks. det Contains classification loss and detection box regression loss.

[0156] Optionally, in this embodiment, the target detection device is also used to: train a preset target detection model based on a two-stage deep network training mechanism. Wherein, the target detection model is a pre-set model for target detection based on the target detection method. Specifically, the two-stage deep network training mechanism includes training in the feature decomposition stage and training in the feature orthogonalization stage. The specific training process is similar to the process shown above and will not be repeated here.

[0157] Based on the above embodiments, the present invention further provides an intelligent terminal, whose principle block diagram can be shown as follows: Fig.13 As shown. The above-mentioned intelligent terminal includes a processor, a memory, a network interface and a display screen connected through a system bus. Among them, the processor of the intelligent terminal is used to provide computing and control capabilities. The memory of the intelligent terminal includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a target detection program. The internal memory provides an environment for the operation of the operating system and the target detection program in the non-volatile storage medium. The network interface of the intelligent terminal is used to communicate with an external terminal through a network connection. When the target detection program is executed by the processor, the steps of any one of the above-mentioned target detection methods are implemented. The display screen of the intelligent terminal can be a liquid crystal display screen or an electronic ink display screen.

[0158] Those skilled in the art will understand that Fig.13 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the scheme of the present invention, and does not constitute a limitation on the smart terminal to which the scheme of the present invention is applied. The specific smart terminal may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0159] In one embodiment, a smart terminal is provided. The smart terminal includes a memory, a processor, and a target detection program stored in the memory and executable on the processor. When the target detection program is executed by the processor, the following operation instructions are performed:

[0160] Obtain source domain images and target domain images;

[0161] Perform feature extraction on the above-mentioned source domain image and target domain image to obtain image features;

[0162] Based on the above image features, domain invariant features are obtained;

[0163] Obtaining domain-invariant region features based on the above domain-invariant features;

[0164] Object detection is performed based on the above domain-invariant region features.

[0165] It should be understood that the serial numbers of the steps in the above embodiments do not imply a sequence of execution, and the execution sequence of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0166] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the above-mentioned device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0167] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0168] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0169] In the embodiments provided by the present invention, it should be understood that the disclosed apparatus / terminal equipment and method can be implemented in other ways. For example, the apparatus / terminal equipment embodiments described above are only illustrative, for example, the division of the above modules or units is only a logical function division, and in actual implementation, other division methods can be used, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0170] If the above-mentioned integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The above-mentioned computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the above-mentioned computer program includes computer program code, and the above-mentioned computer program code can be in source code form, object code form, executable file or some intermediate form. The above-mentioned computer-readable medium may include: any entity or device capable of carrying the above-mentioned computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the above-mentioned computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction.

[0171] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, it should be understood by those skilled in the art that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features therein may be replaced by equivalents. However, these modifications or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.

Claims

1. A target detection method, characterized in that: The method comprises: Obtain source domain images and target domain images; Performing feature extraction on the source domain image and the target domain image to obtain image features; Based on the image features, obtaining domain invariant features; Acquire domain-invariant region features based on the domain-invariant features; Performing target detection based on the domain invariant region features; The acquiring of the source domain image and the target domain image comprises: The training is performed based on a deep network training mechanism of a feature decomposition stage and a feature orthogonalization stage; wherein the feature orthogonalization stage includes: Fixed feature extractor and , using the feature extractor and extractors for extracting domain-invariant features Get image features Domain-invariant features ; Use the candidate region generation network RPN and ROI alignment operation to respectively generate domain invariant features and domain-specific features Extracting domain-invariant regional features and , using domain-invariant region features and Calculating an orthogonal loss, and limiting the domain-specific features and the domain-invariant features to be orthogonal by minimizing the orthogonal loss; The acquiring of domain invariant features based on the image features includes: Inputting the image features into an extractor for extracting domain-invariant features, and extracting and outputting domain-invariant features through the extractor for extracting domain-invariant features; After acquiring domain invariant features based on the image features, the method further includes: Subtracting the domain-invariant feature from the image feature to obtain a domain-specific feature; the domain-specific feature is a feature that is irrelevant to the detection target; After subtracting the domain-invariant feature from the image feature to obtain the domain-specific feature, the object detection method further includes: The domain-specific features and the domain-invariant features are constrained to be orthogonal.

2. The target detection method according to claim 1, characterized in that: The extracting features of the source domain image and the target domain image to obtain image features includes: Extracting features of the source domain image and the target domain image using a feature extractor; The extracted features are used as image features.

3. The target detection method according to claim 1, characterized in that: The acquiring of domain-invariant region features based on the domain-invariant features comprises: Extracting a domain-invariant candidate region based on the domain-invariant features through a candidate region generation network; Feature alignment is performed based on the domain-invariant candidate regions to obtain domain-invariant region features.

4. The target detection method according to claim 3, characterized in that: The performing target detection based on the domain invariant region features comprises: An object detection task is performed on the domain-invariant region features, wherein the object detection task includes instance classification and / or detection box regression.

5. A target detection device, used to implement the target detection method according to any one of claims 1 to 4, characterized in that: The device comprises: An image acquisition module, used to acquire source domain images and target domain images; An image feature extraction module is used to extract features from the source domain image and the target domain image to obtain image features; A domain invariant feature acquisition module, used for acquiring domain invariant features based on the image features; A domain invariant region feature acquisition module, used for acquiring domain invariant region features based on the domain invariant features; A detection module is used to perform target detection based on the domain invariant region features.

6. An intelligent terminal, characterized in that: The intelligent terminal includes a memory, a processor, and a target detection program stored in the memory and executable on the processor. When the target detection program is executed by the processor, the steps of the target detection method according to any one of claims 1 to 4 are implemented.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a target detection program, and when the target detection program is executed by the processor, the steps of the target detection method according to any one of claims 1 to 4 are implemented.

Images