Target detection method and device based on text-guided visual domain transformation

Abstract

The invention relates to an image processing technology, and discloses a target detection method based on text guide visual domain transformation, which comprises the following steps: carrying out text coding on a source domain text cue word and a target domain text cue word to obtain a source domain text code and a target domain text code; acquiring an image sample and extracting original features of the sample; performing feature transformation on the original sample features by using a domain transformation model to obtain domain transformation features; performing deep feature coding on the sample original features and the domain transformation features to obtain sample deep features and domain transformation deep features; and performing inter-domain direction alignment by using domain transformation direction consistency loss to obtain optimized domain transformation features, and performing optimization training on a pre-trained target detection model in combination with original sample features. The invention further provides a target detection device based on text-guided visual domain transformation, electronic equipment and a storage medium. According to the invention, the accuracy of target detection and the detection precision of target detection in a new field environment can be improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to an object detection method and device based on text-guided visual domain transformation. Background Art

[0002] In recent years, driven by deep learning, significant progress has been made in object detection. However, the success of these methods stems from the assumption of independent and identically distributed data, where the training and test data belong to the same distribution. When applying these methods to data with a domain shift relative to the training data, their performance may decline significantly. A direct way to solve this problem is to incorporate data from other distributions into the model training process. Although effective, this method is inefficient and impractical because collecting and annotating data from different distributions is very time-consuming, and it is infeasible to consider all possible distributions encountered in the real world.

[0003] To solve this problem, Domain Adaptive Object Detection (DAOD) proposes using labeled source domains and unlabeled target domains to align the two domains to adapt the model. However, the requirement of predefining the target domain limits the applicability of DAOD in real scenarios. To address this limitation, Single-source Domain Generalized Object Detection (SDGOD) is proposed, which uses only a single source domain to train the model to enable it to generalize to all unknown domains. However, the core challenge of SDGOD lies in how to improve the adaptability and robustness of the model to target domain changes (such as environment, lighting, background, perspective, etc.). In SDGOD, the main technical problem faced is the domain shift between domains. The object detection model is usually trained on a specific source domain (for example, outdoor scenes during the day), and applied to a different target domain during testing (for example, outdoor scenes at night, foggy scenes), but this distribution difference between the source domain and the target domain causes the object detection model to perform poorly on the target domain. Summary of the Invention

[0004] The present invention provides an object detection method and device based on text-guided visual domain transformation, which can improve the accuracy of object detection and the detection accuracy in a new domain environment.

[0005] To achieve the above object, an object detection method based on text-guided visual domain transformation provided by the present invention includes:

[0006] Obtain the source domain text prompt and the target domain text prompt, and use the text encoder to perform text encoding on the source domain text prompt and the target domain text prompt respectively to obtain the source domain text encoding and the target domain text encoding;

[0007] Obtain an image sample, and use a shallow image encoder to extract the original features of the image sample to obtain the sample original features;

[0008] Perform feature transformation on the sample original features using a domain transformation model to obtain domain transformation features;

[0009] Input the sample original features and the domain transformation features into a deep image encoder respectively for deep feature encoding to obtain the sample deep features and the domain transformation deep features;

[0010] Use the domain transformation direction consistency loss to perform inter-domain direction alignment on the source domain text encoding, the target domain text encoding, the sample deep features, and the domain transformation deep features to obtain optimized domain transformation features, and combine the sample original features and the optimized domain transformation features to optimize and train a pre-trained object detection model. After the training is completed, use the original standard detection process of the model to obtain the object detection result.

[0011] Optionally, the using the text encoder to perform text encoding on the source domain text prompt and the target domain text prompt respectively includes:

[0012] Use the source domain text prompt and the target domain text prompt to form multiple groups of domain text pairs;

[0013] Use a shallow text encoder to encode the multiple groups of domain text pairs to obtain multiple groups of embedded text pairs composed of source domain text embeddings and target domain text embeddings, where the parameters of the shallow text encoder are initialized by a pre-trained CLIP model;

[0014] Calculate the difference between the source domain text embedding and the target domain text embedding in the embedded text pair, and obtain the domain transformation direction of the language modality through the difference.

[0015] Optionally, use the domain transformation direction of the language modality to guide the generation of the domain transformation direction of the visual modality.

[0016] Optionally, the performing feature transformation on the sample original features using a domain transformation model to obtain domain transformation features includes:

[0017] Calculate the mean and standard deviation of the sample original features in the feature length dimension and the feature width dimension based on the channel number dimension;

[0018] Use the domain transformation parameters in the domain transformation model to perform parameter calculation on the mean and standard deviation to obtain the updated mean and the updated standard deviation;

[0019] Update the mean and standard deviation in the original features of the sample using the updated mean and updated standard deviation to obtain domain transformation features.

[0020] Optionally, inputting the original features of the sample and the domain transformation features into a deep image encoder respectively for deep feature encoding to obtain the deep features of the sample and the deep features of the domain transformation, includes:

[0021] Input the original features of the sample into the domain transformation model for domain transformation to obtain multiple groups of domain transformation features;

[0022] Input the original features of the sample and each group of domain transformation features into the deep image encoder respectively to obtain the deep features of the sample and the deep features of the domain transformation.

[0023] Optionally, calculate the difference between the deep features of the sample and the deep features of the domain transformation to obtain the domain transformation direction of the visual modality.

[0024] Optionally, the domain - to - domain direction alignment of the source - domain text encoding, target - domain text encoding, deep features of the sample, and deep features of the domain transformation using the domain transformation direction consistency loss includes:

[0025] Align the domain transformation direction of the visual modality and the domain transformation direction of the language modality by using the consistency loss calculated between the domain transformation direction of the visual modality and the domain transformation direction of the language modality. Among them, the consistency loss L dc Adopt the following formula:

[0026]

[0027] Among them, is the domain transformation direction of the visual modality of the i - th group of domain - text pairs, is the domain transformation direction of the language modality of the i - th group of domain - text pairs, i is the number of groups of domain - text pairs, and ||·|| is the modulo calculation.

[0028] To solve the above problems, the present invention also provides an object - detection device based on text - guided visual domain transformation. The device includes:

[0029] A feature extraction module, configured to obtain a source - domain text prompt and a target - domain text prompt, use a text encoder to perform text encoding on the source - domain text prompt and the target - domain text prompt respectively to obtain a source - domain text encoding and a target - domain text encoding; obtain an image sample, and use a shallow image encoder to extract the original features of the image sample to obtain the original features of the sample;

[0030] A feature transformation module, configured to perform feature transformation on the original features of the samples by using a domain transformation model to obtain domain-transformed features; input the original features of the samples and the domain-transformed features into a deep image encoder respectively to perform deep feature encoding to obtain sample deep features and domain-transformed deep features;

[0031] An object detection module, configured to perform inter-domain direction alignment on the source domain text encoding, target domain text encoding, sample deep features, and domain-transformed deep features by using a domain transformation direction consistency loss to obtain optimized domain-transformed features, and optimize and train a pre-trained object detection model by combining the original features of the samples and the optimized domain-transformed features, and after the training is completed, obtain the object detection result by using the original standard detection process of the model.

[0032] To solve the above problems, the present invention also provides an electronic device, which includes:

[0033] At least one processor; and,

[0034] A memory communicatively connected to the at least one processor; wherein,

[0035] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the above-mentioned object detection method based on text-guided visual domain transformation.

[0036] To solve the above problems, the present invention also provides a computer-readable storage medium, in which at least one computer program is stored, and the at least one computer program is executed by a processor in an electronic device to implement the above-mentioned object detection method based on text-guided visual domain transformation.

[0037] The present invention can establish a connection between the source domain and the target domain at the semantic level by obtaining the text prompts of the source domain and the target domain and encoding them respectively. In addition, by using a shallow image encoder to extract the original features of the image samples, the basic visual features of the image can be extracted and the original information can be provided for subsequent deep encoding. Furthermore, by performing feature transformation on the original features of the samples through a domain transformation model, the feature distributions of the source domain and the target domain can be made consistent. Moreover, by using the domain transformation direction consistency loss to perform inter-domain direction alignment on the source domain text encoding, target domain text encoding, sample deep features, and domain-transformed deep features, the inter-domain direction alignment of the features can enable the object detection model to adapt more quickly and improve the detection performance when facing new domain data, and can enable the model to utilize both semantic features and visual features at the same time, improving the accuracy of object detection and the detection accuracy of object detection in a new domain environment. Description of the Drawings

[0038] Figure 1 A flowchart of a target detection method based on text-guided visual domain transformation provided by an embodiment of the present invention;

[0039] Figure 2 A structural flowchart of an example of a target detection method based on text-guided visual domain transformation provided by an embodiment of the present invention;

[0040] Figure 3 A functional module diagram of a target detection device based on text-guided visual domain transformation provided by an embodiment of the present invention;

[0041] Figure 4 A structural diagram of an electronic device for implementing the target detection method based on text-guided visual domain transformation provided by an embodiment of the present invention.

[0042] The realization, functional characteristics, and advantages of the present invention will be further described with reference to the embodiments and the accompanying drawings. Detailed Embodiments

[0043] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0044] An embodiment of the present application provides a target detection method based on text-guided visual domain transformation. The execution subject of the target detection method based on text-guided visual domain transformation includes, but is not limited to, at least one of electronic devices such as a server, a terminal, etc. that can be configured to execute the method provided by the embodiment of the present application. In other words, the target detection method based on text-guided visual domain transformation can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes, but is not limited to: a single server, a server cluster, a cloud server, or a cloud server cluster, etc. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, Content Delivery Network (CDN), and big data and artificial intelligence platforms.

[0045] Refer to Figure 1 As shown, it is a flowchart of a target detection method based on text-guided visual domain transformation provided by an embodiment of the present invention. In this embodiment, the target detection method based on text-guided visual domain transformation includes:

[0046] S1. Obtain the source domain text prompt and the target domain text prompt, and use a text encoder to perform text encoding on the source domain text prompt and the target domain text prompt respectively to obtain the source domain text encoding and the target domain text encoding.

[0047] In the embodiments of the present invention, the source domain text prompt is the text used to describe the source domain image. For example, if the source domain image is an image under sunny weather, its source domain text prompt includes but is not limited to "a fog-free image, a snow-free image, a rain-free image", etc.

[0048] In the embodiments of the present invention, the target domain text prompt is the text used to describe the target domain image. For example, if the target domain image is an image collected under different weather conditions, its target domain text prompt includes but is not limited to "a foggy image, a snowy image, a rainy image", etc.

[0049] As an embodiment of the present invention, the step of using a text encoder to perform text encoding on the source domain text prompt and the target domain text prompt respectively includes:

[0050] Form multiple groups of domain text pairs using the source domain text prompt and the target domain text prompt;

[0051] Use a shallow text encoder to encode the multiple groups of domain text pairs to obtain multiple groups of embedded text pairs composed of source domain text embeddings and target domain text embeddings, where the parameters of the shallow text encoder are initialized by the parameters of a pre-trained CLIP model;

[0052] Calculate the difference between the source domain text embedding and the target domain text embedding in the embedded text pair, and obtain the domain transformation direction of the language modality through the difference.

[0053] In the embodiments of the present invention, the pre-trained CLIP (Contrastive Language-Image Pre-Training) model is a multi-modal contrastive learning model proposed by OpenAI, which consists of a text encoder and an image encoder, aiming to establish the association between vision and text semantics through natural language supervision signals. The parameters of the text encoder are initialized and frozen by the text encoder of the CLIP model.

[0054] Furthermore, use the domain transformation direction of the language modality to guide the generation of the domain transformation direction of the visual modality.

[0055] S2. Obtain an image sample, and use a shallow image encoder to extract the original features of the image sample to obtain the sample original features.

[0056] In the embodiments of the present invention, the image sample refers to an image collected with obvious features.

[0057] Exemplarily, the steps for extracting the original features of an image sample using a shallow image encoder to obtain the original sample features may be as follows:

[0058] For a given image sample x, it passes through the shallow image encoder I of the image encoder e1 Extract the shallow features of the image: f = I e1 (x), where the image encoder I e Consists of a shallow structure I e1 And a deep structure I e2 Composed, and there is also a learnable domain transformation model T at the middle position of the image encoder I e The domain transformation model T contains multiple groups of learnable domain transformation parameters.

[0059] In the embodiments of the present invention, the number of groups of domain transformation parameters in the domain transformation model T is the same as the number of groups of domain text pairs, and the number of groups of domain transformation parameters in the domain transformation model T changes correspondingly with the change in the number of groups of domain text pairs. For example, when the number of groups of domain text pairs increases by one group, the number of groups of domain transformation parameters in the domain transformation model T also increases by one group immediately.

[0060] S3. Use the domain transformation model to perform feature transformation on the original sample features to obtain domain transformation features.

[0061] As an embodiment of the present invention, the step of using the domain transformation model to perform feature transformation on the original sample features to obtain domain transformation features includes:

[0062] Calculate the mean and standard deviation of the original sample features based on the channel number dimension in the feature length dimension and the feature width dimension;

[0063] Use the domain transformation parameters in the domain transformation model to perform parameter calculation on the mean and standard deviation to obtain the updated mean and updated standard deviation;

[0064] Use the updated mean and updated standard deviation to update the mean and standard deviation in the original sample features to obtain domain transformation features.

[0065] Exemplarily, the working implementation process of the domain transformation model may be as shown in the following implementation steps, where the domain transformation model T contains K groups of learnable domain transformation parameters {α i , β i}, where α i Is the first parameter in the i-th group of domain transformation parameters, and β i Is the second parameter in the i-th group of domain transformation parameters:

[0066] a. Calculate the mean μ and standard deviation σ of the input original sample features f:

[0067] Where, Let \(B\) be the size of the training batch, \(C\) be the number of channels, and \(H\) and \(W\) be the length and width of the feature. The mean \(\mu\) and standard deviation \(\sigma\) are calculated over the \(H\) and \(W\) dimensions. Taking the mean as an example, for each channel \(C\), the mean is obtained by averaging all spatial positions (i.e., \(H\times W\) pixels) of that channel, that is

[0068] b. Introduce \(K\) sets of learnable domain transformation parameters \(\{\alpha i ,\beta i \}\), where \(K\) is a positive integer. And use them to obtain the updated mean and standard deviation where \(i\in[1,K]\), and the parameters \(t\) in the mean and standard deviation indicate that the mean and standard deviation are in the updated state:

[0069]

[0070] where \(\odot\) represents the Hadamard product (also known as the element-wise product), which is an element-wise multiplication operation between two matrices or tensors. is the updated standard deviation, is the updated mean. Different from traditional matrix multiplication, the Hadamard product multiplies elements corresponding to each other, rather than involving weighted sums of rows and columns like ordinary matrix multiplication. According to the broadcasting mechanism of Python, when calculating the above formula, will be automatically replicated \(B\) times to become and can then be multiplied element-wise with .

[0071] c. Based on the input feature \(f\), obtain the domain transformation feature by changing the mean and standard deviation of the feature (using the updated mean and standard deviation to replace the original mean and standard deviation)

[0072]

[0073] S4. Input the original sample feature and the domain transformation feature into the deep image encoder respectively to perform deep feature encoding to obtain the sample deep feature and the domain transformation deep feature.

[0074] As an embodiment of the present invention, the step of inputting the original sample feature and the domain transformation feature into the deep image encoder respectively to perform deep feature encoding to obtain the sample deep feature and the domain transformation deep feature includes:

[0075] Input the original sample feature into the domain transformation model for domain transformation to obtain multiple sets of domain transformation features;

[0076] The original features of the samples and the domain transformation features of each group are respectively input into a deep image encoder to obtain the deep features of the samples and the deep features of the domain transformation.

[0077] Further, calculate the difference between the deep features of the samples and the deep features of the domain transformation to obtain the domain transformation direction of the visual modality.

[0078] Exemplarily, the following implementation steps can be adopted to calculate the difference between the output features of the deep features of the samples and the deep features of the domain transformation to obtain the domain transformation direction of the visual modality:

[0079] Send the original features of the samples into the domain transformation model T for domain transformation to obtain the domain transformation features: where i ∈ [1, K], because the module T contains K groups of learnable domain transformation parameters, and one group of parameters can generate one kind of transformation feature. Then, the original features F of the samples and the domain transformation features are sent into the deep image encoder I of the subsequent image encoder e2 , to obtain the output features of the final image encoder: F s = I e2 (f) and where, F s is the feature after the original features of the samples are encoded by the deep image encoder, is the feature after the domain transformation features are encoded by the deep image encoder. Take the difference between the two as the domain transformation direction of the visual modality:

[0080] S5. Use the domain transformation direction consistency loss to perform inter-domain direction alignment on the source domain text encoding, target domain text encoding, deep features of the samples, and deep features of the domain transformation to obtain optimized domain transformation features, and combine the original features of the samples and the optimized domain transformation features to optimize and train the pre-trained object detection model. After the training is completed, use the original standard detection process of the model to obtain the object detection results.

[0081] In the embodiment of the present invention, the pre-trained object detection model refers to the Faster R-CNN model. The Faster R-CNN model is an object detection model that realizes the deep fusion of candidate box generation and object detection through RPN, and the weights of its image encoder are initialized by the network pre-trained on the ImageNet dataset.

[0082] As an embodiment of the present invention, the use of the domain transformation direction consistency loss to perform inter-domain direction alignment on the source domain text encoding, target domain text encoding, deep features of the samples, and deep features of the domain transformation includes:

[0083] Align the domain transformation direction of the visual modality with the domain transformation direction of the language modality by using the consistency loss between the domain transformation direction of the computational visual modality and the domain transformation direction of the language modality, where the consistency loss L between the domain transformation direction of the computational visual modality and the domain transformation direction of the language modality dc Adopt the following formula:

[0084]

[0085] Wherein, Is the domain transformation direction of the visual modality of the i-th group of domain text pairs, Is the domain transformation direction of the language modality of the i-th group of domain text pairs, i is the number of groups of domain text pairs, and ||·|| is the modulo calculation.

[0086] In the present invention, by obtaining the text prompts of the source domain and the target domain and encoding them respectively, the connection between the source domain and the target domain can be established at the semantic level. In addition, by using the shallow image encoder to extract the original features of the image samples, the basic visual features of the image can be extracted and the original information can be provided for the subsequent deep encoding. In addition, by using the domain transformation model to perform feature transformation on the sample original features, the feature distributions of the source domain and the target domain can be made consistent. Furthermore, by using the domain transformation direction consistency loss to perform inter-domain direction alignment on the source domain text encoding, the target domain text encoding, the sample deep features, and the domain transformation deep features, the inter-domain direction alignment of the features can enable the target detection model to adapt more quickly and improve the detection performance when the model faces new domain data, and can enable the model to utilize both semantic features and visual features at the same time, improving the accuracy of target detection and the detection accuracy of target detection in the new domain environment.

[0087] Refer to Figure 2 As shown, it is a structural flowchart of an example of a target detection method based on text-guided visual domain transformation provided by an embodiment of the present invention.

[0088] As Figure 3 Shown, it is a functional module diagram of a target detection device based on text-guided visual domain transformation provided by an embodiment of the present invention.

[0089] The target detection device 100 based on text-guided visual domain transformation described in the present invention can be installed in an electronic device. According to the implemented functions, the target detection device 100 based on text-guided visual domain transformation can include a feature extraction module 101, a feature transformation module 102, and a target detection module 103.

[0090] The module described in the present invention can also be referred to as a unit, which refers to a series of computer program segments that can be executed by the processor of an electronic device and can complete fixed functions, and are stored in the memory of the electronic device.

[0091] In this embodiment, the functions of each module / unit are as follows:

[0092] The feature extraction module 101 is configured to obtain a source domain text prompt and a target domain text prompt, respectively perform text encoding on the source domain text prompt and the target domain text prompt by using a text encoder to obtain a source domain text encoding and a target domain text encoding; obtain an image sample, and extract an original feature of the image sample by using a shallow image encoder to obtain a sample original feature.

[0093] In the embodiment of the present invention, the source domain text prompt is text for describing a source domain image. For example, if the source domain image is an image under sunny weather, its source domain text prompt includes but is not limited to "a fog-free image, a snow-free image, a rain-free image", etc.

[0094] In the embodiment of the present invention, the target domain text prompt is text for describing a target domain image. For example, if the target domain image is an image collected under different weather conditions, its target domain text prompt includes but is not limited to "a foggy image, a snowy image, a rainy image", etc.

[0095] As an embodiment of the present invention, the step of respectively performing text encoding on the source domain text prompt and the target domain text prompt by using a text encoder includes:

[0096] Combining the source domain text prompt and the target domain text prompt to form multiple groups of domain text pairs;

[0097] Encoding the multiple groups of domain text pairs by using a shallow text encoder to obtain multiple groups of embedded text pairs composed of a source domain text embedding and a target domain text embedding, wherein the parameters of the shallow text encoder are initialized by the text encoder of a pre-trained CLIP model;

[0098] Calculating the difference between the source domain text embedding and the target domain text embedding in the embedded text pair, and obtaining the domain transformation direction of the language modality through the difference.

[0099] In the embodiment of the present invention, the pre-trained CLIP (Contrastive Language-Image Pre-Training) model is a multi-modal contrastive learning model proposed by OpenAI, which is jointly composed of a text encoder and an image encoder, aiming to establish the association between vision and text semantics through natural language supervision signals, and the parameters of the text encoder are initialized and frozen by the text encoder of the CLIP model.

[0100] Further, the domain transformation direction of the visual modality is generated by guiding with the domain transformation direction of the language modality.

[0101] In the embodiments of the present invention, an image sample refers to an image collected with obvious features.

[0102] Exemplarily, the following implementation steps can be adopted to extract the original features of the image sample by using the shallow image encoder to obtain the sample original features:

[0103] For a given image sample x, through the shallow image encoder I of the image encoder e1 Extract the shallow features of the image: f = I e1 (x), where the image encoder I e Consists of a shallow structure I e1 And a deep structure I e2 Composed, the middle position of the image encoder I e Also includes a learnable domain transformation model T, and the domain transformation model T contains multiple groups of learnable domain transformation parameters.

[0104] In the embodiments of the present invention, the number of groups of domain transformation parameters in the domain transformation model T is the same as the number of groups of domain text pairs, and the number of groups of domain transformation parameters in the domain transformation model T changes correspondingly with the change of the number of groups of domain text pairs. For example, when the number of groups of domain text pairs increases by one group, the number of groups of domain transformation parameters in the domain transformation model T also increases by one group immediately.

[0105] The feature transformation module 102 is used to perform feature transformation on the sample original features by using the domain transformation model to obtain domain transformation features; input the sample original features and the domain transformation features into the deep image encoder respectively for deep feature encoding to obtain sample deep features and domain transformation deep features.

[0106] As an embodiment of the present invention, the performing feature transformation on the sample original features by using the domain transformation model to obtain domain transformation features includes:

[0107] Calculate the mean and standard deviation of the sample original features based on the channel number dimension in the feature length dimension and the feature width dimension;

[0108] Use the domain transformation parameters in the domain transformation model to perform parameter calculation on the mean and standard deviation to obtain the updated mean and updated standard deviation;

[0109] Use the updated mean and updated standard deviation to update the mean and standard deviation in the sample original features to obtain the domain transformation features.

[0110] Exemplarily, the working implementation process of the domain transformation model can be shown as the following implementation steps, where the domain transformation model T contains K groups of learnable domain transformation parameters {α i , β i}, where α i Is the first parameter in the i-th group of domain transformation parameters, and β iis the second parameter in the domain transformation parameters for the i-th group:

[0111] a. Calculate the mean μ and standard deviation σ of the original features f of the input sample:

[0112] where B is the size of the training batch, C is the number of channels, and H and W are the length and width of the features. The mean μ and standard deviation σ are calculated for the mean and standard deviation in the H and W dimensions. Taking the mean as an example, for each channel C, the mean is obtained by averaging all spatial positions (i.e., H×W pixels) of this channel, that is

[0113] b. Introduce K groups of learnable domain transformation parameters {α i , β i}, where K is a positive integer. And use them to obtain the updated mean and standard deviation on the basis of the original mean μ and standard deviation σ. Where i ∈ [1, K], the parameters t in the mean and standard deviation represent the updated state of the mean and standard deviation:

[0114]

[0115] where ⊙ represents the Hadamard product (also known as the element-wise product), which is an element-wise multiplication operation of two matrices or tensors, is for updating the standard deviation, is for updating the mean. Different from traditional matrix multiplication, the Hadamard product performs multiplication element by element, rather than involving weighted sums of rows and columns like ordinary matrix multiplication. According to the broadcasting mechanism of Python, when calculating the above formula, will be automatically replicated B times to become and can then be multiplied element by element with .

[0116] c. On the basis of the input feature f, obtain the domain transformation feature

[0117]

[0118] by changing the mean and standard deviation of the feature (using the updated mean and standard deviation to replace the original mean and standard deviation). As an embodiment of the present invention, the steps of respectively inputting the original sample feature and the domain transformation feature into a deep image encoder to perform deep feature encoding to obtain the sample deep feature and the domain transformation deep feature include:

[0119] Input the original features of the sample into the domain transformation model for domain transformation to obtain multiple sets of domain-transformed features;

[0120] Input the original features of the sample and each set of domain-transformed features into the deep image encoder respectively to obtain the deep features of the sample and the deep features of the domain transformation.

[0121] Furthermore, calculate the difference between the deep features of the sample and the deep features of the domain transformation to obtain the domain transformation direction of the visual modality.

[0122] Exemplarily, calculating the difference between the output features of the deep features of the sample and the deep features of the domain transformation to obtain the domain transformation direction of the visual modality can adopt the following implementation steps:

[0123] Send the original features of the sample into the domain transformation model T for domain transformation to obtain the domain-transformed features: where i ∈ [1, K], because the module T contains K sets of learnable domain transformation parameters, and one set of parameters can generate one kind of transformed feature. Then send the original features f of the sample and the domain-transformed features into the deep image encoder I of the subsequent image encoder e2 , to obtain the output features of the final image encoder: F s = I e2 (f) and where F s is the feature after encoding the original features of the sample by the deep image encoder, is the feature after encoding the domain-transformed features by the deep image encoder. Take the difference between the two as the domain transformation direction of the visual modality:

[0124] The target detection module 103 is used to perform inter-domain direction alignment on the source domain text encoding, target domain text encoding, deep features of the sample, and deep features of the domain transformation by using the domain transformation direction consistency loss to obtain the optimized domain transformation features, and optimize and train the pre-trained target detection model by combining the original features of the sample and the optimized domain transformation features. After the training is completed, the target detection result is obtained by using the original standard detection process of the model.

[0125] In the embodiment of the present invention, the pre-trained target detection model refers to the Faster R-CNN model. The Faster R-CNN model is a target detection model that realizes the deep fusion of candidate box generation and target detection through RPN, and the weights of its image encoder are initialized by the network pre-trained on the ImageNet dataset.

[0126] As an embodiment of the present invention, the performing inter-domain direction alignment on the source domain text encoding, target domain text encoding, deep features of the sample, and deep features of the domain transformation by using the domain transformation direction consistency loss includes:

[0127] Align the domain transformation direction of the visual modality with the domain transformation direction of the language modality by using the consistency loss between the domain transformation direction of the computational visual modality and the domain transformation direction of the language modality, where the consistency loss L between the domain transformation direction of the computational visual modality and the domain transformation direction of the language modality dc Adopt the following formula:

[0128]

[0129] where is the domain transformation direction of the visual modality of the i-th group of domain text pairs, is the domain transformation direction of the language modality of the i-th group of domain text pairs, i is the number of groups of domain text pairs, and ||·|| is the modulo calculation.

[0130] As Figure 4 shown, it is a schematic structural diagram of an electronic device for implementing an object detection method based on text-guided visual domain transformation provided by an embodiment of the present invention.

[0131] The electronic device may include a processor 10, a memory 11, a communication bus 12, and a communication interface 13, and may also include a computer program stored in the memory 11 and executable on the processor 10, such as an object detection method program based on text-guided visual domain transformation.

[0132] Among them, the processor 10 may be composed of integrated circuits in some embodiments. For example, it may be composed of a single packaged integrated circuit, or may be composed of multiple integrated circuits with the same or different functions, including a combination of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 10 is the control core (ControlUnit) of the electronic device, connects various components of the entire electronic device through various interfaces and lines, and by running or executing programs or modules stored in the memory 11 (such as executing an object detection method program based on text-guided visual domain transformation, etc.), and calling data stored in the memory 11, to perform various functions of the electronic device and process data.

[0133] The memory 11 includes at least one type of readable storage medium, which includes flash memory, mobile hard disk, multimedia card, card-type memory (such as SD or DX memory, etc.), magnetic memory, magnetic disk, optical disk, etc. The memory 11 can be an internal storage unit of the electronic device in some embodiments, such as the mobile hard disk of the electronic device. The memory 11 can also be an external storage device of the electronic device in other embodiments, such as a plug-in mobile hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. equipped on the electronic device. Further, the memory 11 can also include both an internal storage unit and an external storage device of the electronic device. The memory 11 can be used not only to store application software installed on the electronic device and various types of data, such as the code of a target detection method program based on text-guided visual domain transformation, etc., but also to temporarily store data that has been output or will be output.

[0134] The communication bus 12 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. The bus is configured to enable connection communication between the memory 11 and at least one processor 10, etc.

[0135] The communication interface 13 is used for communication between the above-mentioned electronic device and other devices, including a network interface and a user interface. Optionally, the network interface can include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), and is generally used to establish a communication connection between this electronic device and other electronic devices. The user interface can be a display, an input unit (such as a keyboard), and optionally, the user interface can also be a standard wired interface or a wireless interface. Optionally, in some embodiments, the display can be an LED display, a liquid crystal display, a touch liquid crystal display, and an OLED (Organic Light-Emitting Diode) toucher, etc. Among them, the display can also be appropriately referred to as a display screen or a display unit, and is used to display information processed in the electronic device and to display a visual user interface.

[0136] Figure 4 Only the electronic device with components is shown, and those skilled in the art can understand that Figure 4The structures shown do not constitute a limitation on the electronic device, which may include fewer or more components than shown, or combine certain components, or have a different component arrangement.

[0137] For example, although not shown, the electronic device may further include a power source (such as a battery) for powering each component. Preferably, the power source may be logically connected to the at least one processor 10 through a power management device, so as to implement functions such as charging management, discharging management, and power consumption management through the power management device. The power source may also include any components such as one or more DC or AC power sources, a recharge device, a power failure detection circuit, a power converter or inverter, a power status indicator, etc. The electronic device may also include various sensors, a Bluetooth module, a Wi-Fi module, etc., which will not be elaborated here.

[0138] It should be understood that the embodiments are for illustrative purposes only and are not limited by this structure in the scope of the patent application.

[0139] A program of a target detection method based on text-guided visual domain transformation stored in the memory 11 in the electronic device is a combination of multiple instructions. When running in the processor 10, it can implement:

[0140] Obtain a source domain text prompt and a target domain text prompt, and use a text encoder to perform text encoding on the source domain text prompt and the target domain text prompt respectively to obtain a source domain text encoding and a target domain text encoding;

[0141] Obtain an image sample, and use a shallow image encoder to extract the original features of the image sample to obtain sample original features;

[0142] Perform feature transformation on the sample original features using a domain transformation model to obtain domain transformation features;

[0143] Input the sample original features and the domain transformation features into a deep image encoder respectively to perform deep feature encoding to obtain sample deep features and domain transformation deep features;

[0144] Use the domain transformation direction consistency loss to perform inter-domain direction alignment on the source domain text encoding, the target domain text encoding, the sample deep features, and the domain transformation deep features to obtain optimized domain transformation features, and combine the sample original features and the optimized domain transformation features to optimize and train a pre-trained target detection model. After the training is completed, use the original standard detection process of the model to obtain the target detection result.

[0145] Specifically, the specific implementation method of the processor 10 for the above instructions can refer to the description of the relevant steps in the corresponding embodiments of the accompanying drawings, which will not be elaborated here.

[0146] Furthermore, if the modules / units integrated in the electronic device 1 are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, the computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM, Read-Only Memory).

[0147] The present invention also provides a computer-readable storage medium. The readable storage medium stores a computer program. When the computer program is executed by a processor of an electronic device, it can implement:

[0148] Obtain a source domain text prompt and a target domain text prompt, and use a text encoder to perform text encoding on the source domain text prompt and the target domain text prompt respectively to obtain a source domain text encoding and a target domain text encoding;

[0149] Obtain an image sample, and use a shallow image encoder to extract the original features of the image sample to obtain sample original features;

[0150] Use a domain transformation model to perform feature transformation on the sample original features to obtain domain transformation features;

[0151] Input the sample original features and the domain transformation features into a deep image encoder respectively to perform deep feature encoding to obtain sample deep features and domain transformation deep features;

[0152] Use the domain transformation direction consistency loss to perform inter-domain direction alignment on the source domain text encoding, the target domain text encoding, the sample deep features, and the domain transformation deep features to obtain optimized domain transformation features, and combine the sample original features and the optimized domain transformation features to optimize and train a pre-trained object detection model. After the training is completed, use the original standard detection process of the model to obtain the object detection result.

[0153] In several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the modules is only a logical function division, and there can be other division methods in actual implementation.

[0154] The modules described as separate components may or may not be physically separated. The components shown as modules may or may not be physical units, that is, they can be located in one place, or they can be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0155] In addition, in each embodiment of the present invention, each functional module can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware, or in the form of a combination of hardware and software functional modules.

[0156] For those skilled in the art, it is obvious that the present invention is not limited to the details of the above exemplary embodiments, and without departing from the spirit or basic characteristics of the present invention, the present invention can be implemented in other specific forms.

[0157] Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting. The scope of the present invention is defined by the appended claims rather than the above description. Therefore, all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be encompassed by the present invention. Any reference signs in the claims should not be construed as limiting the claims involved.

[0158] The blockchain referred to in the present invention is a new application mode of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, and encryption algorithms. Blockchain, in essence, is a decentralized database, a string of data blocks generated by using cryptographic methods. Each data block contains information about a batch of network transactions, which is used to verify the validity of the information (anti-counterfeiting) and generate the next block. The blockchain can include a blockchain underlying platform, a platform product service layer, an application service layer, etc.

[0159] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is to use a digital computer or a machine controlled by a digital computer to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to obtain the best results of theory, method, technology, and application systems.

[0160] In addition, obviously, the term "including" does not exclude other units or steps, and the singular does not exclude the plural. The multiple units or devices stated in the system claims can also be implemented by one unit or device through software or hardware. Words such as first, second, etc. are used to represent names and do not represent any specific order.

[0161] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not restrictive. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A target detection method based on text-guided visual domain transformation, characterized in that: The method comprises: Obtain source domain text prompt words and target domain text prompt words, and use a text encoder to respectively encode the source domain text prompt words and the target domain text prompt words to obtain source domain text encoding and target domain text encoding; Obtain an image sample, and use a shallow image encoder to extract the original features of the image sample to obtain the original features of the sample; The original features of the samples are transformed using the domain transformation model to obtain domain transformation features; The original features of the sample and the domain transformation features are respectively input into the deep image encoder for deep feature encoding to obtain the deep features of the sample and the deep features of the domain transformation; The domain transformation directional consistency loss is used to perform inter-domain alignment on the source domain text encoding, target domain text encoding, sample deep features and domain transformation deep features to obtain the optimized domain transformation features. The pre-trained target detection model is optimized and trained by combining the original sample features and the optimized domain transformation features. After the training is completed, the model's original standard detection process is used to obtain the target detection results.

2. The object detection method based on text-guided visual domain transformation according to claim 1, characterized in that: The step of using a text encoder to respectively encode the source domain text prompt word and the target domain text prompt word comprises: Using source domain text prompt words and target domain text prompt words to form multiple domain text pairs; Encoding multiple groups of domain text pairs using a shallow text encoder to obtain multiple groups of embedded text pairs consisting of source domain text embeddings and target domain text embeddings, wherein parameters of the shallow text encoder are initialized by parameters of a pre-trained CLIP model; The difference between the source domain text embedding and the target domain text embedding in the embedded text pair is calculated, and the domain transformation direction of the language modality is obtained through the difference.

3. The object detection method based on text-guided visual domain transformation as claimed in claim 2, characterized in that: The domain transformation direction of the language modality is used to guide the generation of the domain transformation direction of the visual modality.

4. The object detection method based on text-guided visual domain transformation according to claim 1, characterized in that: The method of transforming the original features of the sample using the domain transformation model to obtain domain transformation features includes: Calculate the mean and standard deviation of the original features of the samples in the feature length dimension and feature width dimension based on the channel number dimension; The domain transformation parameters in the domain transformation model are used to perform parameter calculation on the mean and the standard deviation to obtain an updated mean and an updated standard deviation; The updated mean and updated standard deviation are used to update the mean and standard deviation of the original features of the sample to obtain the domain transformation features.

5. The object detection method based on text-guided visual domain transformation according to claim 1, characterized in that: The step of inputting the sample original features and the domain transformation features into the deep image encoder for deep feature encoding to obtain the sample deep features and the domain transformation deep features comprises: Input the original features of the samples into the domain transformation model for domain transformation, and obtain multiple groups of domain transformation features; The original features of the sample and each group of domain transformation features are respectively input into the deep image encoder to obtain the sample deep features and domain transformation deep features.

6. The object detection method based on text-guided visual domain transformation according to claim 5, characterized in that: The difference between the sample deep features and the domain transformed deep features is calculated to obtain the domain transformation direction of the visual modality.

7. The object detection method based on text-guided visual domain transformation as claimed in claim 2, characterized in that: The method of using domain transformation direction consistency loss to perform inter-domain direction alignment on source domain text encoding, target domain text encoding, sample deep features, and domain transformation deep features includes: The domain transformation direction of the visual modality is aligned with the domain transformation direction of the language modality by calculating the consistency loss between the domain transformation direction of the visual modality and the domain transformation direction of the language modality, wherein the consistency loss L between the domain transformation direction of the visual modality and the domain transformation direction of the language modality is calculated. dc Use the following formula: in, is the domain transformation direction of the visual modality of the i-th domain-text pair, is the domain transformation direction of the language mode of the i-th domain-text pair, i is the number of domain-text pairs, and ||·|| is the modulo calculation.

8. An object detection device based on text-guided visual domain transformation, characterized in that: The device is used to implement the target detection method based on text-guided visual domain transformation according to any one of claims 1 to 7, and the device includes: The feature extraction module is used to obtain source domain text prompt words and target domain text prompt words, and use a text encoder to respectively encode the source domain text prompt words and the target domain text prompt words to obtain source domain text encoding and target domain text encoding; obtain image samples, and use a shallow image encoder to extract original features of the image samples to obtain original features of the samples; The feature transformation module is used to transform the original features of the sample using the domain transformation model to obtain the domain transformation features; the original features of the sample and the domain transformation features are respectively input into the deep image encoder for deep feature encoding to obtain the deep features of the sample and the deep features of the domain transformation; The target detection module is used to use the domain transformation direction consistency loss to align the source domain text encoding, target domain text encoding, sample deep features and domain transformation deep features between domains to obtain optimized domain transformation features, and to optimize the training of the pre-trained target detection model by combining the original sample features and the optimized domain transformation features. After the training is completed, the model's original standard detection process is used to obtain the target detection results.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the object detection method based on text-guided visual domain transformation as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the object detection method based on text-guided visual domain transformation according to any one of claims 1 to 7 is implemented.

Images