A target detection method and device based on text-guided visual domain transformation

Abstract

The present invention relates to image processing technology and discloses an object detection method based on text-guided visual domain transformation, including: performing text encoding on the source domain text prompt and the target domain text prompt to obtain the source domain text encoding and the target domain text encoding; acquiring an image sample and extracting the original features of the sample; using a domain transformation model to perform feature transformation on the original features of the sample to obtain domain transformation features; respectively performing deep feature encoding on the original features of the sample and the domain transformation features to obtain the deep features of the sample and the deep features of the domain transformation; using the domain transformation direction consistency loss to perform inter-domain direction alignment to obtain optimized domain transformation features, and combining the original features of the sample to optimize and train a pre-trained object detection model. The present invention also proposes an object detection device, an electronic device and a storage medium based on text-guided visual domain transformation. The present invention can improve the accuracy of object detection and the detection accuracy of object detection in a new domain environment.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular, relates to an object detection method and device based on text-guided visual domain transformation. Background Art

[0002] In recent years, driven by deep learning, significant progress has been made in object detection. However, the success of these methods stems from the assumption of independent and identically distributed data, where the training and test data belong to the same distribution. When applying these methods to data with other distributions that exhibit domain shift relative to the training data, their performance may degrade significantly. A straightforward way to address this issue is to incorporate data from other distributions into the model's training process. Although effective, this approach is inefficient and impractical because collecting and annotating data from different distributions is very time-consuming, and considering all possible distributions encountered in the real world is infeasible.

[0003] To solve this problem, Domain Adaptive Object Detection (DAOD) proposes using labeled source domains and unlabeled target domains to align the two domains to adapt the model. However, the requirement of predefined target domains limits the applicability of DAOD in real-world scenarios. To address this limitation, Single-source Domain Generalized Object Detection (SDGOD) is proposed, which uses only a single source domain to train the model so that it can generalize to all unknown domains. However, the core challenge of SDGOD lies in how to improve the model's adaptability and robustness to target domain changes (such as environment, lighting, background, perspective, etc.). In SDGOD, the main technical problem faced is the domain shift between domains. The object detection model is usually trained on a specific source domain (e.g., outdoor scenes during the day), and applied to a different target domain during testing (e.g., outdoor scenes at night, foggy scenes), but this distribution difference between the source domain and the target domain results in poor performance of the object detection model on the target domain. Summary of the Invention

[0004] The present invention provides an object detection method and device based on text-guided visual domain transformation, which can improve the accuracy of object detection and the detection accuracy of object detection in a new domain environment.

[0005] To achieve the above object, an object detection method based on text-guided visual domain transformation provided by the present invention includes:

[0006] Obtain the source domain text prompt and the target domain text prompt, and use the text encoder to perform text encoding on the source domain text prompt and the target domain text prompt respectively to obtain the source domain text encoding and the target domain text encoding;

[0007] Obtain an image sample, and use a shallow image encoder to extract the original features of the image sample to obtain the sample original features;

[0008] Use the domain transformation model to perform feature transformation on the sample original features to obtain domain transformation features;

[0009] Input the sample original features and the domain transformation features into the deep image encoder respectively to perform deep feature encoding to obtain the sample deep features and the domain transformation deep features;

[0010] Use the domain transformation direction consistency loss to perform inter-domain direction alignment on the source domain text encoding, the target domain text encoding, the sample deep features, and the domain transformation deep features to obtain optimized domain transformation features, and combine the sample original features and the optimized domain transformation features to optimize and train the pre-trained object detection model. After the training is completed, use the original standard detection process of the model to obtain the object detection result.

[0011] Optionally, the using the text encoder to perform text encoding on the source domain text prompt and the target domain text prompt respectively includes:

[0012] Use the source domain text prompt and the target domain text prompt to form multiple groups of domain text pairs;

[0013] Use the shallow text encoder to encode the multiple groups of domain text pairs to obtain multiple groups of embedded text pairs composed of source domain text embeddings and target domain text embeddings. Among them, the parameters of the shallow text encoder are initialized by the pre-trained CLIP model;

[0014] Calculate the difference between the source domain text embedding and the target domain text embedding in the embedded text pair, and obtain the domain transformation direction of the language modality through the difference.

[0015] Optionally, use the domain transformation direction of the language modality to guide the generation of the domain transformation direction of the visual modality.

[0016] Optionally, the using the domain transformation model to perform feature transformation on the sample original features to obtain the domain transformation features includes:

[0017] Calculate the mean and standard deviation of the sample original features based on the channel number dimension in the feature length dimension and the feature width dimension;

[0018] Use the domain transformation parameters in the domain transformation model to perform parameter calculation on the mean and standard deviation to obtain the updated mean and the updated standard deviation;

[0019] Updating the mean and standard deviation in the original features of the sample using the updated mean and updated standard deviation to obtain domain transformation features.

[0020] Optionally, inputting the original features of the sample and the domain transformation features into a deep image encoder respectively for deep feature encoding to obtain the deep features of the sample and the deep features of the domain transformation, including:

[0021] Inputting the original features of the sample into a domain transformation model for domain transformation to obtain multiple groups of domain transformation features;

[0022] Inputting the original features of the sample and each group of domain transformation features into a deep image encoder respectively to obtain the deep features of the sample and the deep features of the domain transformation.

[0023] Optionally, calculating the difference between the deep features of the sample and the deep features of the domain transformation to obtain the domain transformation direction of the visual modality.

[0024] Optionally, performing inter-domain direction alignment on the source domain text encoding, the target domain text encoding, the deep features of the sample, and the deep features of the domain transformation using the domain transformation direction consistency loss, including:

[0025] Aligning the domain transformation direction of the visual modality with the domain transformation direction of the language modality using the consistency loss calculated between the domain transformation direction of the visual modality and the domain transformation direction of the language modality, where the consistency loss L dc Adopts the following formula:

[0026]

[0027] Where Is the domain transformation direction of the visual modality of the i-th group of domain text pairs, Is the domain transformation direction of the language modality of the i-th group of domain text pairs, i is the number of groups of domain text pairs, and ||·|| is the modulo calculation.

[0028] To solve the above problems, the present invention also provides an object detection device based on text-guided visual domain transformation, and the device includes:

[0029] A feature extraction module, configured to obtain a source domain text prompt and a target domain text prompt, perform text encoding on the source domain text prompt and the target domain text prompt respectively using a text encoder to obtain a source domain text encoding and a target domain text encoding; obtain an image sample, and extract the original features of the image sample using a shallow image encoder to obtain the original features of the sample;

[0030] A feature transformation module, which is used to perform feature transformation on the original features of the samples by using a domain transformation model to obtain domain-transformed features; input the original features of the samples and the domain-transformed features into a deep image encoder respectively to perform deep feature encoding to obtain sample deep features and domain-transformed deep features;

[0031] An object detection module, which is used to perform inter-domain direction alignment on the source domain text encoding, target domain text encoding, sample deep features, and domain-transformed deep features by using the domain transformation direction consistency loss to obtain optimized domain-transformed features, and combine the original features of the samples and the optimized domain-transformed features to optimize and train a pre-trained object detection model, and after the training is completed, use the original standard detection process of the model to obtain the object detection result.

[0032] To solve the above problems, the present invention also provides an electronic device, and the electronic device includes:

[0033] At least one processor; and,

[0034] A memory communicatively connected to the at least one processor; wherein,

[0035] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the above-mentioned object detection method based on text-guided visual domain transformation.

[0036] To solve the above problems, the present invention also provides a computer-readable storage medium, and at least one computer program is stored in the computer-readable storage medium, and the at least one computer program is executed by a processor in an electronic device to implement the above-mentioned object detection method based on text-guided visual domain transformation.

[0037] The present invention can establish a connection between the source domain and the target domain at the semantic level by obtaining the text prompts of the source domain and the target domain and encoding them respectively. In addition, the original features of the image samples are extracted by using a shallow image encoder, which can extract the basic visual features of the image and provide the original information for subsequent deep encoding. Furthermore, the feature transformation of the original features of the samples by using a domain transformation model can make the feature distributions of the source domain and the target domain consistent. Moreover, by using the domain transformation direction consistency loss to perform inter-domain direction alignment on the source domain text encoding, target domain text encoding, sample deep features, and domain-transformed deep features, the inter-domain direction alignment of the feature domains can enable the object detection model to adapt more quickly and improve the detection performance when facing new domain data, and can enable the model to simultaneously utilize semantic features and visual features, improving the accuracy of object detection and the detection accuracy of object detection in a new domain environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A flowchart of a target detection method based on text-guided visual domain transformation provided by an embodiment of the present invention;

[0039] Figure 2 A structural flowchart of an example of a target detection method based on text-guided visual domain transformation provided by an embodiment of the present invention;

[0040] Figure 3 A functional module diagram of a target detection device based on text-guided visual domain transformation provided by an embodiment of the present invention;

[0041] Figure 4 A structural schematic diagram of an electronic device for implementing the target detection method based on text-guided visual domain transformation provided by an embodiment of the present invention.

[0042] The realization, functional features and advantages of the object of the present invention will be further described with reference to the embodiments and the accompanying drawings. Detailed implementation manners

[0043] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0044] An embodiment of the present application provides a target detection method based on text-guided visual domain transformation. The execution subject of the target detection method based on text-guided visual domain transformation includes, but is not limited to, at least one of an electronic device such as a server, a terminal, etc. that can be configured to execute the method provided by the embodiment of the present application. In other words, the target detection method based on text-guided visual domain transformation can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes, but is not limited to: a single server, a server cluster, a cloud server, or a cloud server cluster, etc. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, Content Delivery Network (CDN), and big data and artificial intelligence platforms.

[0045] Referring to Figure 1 As shown, it is a flowchart of a target detection method based on text-guided visual domain transformation provided by an embodiment of the present invention. In this embodiment, the target detection method based on text-guided visual domain transformation includes:

[0046] S1. Obtain the source-domain text prompt and the target-domain text prompt, and use a text encoder to perform text encoding on the source-domain text prompt and the target-domain text prompt respectively to obtain the source-domain text encoding and the target-domain text encoding.

[0047] In the embodiments of the present invention, the source-domain text prompt is the text used to describe the source-domain image. For example, if the source-domain image is an image under sunny weather, its source-domain text prompt includes but is not limited to "a fog-free image, a snow-free image, a rain-free image", etc.

[0048] In the embodiments of the present invention, the target-domain text prompt is the text used to describe the target-domain image. For example, if the target-domain image is an image collected under different weather conditions, its target-domain text prompt includes but is not limited to "a foggy image, a snowy image, a rainy image", etc.

[0049] As an embodiment of the present invention, the step of using a text encoder to perform text encoding on the source-domain text prompt and the target-domain text prompt respectively includes:

[0050] Combine the source-domain text prompt and the target-domain text prompt to form multiple groups of domain text pairs;

[0051] Use a shallow text encoder to encode the multiple groups of domain text pairs to obtain multiple groups of embedded text pairs composed of source-domain text embeddings and target-domain text embeddings, where the parameters of the shallow text encoder are initialized by the pre-trained CLIP model;

[0052] Calculate the difference between the source-domain text embedding and the target-domain text embedding in the embedded text pair, and obtain the domain transformation direction of the language modality through the difference value.

[0053] In the embodiments of the present invention, the pre-trained CLIP (Contrastive Language-Image Pre-Training) model is a multi-modal contrastive learning model proposed by OpenAI, which is jointly composed of a text encoder and an image encoder, aiming to establish the association between vision and text semantics through natural language supervision signals. The parameters of the text encoder are initialized by the text encoder of the CLIP model and frozen.

[0054] Furthermore, use the domain transformation direction of the language modality to guide the generation of the domain transformation direction of the visual modality.

[0055] S2. Obtain an image sample, and use a shallow image encoder to extract the original features of the image sample to obtain the sample original features.

[0056] In the embodiments of the present invention, the image sample refers to an image collected with obvious features.

[0057] Exemplarily, the following implementation steps can be adopted to extract the original features of an image sample by using a shallow image encoder to obtain the original sample features:

[0058] For a given image sample x, it passes through the shallow image encoder I of the image encoder e1 Extract the shallow features of the image: f = I e1 (x), where the image encoder I e Consists of a shallow structure I e1 And a deep structure I e2 The middle position of the image encoder I e Also includes a learnable domain transformation model T, and the domain transformation model T contains multiple groups of learnable domain transformation parameters.

[0059] In the embodiment of the present invention, the number of groups of domain transformation parameters in the domain transformation model T is the same as the number of groups of domain text pairs, and the number of groups of domain transformation parameters in the domain transformation model T changes correspondingly with the change in the number of groups of domain text pairs. For example, when the number of groups of domain text pairs increases by one group, the number of groups of domain transformation parameters in the domain transformation model T also increases by one group accordingly.

[0060] S3. Use the domain transformation model to perform feature transformation on the original sample features to obtain domain transformation features.

[0061] As an embodiment of the present invention, the step of using the domain transformation model to perform feature transformation on the original sample features to obtain domain transformation features includes:

[0062] Calculate the mean and standard deviation of the original sample features based on the channel number dimension in the feature length dimension and the feature width dimension;

[0063] Use the domain transformation parameters in the domain transformation model to perform parameter calculation on the mean and standard deviation to obtain the updated mean and updated standard deviation;

[0064] Use the updated mean and updated standard deviation to update the mean and standard deviation in the original sample features to obtain the domain transformation features.

[0065] Exemplarily, the working implementation process of the domain transformation model can be shown as the following implementation steps, where the domain transformation model T contains K groups of learnable domain transformation parameters {α i , β i}, where α i Is the first parameter in the i-th group of domain transformation parameters, and β i Is the second parameter in the i-th group of domain transformation parameters:

[0066] a. Calculate the mean μ and standard deviation σ of the input original sample features f:

[0067] Where, B is the size of the training batch, C is the number of channels, and H and W are the length and width of the feature. The mean μ and standard deviation σ are calculated for the mean and standard deviation in the H and W dimensions. Taking the mean as an example, for each channel C, the mean is obtained by averaging all spatial positions (i.e., H×W pixels) of this channel, that is

[0068] b. Introduce K groups of learnable domain transformation parameters {α i , β i}, where K is a positive integer. And use them to obtain the updated mean and standard deviation where i ∈ [1, K], and the parameters t in the mean and standard deviation represent the updated state of the mean and standard deviation:

[0069]

[0070] where ⊙ represents the Hadamard product (also known as the element-wise product), which is the element-wise product operation of two matrices or tensors. is the updated standard deviation, is the updated mean. Different from the traditional matrix multiplication, the Hadamard product performs multiplication element by element, rather than involving the weighted sum of rows and columns like ordinary matrix multiplication. According to the broadcasting mechanism of Python, when calculating the above formula, will be automatically replicated B times to become and can then be multiplied element by element with .

[0071] c. Based on the input feature f, obtain the domain transformation feature by changing the mean and standard deviation of the feature (using the updated mean and standard deviation instead of the original mean and standard deviation)

[0072]

[0073] S4. Input the original sample feature and the domain transformation feature into the deep image encoder respectively to perform deep feature encoding to obtain the sample deep feature and the domain transformation deep feature.

[0074] As an embodiment of the present invention, the step of inputting the original sample feature and the domain transformation feature into the deep image encoder respectively to perform deep feature encoding to obtain the sample deep feature and the domain transformation deep feature includes:

[0075] Input the original sample feature into the domain transformation model for domain transformation to obtain multiple groups of domain transformation features;

[0076] The original features of the sample and the domain transformation features of each group are respectively input into the deep image encoder to obtain the deep features of the sample and the deep features of the domain transformation.

[0077] Further, calculate the difference between the deep features of the sample and the deep features of the domain transformation to obtain the domain transformation direction of the visual modality.

[0078] Exemplarily, the following implementation steps can be adopted to calculate the difference between the output features of the deep features of the sample and the deep features of the domain transformation to obtain the domain transformation direction of the visual modality:

[0079] Send the original features of the sample into the domain transformation model T for domain transformation to obtain the domain transformation features: where i ∈ [1, K], because the module T contains K groups of learnable domain transformation parameters, and one group of parameters can generate one kind of transformed feature. Then the original features F of the sample and the domain transformation features are sent into the deep image encoder I of the subsequent image encoder e2 , to obtain the output features of the final image encoder: F s = I e2 (f) and where, F s is the feature after the original features of the sample are encoded by the deep image encoder, is the feature after the domain transformation features are encoded by the deep image encoder. Take the difference between the two as the domain transformation direction of the visual modality:

[0080] S5. Use the domain transformation direction consistency loss to perform inter-domain direction alignment on the source domain text encoding, target domain text encoding, deep features of the sample, and deep features of the domain transformation to obtain optimized domain transformation features, and combine the original features of the sample and the optimized domain transformation features to optimize and train the pre-trained object detection model. After the training is completed, use the original standard detection process of the model to obtain the object detection result.

[0081] In the embodiment of the present invention, the pre-trained object detection model refers to the Faster R-CNN model. The Faster R-CNN model is an object detection model that realizes the deep fusion of candidate box generation and object detection through RPN, and the weights of its image encoder are initialized by the network pre-trained on the ImageNet dataset.

[0082] As an embodiment of the present invention, the using the domain transformation direction consistency loss to perform inter-domain direction alignment on the source domain text encoding, target domain text encoding, deep features of the sample, and deep features of the domain transformation includes:

[0083] Align the domain transformation direction of the visual modality with the domain transformation direction of the language modality by using the consistency loss between the domain transformation direction of the computational visual modality and the domain transformation direction of the language modality, where the consistency loss L between the domain transformation direction of the computational visual modality and the domain transformation direction of the language modality dc Adopt the following formula:

[0084]

[0085] Where Is the domain transformation direction of the visual modality of the i-th group of domain text pairs, Is the domain transformation direction of the language modality of the i-th group of domain text pairs, i is the number of groups of domain text pairs, and ||·|| is the modulo calculation.

[0086] In the present invention, by obtaining the text prompts of the source domain and the target domain and encoding them respectively, the connection between the source domain and the target domain can be established at the semantic level. In addition, by using the shallow image encoder to extract the original features of the image samples, the basic visual features of the image can be extracted and the original information can be provided for the subsequent deep encoding. In addition, by using the domain transformation model to perform feature transformation on the sample original features, the feature distributions of the source domain and the target domain can be made consistent. Furthermore, by using the domain transformation direction consistency loss to perform inter-domain direction alignment on the source domain text encoding, the target domain text encoding, the sample deep features, and the domain transformation deep features, the inter-domain direction alignment of the features can enable the target detection model to adapt more quickly and improve the detection performance when the model faces new domain data, and can enable the model to utilize both semantic features and visual features at the same time, improving the accuracy of target detection and the detection accuracy of target detection in the new domain environment.

[0087] Refer to Figure 2 As shown, it is a structural flowchart of an example of the target detection method based on text-guided visual domain transformation provided by an embodiment of the present invention.

[0088] As Figure 3 Shown, it is a functional module diagram of a target detection device based on text-guided visual domain transformation provided by an embodiment of the present invention.

[0089] The target detection device 100 based on text-guided visual domain transformation described in the present invention can be installed in an electronic device. According to the functions implemented, the target detection device 100 based on text-guided visual domain transformation can include a feature extraction module 101, a feature transformation module 102, and a target detection module 103.

[0090] The module described in the present invention can also be referred to as a unit, which refers to a series of computer program segments that can be executed by the processor of an electronic device and can complete fixed functions, and are stored in the memory of the electronic device.

[0091] In this embodiment, the functions of each module / unit are as follows:

[0092] The feature extraction module 101 is configured to obtain a source domain text prompt and a target domain text prompt, respectively perform text encoding on the source domain text prompt and the target domain text prompt by using a text encoder to obtain a source domain text encoding and a target domain text encoding; obtain an image sample, and extract the original features of the image sample by using a shallow image encoder to obtain sample original features.

[0093] In the embodiment of the present invention, the source domain text prompt is the text used to describe the source domain image. For example, if the source domain image is an image under sunny weather, its source domain text prompt includes but is not limited to "a fog-free image, a snow-free image, a rain-free image", etc.

[0094] In the embodiment of the present invention, the target domain text prompt is the text used to describe the target domain image. For example, if the target domain image is an image collected under different weather conditions, its target domain text prompt includes but is not limited to "a foggy image, a snowy image, a rainy image", etc.

[0095] As an embodiment of the present invention, the step of respectively performing text encoding on the source domain text prompt and the target domain text prompt by using a text encoder includes:

[0096] Combining the source domain text prompt and the target domain text prompt to form multiple groups of domain text pairs;

[0097] Encoding the multiple groups of domain text pairs by using a shallow text encoder to obtain multiple groups of embedded text pairs composed of source domain text embeddings and target domain text embeddings, wherein the parameters of the shallow text encoder are initialized by the text encoder of the pre-trained CLIP model;

[0098] Calculating the difference between the source domain text embedding and the target domain text embedding in the embedded text pair, and obtaining the domain transformation direction of the language modality through the difference.

[0099] In the embodiment of the present invention, the pre-trained CLIP (Contrastive Language-Image Pre-Training) model is a multi-modal contrastive learning model proposed by OpenAI, which is jointly composed of a text encoder and an image encoder, aiming to establish the association between vision and text semantics through natural language supervision signals. The parameters of the text encoder are initialized and frozen by the text encoder of the CLIP model.

[0100] Furthermore, the domain transformation direction of the visual modality is generated by guiding with the domain transformation direction of the language modality.

[0101] In the embodiments of the present invention, an image sample refers to an image collected with obvious features.

[0102] Exemplarily, the steps for extracting the original features of an image sample by using a shallow image encoder to obtain sample original features may be as follows:

[0103] For a given image sample x, through the shallow image encoder I of the image encoder e1 Extract the shallow features of the image: f = I e1 (x), where the image encoder I e Consists of a shallow structure I e1 And a deep structure I e2 The middle position of the image encoder I e Also includes a learnable domain transformation model T, and the domain transformation model T contains multiple groups of learnable domain transformation parameters.

[0104] In the embodiments of the present invention, the number of groups of domain transformation parameters in the domain transformation model T is the same as the number of groups of domain text pairs, and the number of groups of domain transformation parameters in the domain transformation model T changes correspondingly with the change in the number of groups of domain text pairs. For example, when the number of groups of domain text pairs increases by one group, the number of groups of domain transformation parameters in the domain transformation model T also increases by one group accordingly.

[0105] The feature transformation module 102 is used to perform feature transformation on the sample original features by using the domain transformation model to obtain domain transformation features; input the sample original features and the domain transformation features into the deep image encoder respectively for deep feature encoding to obtain sample deep features and domain transformation deep features.

[0106] As an embodiment of the present invention, the step of performing feature transformation on the sample original features by using the domain transformation model to obtain domain transformation features includes:

[0107] Calculate the mean and standard deviation of the sample original features based on the channel number dimension in the feature length dimension and the feature width dimension;

[0108] Use the domain transformation parameters in the domain transformation model to perform parameter calculation on the mean and standard deviation to obtain an updated mean and an updated standard deviation;

[0109] Use the updated mean and the updated standard deviation to update the mean and the standard deviation in the sample original features to obtain domain transformation features.

[0110] Exemplarily, the working implementation process of the domain transformation model may be as shown in the following implementation steps, where the domain transformation model T contains K groups of learnable domain transformation parameters {α i , β i}, where α i Is the first parameter in the i-th group of domain transformation parameters, and β iis the second parameter in the i-th group of domain transformation parameters:

[0111] a. Calculate the mean μ and standard deviation σ of the original feature f of the input sample:

[0112] in, B is the size of the training batch, C is the number of channels, H and W are the length and width of the feature. The mean μ and standard deviation σ are calculated on the H and W dimensions. Taking the mean as an example, for each channel C, the mean is calculated by averaging all spatial positions of the channel (i.e. H×W pixels), that is,

[0113] b. Introducing K sets of learnable domain transformation parameters {α i ,β i}, K is a positive integer. And use them to get the updated mean based on the original mean μ and standard deviation σ and standard deviation Among them, i∈[1,K], mean and standard deviation The parameter t in represents the updated state of the mean and standard deviation:

[0114]

[0115] Where ⊙ indicates that the Hadamard product (also known as element-wise product) is the element-wise product operation of two matrices or tensors. To update the standard deviation, To update the mean. Unlike traditional matrix multiplication, Hadamard product is element-wise multiplication, rather than involving the weighted sum of rows and columns like ordinary matrix multiplication. According to Python's broadcast mechanism, when calculating the above formula, will be automatically copied B times to become You can Element-wise multiplication.

[0116] c. Based on the input feature f, the domain transformation feature is obtained by changing the mean and standard deviation of the feature (using the updated mean and standard deviation instead of the original mean and standard deviation)

[0117]

[0118] As an embodiment of the present invention, the step of inputting the sample original features and the domain transformed features into a deep image encoder for deep feature encoding to obtain the sample deep features and the domain transformed deep features includes:

[0119] Input the original features of the sample into the domain transformation model for domain transformation to obtain multiple sets of domain-transformed features;

[0120] Input the original features of the sample and each set of domain-transformed features into the deep image encoder respectively to obtain the deep features of the sample and the deep features of the domain transformation.

[0121] Furthermore, calculate the difference between the deep features of the sample and the deep features of the domain transformation to obtain the domain transformation direction of the visual modality.

[0122] Exemplarily, calculating the difference between the output features of the deep features of the sample and the deep features of the domain transformation to obtain the domain transformation direction of the visual modality can adopt the following implementation steps:

[0123] Send the original features of the sample into the domain transformation model T for domain transformation to obtain the domain-transformed features: where \(i\in[1, K]\), because the module T contains K sets of learnable domain transformation parameters, and one set of parameters can generate one kind of transformed feature. Then, send the original features \(f\) of the sample and the domain-transformed features into the deep image encoder I of the subsequent image encoder e2 , to obtain the output features of the final image encoder: F s = I e2 (f) and where, F s is the feature after the original features of the sample are encoded by the deep image encoder, is the feature after the domain-transformed features are encoded by the deep image encoder. Take the difference between the two as the domain transformation direction of the visual modality:

[0124] The target detection module 103 is used to perform inter-domain direction alignment on the source domain text encoding, target domain text encoding, deep features of the sample, and deep features of the domain transformation by using the domain transformation direction consistency loss to obtain optimized domain transformation features, and optimize and train the pre-trained target detection model by combining the original features of the sample and the optimized domain transformation features. After the training is completed, the target detection result is obtained by using the original standard detection process of the model.

[0125] In the embodiment of the present invention, the pre-trained target detection model refers to the Faster R-CNN model. The Faster R-CNN model is a target detection model that realizes the deep fusion of candidate box generation and target detection through RPN, and the weights of its image encoder are initialized by the network pre-trained on the ImageNet dataset.

[0126] As an embodiment of the present invention, the performing inter-domain direction alignment on the source domain text encoding, target domain text encoding, deep features of the sample, and deep features of the domain transformation by using the domain transformation direction consistency loss includes:

[0127] Align the domain transformation direction of the computational vision modality with the domain transformation direction of the language modality by using the consistency loss between the domain transformation direction of the computational vision modality and the domain transformation direction of the language modality, where the consistency loss L between the domain transformation direction of the computational vision modality and the domain transformation direction of the language modality dc Adopt the following formula:

[0128]

[0129] where is the domain transformation direction of the vision modality of the i-th group of domain text pairs is the domain transformation direction of the language modality of the i-th group of domain text pairs, i is the number of groups of domain text pairs, and ||·|| is the modulo calculation.

[0130] As Figure 4 shown, it is a schematic structural diagram of an electronic device for implementing an object detection method based on text-guided visual domain transformation provided by an embodiment of the present invention.

[0131] The electronic device may include a processor 10, a memory 11, a communication bus 12, and a communication interface 13, and may also include a computer program stored in the memory 11 and executable on the processor 10, such as an object detection method program based on text-guided visual domain transformation.

[0132] Among them, the processor 10 may be composed of integrated circuits in some embodiments. For example, it may be composed of a single packaged integrated circuit, or may be composed of multiple integrated circuits with the same or different functions, including a combination of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 10 is the control core (ControlUnit) of the electronic device, connects various components of the entire electronic device through various interfaces and lines, and by running or executing programs or modules stored in the memory 11 (such as executing an object detection method program based on text-guided visual domain transformation, etc.), and calling data stored in the memory 11, to perform various functions of the electronic device and process data.

[0133] The memory 11 at least includes one type of readable storage medium, and the readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (such as SD or DX memory, etc.), magnetic memory, magnetic disk, optical disk, etc. The memory 11 can be an internal storage unit of the electronic device in some embodiments, such as the mobile hard disk of the electronic device. The memory 11 can also be an external storage device of the electronic device in some other embodiments, such as a plug-in mobile hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. equipped on the electronic device. Further, the memory 11 can also include both the internal storage unit and the external storage device of the electronic device. The memory 11 can be used not only to store application software installed in the electronic device and various types of data, such as the code of a target detection method program based on text-guided visual domain transformation, etc., but also to temporarily store the data that has been output or will be output.

[0134] The communication bus 12 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. The bus is configured to implement the connection and communication between the memory 11 and at least one processor 10, etc.

[0135] The communication interface 13 is used for the communication between the above-mentioned electronic device and other devices, including a network interface and a user interface. Optionally, the network interface can include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), and is usually used to establish a communication connection between this electronic device and other electronic devices. The user interface can be a display, an input unit (such as a keyboard), and optionally, the user interface can also be a standard wired interface, a wireless interface. Optionally, in some embodiments, the display can be an LED display, a liquid crystal display, a touch liquid crystal display, and an OLED (Organic Light-Emitting Diode) toucher, etc. Among them, the display can also be appropriately referred to as a display screen or a display unit, and is used to display the information processed in the electronic device and to display a visual user interface.

[0136] Figure 4 Only the electronic device with components is shown, and those skilled in the art can understand that Figure 4The structures shown do not constitute a limitation on the electronic device, and it may include fewer or more components than those shown, or combine certain components, or have different component arrangements.

[0137] For example, although not shown, the electronic device may further include a power source (such as a battery) for powering each component. Preferably, the power source can be logically connected to the at least one processor 10 through a power management device, so as to implement functions such as charging management, discharging management, and power consumption management through the power management device. The power source may also include any components such as one or more DC or AC power sources, a recharge device, a power failure detection circuit, a power converter or inverter, and a power status indicator. The electronic device may also include various sensors, a Bluetooth module, a Wi-Fi module, etc., which will not be elaborated here.

[0138] It should be understood that the embodiments are only for illustrative purposes and are not limited by this structure in the scope of the patent application.

[0139] A target detection method program based on text-guided visual domain transformation stored in the memory 11 of the electronic device is a combination of multiple instructions. When running in the processor 10, it can implement:

[0140] Obtain a source domain text prompt and a target domain text prompt, and use a text encoder to perform text encoding on the source domain text prompt and the target domain text prompt respectively to obtain a source domain text encoding and a target domain text encoding;

[0141] Obtain an image sample, and use a shallow image encoder to extract the original features of the image sample to obtain sample original features;

[0142] Perform feature transformation on the sample original features using a domain transformation model to obtain domain transformation features;

[0143] Input the sample original features and the domain transformation features into a deep image encoder respectively to perform deep feature encoding to obtain sample deep features and domain transformation deep features;

[0144] Use the domain transformation direction consistency loss to perform inter-domain direction alignment on the source domain text encoding, the target domain text encoding, the sample deep features, and the domain transformation deep features to obtain optimized domain transformation features, and combine the sample original features and the optimized domain transformation features to optimize the training of a pre-trained target detection model. After the training is completed, use the original standard detection process of the model to obtain the target detection result.

[0145] Specifically, the specific implementation method of the processor 10 for the above instructions can refer to the description of the relevant steps in the corresponding embodiments of the attached drawings, which will not be elaborated here.

[0146] Furthermore, if the modules / units integrated in the electronic device 1 are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, the computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disc, a computer memory, a read-only memory (ROM, Read-Only Memory).

[0147] The present invention also provides a computer-readable storage medium storing a computer program, which when executed by a processor of an electronic device, can implement:

[0148] Obtain a source domain text prompt and a target domain text prompt, and use a text encoder to perform text encoding on the source domain text prompt and the target domain text prompt respectively to obtain a source domain text encoding and a target domain text encoding;

[0149] Obtain an image sample, and use a shallow image encoder to extract the original features of the image sample to obtain sample original features;

[0150] Perform feature transformation on the sample original features using a domain transformation model to obtain domain transformation features;

[0151] Input the sample original features and the domain transformation features into a deep image encoder respectively for deep feature encoding to obtain sample deep features and domain transformation deep features;

[0152] Use the domain transformation direction consistency loss to perform inter-domain direction alignment on the source domain text encoding, the target domain text encoding, the sample deep features, and the domain transformation deep features to obtain optimized domain transformation features, and combine the sample original features and the optimized domain transformation features to optimize and train a pre-trained object detection model. After the training is completed, use the original standard detection process of the model to obtain the object detection result.

[0153] In several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the modules is only a logical function division, and there can be other division methods in actual implementation.

[0154] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0155] In addition, in each embodiment of the present invention, each functional module may be integrated into a processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware, or in the form of a combination of hardware and software functional modules.

[0156] For those skilled in the art, it is obvious that the present invention is not limited to the details of the above-mentioned exemplary embodiments, and without departing from the spirit or basic characteristics of the present invention, the present invention can be implemented in other specific forms.

[0157] Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting. The scope of the present invention is defined by the appended claims rather than the above description. Therefore, all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be included in the present invention. Any reference signs in the claims should not be regarded as limiting the claimed invention.

[0158] The blockchain referred to in the present invention is a new application mode of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, and encryption algorithm. Blockchain, essentially a decentralized database, is a string of data blocks generated by using cryptographic methods. Each data block contains information about a batch of network transactions, which is used to verify the validity of the information (anti-counterfeiting) and generate the next block. The blockchain may include a blockchain underlying platform, a platform product service layer, an application service layer, etc.

[0159] The embodiments of the present application may acquire and process relevant data based on artificial intelligence technology. Among them, Artificial Intelligence (AI) is a theory, method, technology, and application system that uses a digital computer or a machine controlled by a digital computer to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use the knowledge to obtain the best results.

[0160] In addition, obviously, the term "including" does not exclude other units or steps, and the singular does not exclude the plural. The multiple units or devices stated in the system claims may also be implemented by one unit or device through software or hardware. The terms such as first and second are used to represent names and do not indicate any specific order.

[0161] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A target detection method based on text-guided visual domain transformation, characterized in that, The method includes: Obtain a source-domain text prompt and a target-domain text prompt, and use a text encoder to perform text encoding on the source-domain text prompt and the target-domain text prompt respectively to obtain a source-domain text encoding and a target-domain text encoding. Among them, the step of using the text encoder to perform text encoding on the source-domain text prompt and the target-domain text prompt respectively includes: forming multiple groups of domain text pairs with the source-domain text prompt and the target-domain text prompt; using a shallow text encoder to encode the multiple groups of domain text pairs to obtain multiple groups of embedded text pairs composed of source-domain text embeddings and target-domain text embeddings. Among them, the parameters of the shallow text encoder are initialized by a pre-trained CLIP model; Calculate the difference between the source-domain text embedding and the target-domain text embedding in the embedded text pair, and obtain the domain transformation direction of the language modality through the difference; Obtain an image sample, and use a shallow image encoder to extract the original features of the image sample to obtain sample original features; Use a domain transformation model to perform feature transformation on the sample original features to obtain domain transformation features; Input the sample original features and the domain transformation features into a deep image encoder respectively to perform deep feature encoding to obtain sample deep features and domain transformation deep features; Align the source domain text encoding, target domain text encoding, sample deep features, and domain transformation deep features in terms of inter-domain direction using the domain transformation direction consistency loss to obtain optimized domain transformation features, and optimize the pre-trained object detection model by combining the sample original features and the optimized domain transformation features. After the training is completed, obtain the object detection results using the original standard detection process of the model. Among them, aligning the source domain text encoding, target domain text encoding, sample deep features, and domain transformation deep features in terms of inter-domain direction using the domain transformation direction consistency loss includes: aligning the domain transformation direction of the visual modality with the domain transformation direction of the language modality using the consistency loss between the domain transformation direction of the visual modality and the domain transformation direction of the language modality. Among them, the consistency loss L dc is calculated using the following formula: where is the domain transformation direction of the visual modality of the i-th group of domain text pairs, is the domain transformation direction of the language modality of the i-th group of domain text pairs, i is the number of groups of domain text pairs, and ||·|| is the modulus calculation.

2. The object detection method based on text-guided visual domain transformation according to claim 1, characterized in that, Use the domain transformation direction of the language modality to guide the generation of the domain transformation direction of the visual modality.

3. The object detection method based on text-guided visual domain transformation according to claim 1, wherein, The step of using a domain transformation model to perform feature transformation on the sample original features to obtain domain transformation features includes: Calculate the mean and standard deviation of the sample original features based on the channel number dimension in the feature length dimension and the feature width dimension; Use the domain transformation parameters in the domain transformation model to perform parameter calculation on the mean and standard deviation to obtain an updated mean and an updated standard deviation; Use the updated mean and updated standard deviation to update the mean and standard deviation in the sample original features to obtain domain transformation features.

4. The object detection method based on text-guided visual domain transformation according to claim 1, characterized in that, The step of inputting the sample original features and the domain transformation features into a deep image encoder respectively to perform deep feature encoding to obtain sample deep features and domain transformation deep features includes: Input the sample original features into the domain transformation model for domain transformation to obtain multiple groups of domain transformation features; Input the sample original features and each group of domain transformation features into the deep image encoder respectively to obtain sample deep features and domain transformation deep features.

5. The object detection method based on text-guided visual domain transformation according to claim 4, wherein Calculate the difference between the sample deep features and the domain transformation deep features to obtain the domain transformation direction of the visual modality.

6. An object detection device based on text-guided visual domain transformation, characterized in that, The device is used to implement the object detection method based on text-guided visual domain transformation according to any one of claims 1 to 5. The device includes: A feature extraction module, configured to obtain a source-domain text prompt and a target-domain text prompt, and use a text encoder to perform text encoding on the source-domain text prompt and the target-domain text prompt respectively to obtain a source-domain text encoding and a target-domain text encoding. Among them, the step of using the text encoder to perform text encoding on the source-domain text prompt and the target-domain text prompt respectively includes: forming multiple groups of domain text pairs by using the source-domain text prompt and the target-domain text prompt; using a shallow text encoder to encode the multiple groups of domain text pairs to obtain multiple groups of embedded text pairs composed of source-domain text embeddings and target-domain text embeddings. Among them, the parameters of the shallow text encoder are initialized by a pre-trained CLIP model; obtaining an image sample, and using a shallow image encoder to extract the original features of the image sample to obtain sample original features; A feature transformation module, configured to use a domain transformation model to perform feature transformation on the sample original features to obtain domain transformation features; input the sample original features and the domain transformation features into a deep image encoder respectively to perform deep feature encoding to obtain sample deep features and domain transformation deep features; The target detection module is used to perform inter-domain direction alignment on the source domain text encoding, target domain text encoding, sample deep features, and domain transformation deep features by using the domain transformation direction consistency loss to obtain optimized domain transformation features, and optimize and train the pre-trained target detection model by combining the sample original features and the optimized domain transformation features. After the training is completed, the target detection result is obtained by using the original standard detection process of the model. Among them, the inter-domain direction alignment of the source domain text encoding, target domain text encoding, sample deep features, and domain transformation deep features by using the domain transformation direction consistency loss includes: aligning the domain transformation direction of the visual modality with the domain transformation direction of the language modality by using the consistency loss between the domain transformation direction of the visual modality and the domain transformation direction of the language modality. Among them, the consistency loss L dc is calculated using the following formula: where is the domain transformation direction of the visual modality of the i-th group of domain text pairs, is the domain transformation direction of the language modality of the i-th group of domain text pairs, i is the number of groups of domain text pairs, and ||·|| is the modulo calculation.

7. An electronic device, characterized in that, The electronic device includes: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the object detection method based on text-guided visual domain transformation according to any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the object detection method based on text-guided visual domain transformation according to any one of claims 1 to 5.

Images