A target generalization identification method and device based on virtual data generation

Abstract

The application discloses a target generalization identification method and device based on virtual data generation. The target generalization identification method of the application relies on generated virtual data, designs a target-style text prompt template, and combines a text-to-image model to generate high-fidelity and diversified virtual data. The target content features are extracted and optimized by using a visual language model and a mapping network, and the influence of style features on target content features is reduced. The cosine similarity between the target content features of a real target image and the target text features of a reference target image is calculated to realize target generalization identification of the real target image, improve the adaptability and identification accuracy of the model trained by the virtual data in a real scene, and provide strong support for value mining of virtual data in actual application.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and in particular to a target generalization recognition method and apparatus based on virtual data generation. Background Technology

[0002] In certain specific scenarios, acquiring real-world data faces significant challenges, such as image acquisition in extreme environments, recording rare events, or in privacy-sensitive fields (e.g., healthcare, security). In these scenarios, real-world data is often scarce, costly, or even unavailable, severely limiting the training and application of deep learning models. To compensate for this scarcity, virtual data generation techniques are needed to generate large amounts of realistic virtual data using computer technology. However, traditional virtual data generation typically relies on game engines, simulating real-world environments through 3D scene modeling and saving images of these 3D scenes to generate virtual data. This method is not only complex and costly, but also struggles to capture the diversity of real-world scenarios, and the generated data style deviates significantly from the natural distribution of real-world data.

[0003] In the medical field, early detection and accurate diagnosis are crucial for improving cure rates and survival rates for various diseases. However, obtaining surgical scenario data under real-world conditions is difficult, and existing surgical scenario datasets struggle to fully unlock the potential of deep learning models and effectively identify lesion areas. Therefore, it is necessary to generate virtual data of surgical scenarios to supplement real surgical scenario data, thereby increasing the data volume and fully exploring the potential of deep learning models. However, existing virtual data generation technologies typically focus on simulating real-world scenarios through game engines or 3D modeling, but these processes are complex and difficult to dynamically adjust, unable to quickly adapt to diverse real-world scenario needs, and thus cannot effectively supplement real surgical scenario data or significantly improve the model's ability to identify lesion areas.

[0004] Therefore, how to design a target generalization recognition method that can make up for the lack of real data and has high recognition efficiency is a technical problem to be solved. Summary of the Invention

[0005] Therefore, it is necessary to provide a target generalization recognition method and apparatus based on virtual data generation to address the problems of existing technologies.

[0006] In a first aspect, embodiments of this application disclose a target generalization identification method based on virtual data generation, comprising the following steps:

[0007] S1: Generate prompt text based on the relevant features of the target's training data in real-world application scenarios;

[0008] S2: Input the prompt text into the pre-trained text-to-image model to generate virtual data of the target in the actual application scenario;

[0009] S3: Input the virtual data into the visual language big model to generate visual global features, target pseudowords, and style pseudowords of the virtual data; the visual language big model includes CLIP visual encoder, CLIP text encoder, target mapping network, style mapping network, and content mapping network.

[0010] S4: Embed the target pseudowords and style pseudowords into a predefined target-style text template, and embed the target pseudowords into the predefined target text template, and process them respectively through the CLIP text encoder to obtain target-style text features and target text features;

[0011] S5: Construct the visual global features and target-style text features into a first positive sample pair, so that the visual language big model learns the target mapping network and style mapping network based on the first positive sample pair to update the CLIP visual encoder;

[0012] S6: Project the visual global features into the content mapping network to obtain the target content features;

[0013] S7: Construct the target content features and the target text features into a second positive sample pair, and enable the content mapping network to perform comparative learning based on the second positive sample pair to update the content mapping network;

[0014] S8: Calculate the target text features of the reference target image based on the updated visual language big model, and calculate the target content features of the real target image based on the updated visual language big model.

[0015] S9: Calculate the cosine similarity between the target content features of the real target image and the target text features of the reference target image, and obtain the target generalization recognition result based on the cosine similarity ranking result.

[0016] Preferably, step S2 includes:

[0017] S21: Design an image partitioning diagram to specify the generation area for each image data, used to store data of the same target under different styles;

[0018] S22: Obtain the target's morphological data;

[0019] S23: Generate multi-style, multi-objective training data based on prompt text, image segmentation diagrams, and morphological data;

[0020] S24: Preprocess the multi-style multi-objective training data and crop it based on the image separator box to obtain virtual data.

[0021] Preferably, step S3 includes:

[0022] S31: The virtual data is processed by the CLIP visual encoder of the large visual language model to obtain global visual features;

[0023] S32: Project the visual global features using a target mapping network to obtain target pseudo-words;

[0024] S33: Use the output of the penultimate layer of the CLIP visual encoder as a visual local feature;

[0025] S34: Calculate the cross-attention between the visual local features and the visual global features, and filter out visual local features that are highly related to the image semantics based on the attention score;

[0026] S35: The selected visual local features are projected through a style mapping network to obtain style pseudowords.

[0027] Preferably, step S4 includes:

[0028] S41: Design two pseudoword-based text templates: a target-style text template and a target text template.

[0029] S42: The target-style text template and the target text template are processed by the CLIP text encoder to obtain the target-style text template features and the target text template features;

[0030] S43: Embed the target pseudowords and style pseudowords into the target-style text template features to obtain the target-style text features;

[0031] S44: Embed the target pseudoword into the target text template features to obtain the target text features.

[0032] Secondly, embodiments of this application disclose a target generalization recognition device based on virtual data generation, comprising:

[0033] The prompt text generation unit is used to generate prompt text based on the relevant features of the target's training data in the actual application scenario;

[0034] The virtual data generation unit is used to input the prompt text into a pre-trained text-to-image model to generate virtual data of the target in the actual application scenario.

[0035] The virtual processing unit is used to input the virtual data into the visual language large model and generate visual global features, target pseudowords, and style pseudowords of the virtual data; the visual language large model includes CLIP visual encoder, CLIP text encoder, target mapping network, style mapping network, and content mapping network.

[0036] The text feature generation unit is used to embed the target pseudoword and style pseudoword into a predefined target-style text template and embed the target pseudoword into a predefined target text template, and process them respectively through the CLIP text encoder to obtain target-style text features and target text features;

[0037] The visual encoder update unit is used to construct the visual global features and target-style text features into a first positive sample pair, so that the visual language big model learns the target mapping network and style mapping network based on the first positive sample pair to update the CLIP visual encoder.

[0038] A content feature generation unit is used to project the visual global features into the content mapping network to obtain target content features;

[0039] The content mapping network update unit is used to construct a second positive sample pair from the target content features and the target text features, so that the content mapping network can perform comparative learning based on the second positive sample pair to update the content mapping network;

[0040] The image feature calculation unit is used to calculate the target text features of the reference target image based on the updated visual language big model, and to calculate the target content features of the real target image based on the updated visual language big model.

[0041] The target generalization recognition unit is used to calculate the cosine similarity between the target content features of the real target image and the target text features of the reference target image, and obtain the target generalization recognition result based on the ranking result of the cosine similarity.

[0042] Compared with existing technologies, this invention has the following advantages: A target generalization recognition method based on virtual data generation designs target-style text prompt templates and combines them with a large-scale text-to-image model to generate high-fidelity and diverse virtual data; utilizing a large-scale visual language model and mapping network, it extracts and optimizes target content features, reducing the influence of style features on target content features; by calculating the cosine similarity between the target content features of a real target image and the target text features of a reference target image, it achieves target generalization recognition of real target images, improving the adaptability and recognition accuracy of the model trained on virtual data in real-world scenarios, and providing strong support for the value mining of virtual data in practical applications. Attached Figure Description

[0043] Exemplary embodiments of the present invention can be more fully understood by referring to the accompanying drawings. The drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain the present invention and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0044] Figure 1 A flowchart illustrating a target generalization identification method based on virtual data generation, provided in this application embodiment;

[0045] Figure 2 A schematic diagram of a target generalization recognition device based on virtual data generation provided in this application embodiment; Detailed Implementation

[0046] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0047] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0048] Furthermore, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0049] Reference Figure 1 This embodiment discloses a target generalization recognition method based on virtual data generation, including the following steps:

[0050] S1: Generate prompt text based on the relevant features of the target's training data in real-world application scenarios;

[0051] Specifically, by summarizing the types of targets that need to be identified in real-world application scenarios and the style types that these targets may exhibit due to the influence of other factors, a dedicated text prompt template is designed to construct prompt text for generating multi-style, multi-target virtual data. The prompt text content includes fixed descriptive terms, feature descriptive terms, and decorative descriptive terms. Fixed descriptive terms are used to control the quality of virtual data generation and describe the general characteristics of real-world application scenarios, making the generated virtual data closer to the real scene. Feature descriptive terms are used to control the representative characteristics of each target, making the generated virtual data more biased towards the specified target. Decorative descriptive terms are used to expand upon the fixed and feature descriptive terms, allowing the generated virtual data to present different styles and increasing diversity. Through different combinations of fixed, feature, and decorative descriptive terms, various prompt texts for generating multi-style, multi-target virtual data are constructed.

[0052] S2: Input the prompt text into the pre-trained text-to-image model to generate virtual data of the target in the actual application scenario;

[0053] Specifically, using the prompt text obtained in step S1, and combining the image segmentation bounding boxes (the image segmentation boxes are pre-designed, and the final display effect is an image similar to a 5×2 table. The image generation model is controlled by the edge detection model in ControlNet and the multi-view generation model in Lora to generate each image within each segmentation box) and target morphology data (the target morphology data comes from existing datasets. The pose detection model in ControlNet extracts the pose information that the target may exhibit in the existing dataset to generate virtual data that is closer to reality and more natural), the image generation model is guided to generate virtual data that is both close to the real scene and easy to compare and observe. The image segmentation bounding boxes are set to 2 rows and 6 columns. The Canny edge detection model is used to identify the segmentation bounding boxes and restrict the model to generate virtual data within the bounding boxes. The image segmentation bounding boxes consist of black borders and white images. The edge detection algorithm distinguishes the position of the bounding boxes and controls the generation of virtual data according to the order of the bounding boxes.

[0054] Furthermore, the OpenPose model is used to identify the morphological data of the targets from the provided reference target data, guiding the model to generate the morphology of the virtual data. The obtained prompt text is input into the Stable Diffusion text-to-image model, and constrained by image segmentation diagrams and target morphological data, to generate virtual data of different targets in different styles. The generated virtual data is manually screened to remove data that is insufficiently generated, of poor quality, or does not meet the actual application scenario. The screened images are then cropped according to the image segmentation diagrams to obtain a virtual dataset that can be used for training.

[0055] S3: Input virtual data into the visual language big model to generate visual global features, target pseudowords, and style pseudowords of the virtual data; the visual language big model includes CLIP visual encoder, CLIP text encoder, target mapping network, style mapping network, and content mapping network;

[0056] Specifically, target pseudo-words, style pseudo-words, and content pseudo-words are essentially three types of features representing different meanings. Because they are features rather than specific text, they are called "pseudo-words." In CLIP (the Visual Language Big Model of this application), after computation by the CLIP visual encoder and text encoder, the image and text are unified into a single feature space. Therefore, features originally derived from the image can also be considered text features in this space. The specific meaning of these three pseudo-words is: using visual features to enhance text representation, so that the text features can precisely express the information contained in the image, avoiding the drawback of simply using text and failing to fully express visual information. The significance of using a mapping network is that the original image features contain too much redundant information; the mapping network simplifies the information related to specific content. Because the essence of pseudo-words is features, when embedding templates, the template is first calculated as a feature before embedding, and then the text features are calculated again after embedding, rather than directly embedding pseudo-words into the template and then calculating text features.

[0057] Specifically, the process of using the CLIP visual language large model to calculate image and text features, and using the CLIP model's visual encoder to calculate the visual global features of virtual data, can be represented as follows:

[0058] (1);

[0059] in, Representing the visual global features of virtual data. The visual encoder representing the CLIP visual language big model, This represents the input virtual data. The target mapping network is used to learn target pseudo-word features. The process of projecting visual global features onto the target pseudo-word using the target mapping network can be represented as:

[0060] (2);

[0061] in, Indicates the target pseudoword, This represents the target mapping network. The process of computing the visual local features of virtual data using the visual encoder of the CLIP visual language large model can be represented as:

[0062] (3);

[0063] in, Representing the visual local features of virtual data, This represents the visual encoder of the large visual language model that does not include the last layer, where L represents the number of layers in the visual encoder.

[0064] Next, the cross-attention scores of the visual local and global features of the virtual data are calculated, and the top k visual local features that are highly relevant to the image semantics are selected. This process can be represented as:

[0065] (4);

[0066] (5);

[0067] (6);

[0068] in, , , This represents the learnable weight matrix. Represents the query vector. Represents the key vector. Let A represent the value vector, and let A represent the attention weight matrix. This represents the attention weight after filtering. This represents the corresponding value vector. This represents the filtered visual local features.

[0069] Using a multilayer perceptron (MLP) as a style mapping network to learn style pseudoword features, the process of projecting visual local features onto style pseudowords using the style mapping network can be represented as:

[0070] (7);

[0071] in, Stylistic pseudo-words Representation style mapping network.

[0072] S4: Embed the target pseudowords and style pseudowords into a predefined target-style text template, and embed the target pseudowords into a predefined target text template. Process them separately through the CLIP text encoder to obtain target-style text features and target text features.

[0073] Specifically, the pseudo-word-based target-style text template uses the following format: "A [x] style of aphoto of [y]"., while the pseudo-word-based target text template uses the following format: "A photo of [y]".

[0074] In this context, square brackets "[]" indicate that pseudo-words need to be embedded, x represents style pseudo-words, and y represents target pseudo-words. The process of calculating text template features can be represented as follows:

[0075] (8);

[0076] (9);

[0077] in, The text encoder representing the CLIP visual language big model. This indicates the characteristics of the target-style text template. Indicates target-style text template, Indicates the target text template features. This indicates the target text template.

[0078] Use the target pseudoword obtained in step S3 and stylistic pseudo-words Embedded target-style text template features The process can be represented as:

[0079] (10);

[0080] in, Indicates target-style text features. This is a concatenation function used to embed pseudo-words into the template. The calculation process involves concatenating the pseudo-word features and template features together to form a feature sum. The target pseudo-word obtained in step S3 is used... Embedded target text template features The process can be represented as:

[0081] (11);

[0082] in, Indicates the features of the target text.

[0083] S5: Visual global features and target-style text features are used to construct the first positive sample pair, enabling the visual language large model to learn the target mapping network and style mapping network based on the first positive sample pair, so as to update the CLIP visual encoder;

[0084] Specifically, fine-tuning the visual encoder, target mapping network, and style mapping network of the CLIP visual language big model is carried out in two steps.

[0085] Visual global features of the virtual data obtained in step S3 and target-style text features Contrastive learning is performed to fine-tune the visual encoder of the large visual language model, learning the target-style mapping network and the style mapping network, using the following contrastive loss:

[0086] (12);

[0087] in, Represents each sample in a batch of data, with all subscripts containing... This refers to the first image in a batch of images. Each image represents a set of images for which the contrast loss is calculated individually. The contrast losses of all images are then summed to obtain the total contrast loss for this batch of images. This sum is then used for backpropagation to optimize the model. Similarly, the subscripts in the denominator... This represents the calculation of the first in this batch. When calculating the similarity between an image and all text features, the calculation is performed up to the [number]th image. One text feature; It is a similarity function. It is a temperature parameter;

[0088] N is the batch size, specifically the number of data points processed in a single computation. The computation process is not performed image by image, but rather in batches. Therefore, the contrastive loss function aims to maximize the similarity between each image in a batch and its corresponding text features, while minimizing the similarity between each image and the corresponding text of other images.

[0089] Contrastive learning involves calculating the similarity between visual global features and target-style text features using a contrastive loss function. Minimizing this loss function aims to make the visual global features of the image being calculated as similar as possible to its corresponding target-style text features, while minimizing their dissimilarity to the target-style text features of other images. This process optimizes the model by minimizing the loss function through backpropagation. In other words, this step is called contrastive learning because it uses a contrastive loss function. The loss function measures the difference between the current model's output and the true result; fine-tuning is the process of minimizing this loss function. All parameters of the model are fine-tuned automatically through backpropagation.

[0090] S6: Project the global visual features into the content mapping network to obtain the target content features;

[0091] Specifically: Using a Multilayer Perceptron (MLP) as the content mapping network to learn target content features, the process of projecting the visual global features obtained in step 3 into the target content feature space using the content mapping network can be represented as:

[0092] (13);

[0093] in, Indicates the characteristics of the target content. This indicates a content mapping network.

[0094] S7: Construct a second positive sample pair by combining the target content features and the target text features, and enable the content mapping network to perform comparative learning based on the second positive sample pair in order to update the content mapping network;

[0095] Specifically, through the target content features of virtual data and target text features Contrastive learning is performed to train a content mapping network, enabling the content mapping network to effectively learn the features of the target content. The contrastive loss used is:

[0096] (14);

[0097] S8: Calculate the target text features of the reference target image based on the updated visual language big model, and calculate the target content features of the real target image based on the updated visual language big model.

[0098] S9: Calculate the cosine similarity between the target content features of the real target image and the target text features of the reference target image, and obtain the target generalization recognition result based on the cosine similarity ranking result.

[0099] Specifically, the visual encoder and target-style mapping network of the large visual language model trained in step S5 are used to compute the target text features of the reference target image. The visual encoder of the large visual language model trained in step S5 and the content mapping network trained in step S7 are used to calculate the target content features of the real target image. Calculate the target content features of a real target image and target text features of the reference target image The cosine similarity between the targets is calculated and sorted according to the similarity to obtain the target generalization recognition result.

[0100] Specifically, the reference target image is the image used in target recognition to define which target needs to be identified. This concept comes from the task itself and is equivalent to providing the target to be identified. The real target image is the image used in the actual application. Virtual data is generated to compensate for the difficulty in obtaining real data and is only used for model training. When applying the model in practice, that is, verifying the model's generalization to real scenes, images of real scenes are needed, not the virtual data used during training. During the learning process, the model only focuses on extracting generalization features. Since the positive examples used in contrastive learning are images and text, a reference target image is not needed. However, during inference, the target that the model wants to find needs to be defined. In general contrastive learning, this contrast target is text (category), but this method does not involve text during inference; text is only used for training. The reference target image is used as the contrast target to achieve the purpose of finding the same target in different images. Directly using images as the target definition avoids the problem that similar targets are difficult to distinguish simply by text, and can achieve more granular target generalization recognition. The reference target image is equivalent to the text used in the inference process in general contrastive learning. The real target image is the image used in the actual application.

[0101] Specifically, the content feature shape of the real target image is [B, D], representing B images, each with D-dimensional features; the target text feature shape of the reference target image is [N, D], representing N reference target images / candidates (multi-image recognition; if only one image is recognized, N=1), each also with D-dimensional features. After calculating the cosine similarity, the shape becomes [B, N], where the th... The line represents the first The similarity between the nth image and all N texts; Column represents the first The similarity between a text and all B images is calculated. If there is only one reference target image, the cosine similarity matrix has a shape of [B, 1], where each value in the first column represents the similarity between a real target image and the reference target image, and these values ​​are used for sorting. The purpose of this method is to identify images with the same target as the reference target image from multiple real target images, given one (or more) reference target images. The essence of obtaining the generalized recognition result based on cosine similarity sorting is to match the real target image that is most similar to the reference target image. The final result is the real target image containing the same target as the reference target image. In other words, without inputting text information, only the reference target image is needed to find images containing the same target from other images.

[0102] This application presents a target generalization recognition method based on virtual data generation. It designs target-style text prompt templates and combines them with a large-scale text-to-image model to generate high-fidelity and diverse virtual data. Utilizing a large-scale visual language model and mapping network, it extracts and optimizes target content features, reducing the influence of style features on target content features. By calculating the cosine similarity between the target content features of a real target image and the target text features of a reference target image, it achieves target generalization recognition of real target images. This improves the adaptability and recognition accuracy of the model trained on virtual data in real-world scenarios, providing strong support for the value mining of virtual data in practical applications.

[0103] Based on the same inventive concept, this application also provides an apparatus for implementing the method described above. The solution provided by this apparatus is similar to the implementation described in the above method. Therefore, the specific limitations of the target generalization recognition apparatus embodiment based on virtual data generation provided below can be found in the limitations of the target generalization recognition method based on virtual data generation described above, and will not be repeated here.

[0104] Reference Figure 2 This embodiment discloses a target generalization recognition device 20 based on virtual data generation, comprising:

[0105] The prompt text generation unit 201 is used to generate prompt text based on the relevant features of the target's training data in the actual application scenario;

[0106] Virtual data generation unit 202 is used to input the prompt text into a pre-trained text-to-image model to generate virtual data of the target in the actual application scenario;

[0107] The virtual processing unit 203 is used to input the virtual data into the visual language big model and generate visual global features, target pseudowords, and style pseudowords of the virtual data; the visual language big model includes CLIP visual encoder, CLIP text encoder, target mapping network, style mapping network, and content mapping network.

[0108] The text feature generation unit 204 is used to embed the target pseudoword and style pseudoword into a predefined target-style text template and embed the target pseudoword into a predefined target text template, and process them respectively through the CLIP text encoder to obtain target-style text features and target text features;

[0109] The visual encoder update unit 205 is used to construct the visual global features and target-style text features into a first positive sample pair, so that the visual language big model learns the target mapping network and style mapping network based on the first positive sample pair to update the CLIP visual encoder.

[0110] The content feature generation unit 206 is used to project the visual global features into the content mapping network to obtain target content features;

[0111] The content mapping network update unit 207 is used to construct a second positive sample pair from the target content features and the target text features, so that the content mapping network can perform comparative learning based on the second positive sample pair to update the content mapping network;

[0112] Image feature calculation unit 208 is used to calculate target text features of reference target image based on the updated visual language big model, and to calculate target content features of real target image based on the updated visual language big model.

[0113] The target generalization recognition unit 209 is used to calculate the cosine similarity between the target content features of the real target image and the target text features of the reference target image, and obtain the target generalization recognition result based on the cosine similarity ranking result.

[0114] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0115] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0116] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0117] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0118] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0119] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0120] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application, and they should all be covered within the scope of the claims and specification of this application.

Claims

1. A target generalization recognition method based on virtual data generation, characterized in that, Includes the following steps: S1: Generate prompt text based on the relevant features of the target's training data in real-world application scenarios; S2: Input the prompt text into the pre-trained text-to-image model to generate virtual data of the target in the actual application scenario; S3: Input the virtual data into the visual language big model to generate visual global features, target pseudowords, and style pseudowords of the virtual data; the visual language big model includes CLIP visual encoder, CLIP text encoder, target mapping network, style mapping network, and content mapping network. S4: Embed the target pseudowords and style pseudowords into a predefined target-style text template, and embed the target pseudowords into the predefined target text template, and process them respectively through the CLIP text encoder to obtain target-style text features and target text features; S5: Construct the visual global features and target-style text features into a first positive sample pair, so that the visual language big model learns the target mapping network and style mapping network based on the first positive sample pair to update the CLIP visual encoder; S6: Project the visual global features into the content mapping network to obtain the target content features; S7: Construct the target content features and the target text features into a second positive sample pair, and enable the content mapping network to perform comparative learning based on the second positive sample pair to update the content mapping network; S8: Calculate the target text features of the reference target image based on the updated visual language big model, and calculate the target content features of the real target image based on the updated visual language big model. S9: Calculate the cosine similarity between the target content features of the real target image and the target text features of the reference target image, and obtain the target generalization recognition result based on the cosine similarity ranking result.

2. The method according to claim 1, characterized in that, Step S2 includes: S21: Design an image partitioning diagram to specify the generation area for each image data, used to store data of the same target under different styles; S22: Obtain the target's morphological data; S23: Generate multi-style, multi-objective training data based on prompt text, image segmentation diagrams, and morphological data; S24: Preprocess the multi-style multi-objective training data and crop it based on the image separator box to obtain virtual data.

3. The method according to claim 2, characterized in that, Step S3 includes: S31: The virtual data is processed by the CLIP visual encoder of the large visual language model to obtain global visual features; S32: Project the visual global features using a target mapping network to obtain target pseudo-words; S33: Use the output of the penultimate layer of the CLIP visual encoder as a visual local feature; S34: Calculate the cross-attention between the visual local features and the visual global features, and filter out visual local features that are highly related to the image semantics based on the attention score; S35: The selected visual local features are projected through a style mapping network to obtain style pseudowords.

4. The method according to claim 3, characterized in that, Step S4 includes: S41: Design two pseudoword-based text templates: a target-style text template and a target text template. S42: The target-style text template and the target text template are processed by the CLIP text encoder to obtain the target-style text template features and the target text template features; S43: Embed the target pseudowords and style pseudowords into the target-style text template features to obtain the target-style text features; S44: Embed the target pseudoword into the target text template features to obtain the target text features.

5. A target generalization recognition device based on virtual data generation, characterized in that, include: The prompt text generation unit is used to generate prompt text based on the relevant features of the target's training data in the actual application scenario; The virtual data generation unit is used to input the prompt text into a pre-trained text-to-image model to generate virtual data of the target in the actual application scenario. A virtual data processing unit is used to input the virtual data into a large visual language model to generate visual global features, target pseudowords, and style pseudowords of the virtual data; the large visual language model includes a CLIP visual encoder, a CLIP text encoder, a target mapping network, a style mapping network, and a content mapping network. The text feature generation unit is used to embed the target pseudoword and style pseudoword into a predefined target-style text template and embed the target pseudoword into a predefined target text template, and process them respectively through the CLIP text encoder to obtain target-style text features and target text features; The visual encoder update unit is used to construct the visual global features and target-style text features into a first positive sample pair, so that the visual language big model learns the target mapping network and style mapping network based on the first positive sample pair to update the CLIP visual encoder. A content feature generation unit is used to project the visual global features into the content mapping network to obtain target content features; The content mapping network update unit is used to construct a second positive sample pair from the target content features and the target text features, so that the content mapping network can perform comparative learning based on the second positive sample pair to update the content mapping network; The image feature calculation unit is used to calculate the target text features of the reference target image based on the updated visual language big model, and to calculate the target content features of the real target image based on the updated visual language big model. The target generalization recognition unit is used to calculate the cosine similarity between the target content features of the real target image and the target text features of the reference target image, and obtain the target generalization recognition result based on the ranking result of the cosine similarity.

Images