Image object positioning method and system based on a generative model
By using image feature extraction and text feature alignment, the generative model can locate objects while generating images, solving the problem of missing object location information in existing technologies and achieving efficient generation of large-scale image object localization datasets.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHANGHAI JIAOTONG UNIV
- Filing Date
- 2022-12-19
- Publication Date
- 2026-07-24
AI Technical Summary
Existing generative models ignore object location information during image generation, resulting in the need for manual annotation and limited generation quality, making it difficult to generate large-scale, multi-category object localization datasets.
Image and text features are extracted and aligned during the generation process using an image feature extraction network, a text encoder, a cross-modal feature alignment network, and a mask generation network to generate an image mask for object localization.
It accurately segments objects while generating images, producing image datasets with pixel-level labels, thus solving the need for manual annotation and is suitable for downstream tasks such as image segmentation and classification.
Smart Images
Figure CN118261971B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a method in the fields of computer vision and image generation, and particularly to an image object localization method and system based on a generative model. Background Technology
[0002] In recent years, with the proposal and application of various large-scale models, the demand for large-scale datasets has been increasing. Models can be pre-trained on large-scale datasets to learn powerful semantic information, which in turn helps various downstream tasks. Among them, the visual localization dataset required for visual localization tasks is a challenging task because it requires images, text titles / descriptions describing the content of the images, and pixel-level label segmentation information of each object in the text description. Manually annotating images is too costly, so some methods attempt to train generative models to generate large-scale object localization datasets through small-scale manual annotation. However, these methods currently have significant limitations. On the one hand, manual annotation is still required, and on the other hand, they can only generate images of a certain category, resulting in relatively limited effects.
[0003] Recently, text-to-image generative models have attracted increasing attention. These models learn semantic associations between images and text titles from a large number of image-text title pairs, thus generating highly relevant and realistic images with high clarity. However, current methods mostly focus on generating better images, neglecting another important issue in the generation process: object location information. This is because since the generative model can generate corresponding objects based on given text, there must be object location information involved in this process. By extracting this information, object localization can be achieved. While there are many existing image generation models, they are primarily focused on image generation and do not consider the aforementioned technical issues of this application.
[0004] A search revealed that invention application CN115171838A discloses a training method for a medical report generation model based on cross-modal fusion. This invention achieves cross-modal recognition and improves the model's recognition accuracy and robustness. However, this invention mainly utilizes medical images to generate corresponding medical reports, using image features to generate new text features, and cannot generate the positional information of objects in the image simultaneously. Summary of the Invention
[0005] To address the shortcomings of existing technologies, this invention considers utilizing existing generative models to extract object position information during the generation process, and provides an image object localization method and system based on generative models.
[0006] According to one aspect of the present invention, an image object localization method based on a generative model is provided, wherein the process of generating an image from noise and textual cues using the generative model includes:
[0007] An image feature extraction network is used to obtain image features during the generation process of synthesizing images from noise and text prompts;
[0008] For a specific object category in the text prompt, the corresponding text features are extracted using a text encoder;
[0009] The text features and image features are aligned using a cross-modal feature alignment network to obtain the aligned representation;
[0010] A mask generation network is used to map the aligned representation to an image mask, and the image object localization result is obtained based on the image mask.
[0011] Optionally, the image feature extraction network extracts image features from different locations of the generative model, aggregates the image features, and then uses a convolutional neural network to extract and reduce the dimensionality of the features through convolution operations to obtain more refined image features.
[0012] Optionally, the step of using an image feature extraction network to obtain image features during the generation process includes:
[0013] Input features Where h is the height of the generated image, w is the width of the generated image, and d i For feature f i The dimension;
[0014] The image feature extraction network first upsamples each feature, then performs a convolution operation using a convolutional neural network, and finally connects them together;
[0015] The entire process of feature extraction by the image feature extraction network is represented as follows:
[0016]
[0017] in, This represents the image features finally extracted and processed by the image feature extraction network. That is, the feature resolution remains consistent with the original image at this point, where D represents the final feature dimension, and Φ v-enc This represents an image feature extraction network.
[0018] Optionally, the text encoder extracts the corresponding text features. For the text prompts in the generation model: first, extract the corresponding object keywords from the text prompt statements; then, put the object keywords into the template statements to form complete sentences, and send them to the text encoder to extract the corresponding text features.
[0019] Optionally, the text encoder is used to extract corresponding text features, wherein the text encoder Φ is used. t-enc This is a pre-trained CLIP model that has a strong ability to map textual information to the feature domain, i.e.
[0020] E object =Φ t-enc (g(y))
[0021] Where y represents the input text information, E object The text features that represent the final output Here, N represents the total number of objects in the generated image, and d text Dimensions of text features.
[0022] Optionally, aligning the text features and the image features using a cross-modal feature alignment network includes:
[0023] The mapping network first maps the input text features and image features to the same feature dimension d. e ;
[0024] Then, a cross-modal feature alignment network is used for alignment. This network is based on a transformer decoder, using the input text features as the query value and the input image features as the key and value, and outputting the final aligned representation.
[0025]
[0026] Among them, E aligned Φ represents the final aligned representation. transformer-D Represents a cross-modal feature alignment network. W Q W K W V These represent the learnable mapping parameters corresponding to the query value, key value, and value, respectively. This represents the image features finally extracted and processed by the image feature extraction network.
[0027] Optionally, the cross-modal feature alignment network also uses a cross-attention structure to implement the attention mechanism, including:
[0028] First, using the mapping parameter W Q Map text features to a query matrix Q = W Q ·E object ;
[0029] Secondly, using the mapping parameter W K W V The input image features are mapped into key and value matrices.
[0030] Finally, the attention level is calculated:
[0031]
[0032] Where d is the dimension of the mapped matrix, and K T The key-value matrix is the transpose of the key-value matrix, and softmax is the activation function. The attention mechanism described above calculates the similarity between the query matrix and the key-value matrix, and then recombines the values of the value matrix based on the calculation results.
[0033] Optionally, the step of using a mask generation network to map the aligned representation to an image mask, and obtaining the image object localization result based on the image mask, includes:
[0034] For the output features obtained by the cross-modal feature alignment network, a mask generation network is used to map them into an image mask:
[0035]
[0036] Where, Φ MLP The representative mask generation network consists of multiple multilayer perceptron networks, Φ MLP (E aligned The output is a feature representation of the mask, which is compared with the output of the image feature extraction network. After performing the dot product calculation, the final mask result is obtained. Where h is the height of the generated image and w is the width of the generated image;
[0037] Set a first threshold, and the mask result After processing with the sigmoid function, the values of the portion greater than the first threshold are set to 1, representing the portion where the target object is located, and the values of the portion less than the first threshold are set to 0, representing the background area in the image.
[0038] Optionally, the loss function L for the entire method is:
[0039]
[0040] in, The real image mask is used to supervise all network structures except the generative model. N is the total number of training samples. This loss function is essentially a binomial cross-entropy loss.
[0041] A second aspect of the present invention provides an image object localization system based on a generative model, used for the generation process of synthesizing an image from noise and text prompts using a generative model, comprising:
[0042] The image feature extraction module uses an image feature extraction network to obtain image features from the process of generating an image synthesized from noise and text prompts;
[0043] The text feature extraction module extracts the corresponding text features for a specific object category in the text prompt using a text encoder.
[0044] The cross-modal feature alignment module uses a cross-modal feature alignment network to align the text features and the image features to obtain the aligned representation;
[0045] The mask generation module uses a mask generation network to map the aligned representation to an image mask, and obtains the image object localization result based on the image mask.
[0046] The above modules can be referenced from the specific implementation techniques of each step in the image object localization method based on generative models.
[0047] Compared with the prior art, the embodiments of the present invention have at least one of the following advantages:
[0048] This invention provides an image object localization method and system based on a generative model. First, an image feature extraction network is used to extract image features from the image generation process of text prompts and noise synthesis. Then, for a specific object category in the text prompt, a text encoder is used to extract corresponding text features. The obtained text and image features are then aligned and fused using a cross-modal feature alignment network to obtain an aligned representation. Finally, a mask generation network maps the aligned representation to an image mask. By utilizing feature representations from the natural image generation process and aligning them with text features of the corresponding object category, object localization information is obtained. Simultaneously, the mask of the corresponding object is obtained during image generation, thus enabling object localization.
[0049] This invention provides an image object localization method and system based on a generative model, which aligns image features with text features in the feature space. Guided by text features, it obtains the positional information of objects in the image during the image generation process, and can accurately segment objects in the generated image from the background.
[0050] This invention provides an image object localization method and system based on a generative model, which can generate masks corresponding to objects in the image while generating a natural image. The mask contains pixel-level label segmentation information of the objects in the image, thereby generating a large-scale image visual localization dataset, solving the problems of existing datasets requiring manual annotation and having limited effectiveness. At the same time, the generated dataset can be used for downstream tasks such as image segmentation, image classification, and model pre-training. Attached Figure Description
[0051] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:
[0052] Figure 1 This is a flowchart of an image object localization method based on a generative model according to an embodiment of the present invention;
[0053] Figure 2 This is a block diagram of an image object localization system based on a generative model according to an embodiment of the present invention. Detailed Implementation
[0054] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.
[0055] Current methods for generative models largely focus on generating better images, neglecting another crucial issue in the generation process: object location information. This is because since generative models can generate corresponding objects from given text, object location information must exist during this process. By extracting this information, object localization can be achieved. Based on this research finding, this invention proposes an image object localization method and system based on a generative model.
[0056] like Figure 1 The diagram shows a flowchart of an image object localization method based on a generative model, according to an embodiment of the present invention. Specifically, for the process of generating an image from noise and text prompts using a generative model, the image object localization method based on the generative model can be performed as follows:
[0057] S11: Use an image feature extraction network to obtain image features during the generation process of synthesizing images from noise and text prompts;
[0058] S12: For a specific object category in the text prompt, use a text encoder to extract the corresponding text features;
[0059] S13: Use a cross-modal feature alignment network to align the text features obtained in S12 and the image features obtained in S11 to obtain the aligned representation;
[0060] S14: Use a mask generation network to map the aligned representation to an image mask, and obtain the image object localization result based on the image mask.
[0061] The above embodiments of the present invention utilize feature representations in the natural image generation process to obtain object location information by aligning with text features of the corresponding object category. Simultaneously, the mask of the corresponding object is obtained while generating the image, thereby generating the object location.
[0062] In the above embodiments, S11 and S12 can be executed in any order, without any particular sequence, or they can be executed in parallel.
[0063] In some embodiments of the present invention, the image feature extraction network in S11 includes two steps: feature aggregation and convolution operation. Features from different locations of the generation model are fed into the image feature extraction network, which aggregates the features and then performs convolution operation using a convolutional neural network.
[0064] Specifically, the features of the input Where h is the height of the generated image, w is the width of the generated image, and d i For feature f i The dimensions of these features are different because they are extracted from different locations and therefore have different dimensions. Therefore, the image feature extraction network first upsamples each feature, then performs convolution operations using a convolutional neural network, and finally connects them together.
[0065] The entire process of feature extraction by the image feature extraction network can be represented as:
[0066]
[0067] in, This represents the final image features output by the image feature extraction network. That is, the feature resolution remains consistent with the original image at this point, where D represents the final feature dimension, and Φ v-enc This represents an image feature extraction network.
[0068] The purpose of the image feature extraction network in S11 above is to aggregate and convolve the features from the image generation process of the generative model, and output the final image features used for alignment.
[0069] In some embodiments of the present invention, in S12, for the text prompt statements in the generated model, firstly, the corresponding object keywords, such as "cat," "dog," "airplane," etc., are extracted from the text prompt statements. Then, in order to obtain the corresponding text features, these object keywords are put into template statements to form complete sentences. The above process is represented by g. Meanwhile, the template statements used in the method include: "a photograph of...", "a scene containing...", and "a picture of...".
[0070] For the required features representing textual information, the text prompts input during image generation are used to extract corresponding text features through a pre-trained text encoder. The text encoder Φ is then used. t-enc This is a pre-trained CLIP model that has a strong ability to map textual information to the feature domain, i.e.
[0071] E object =Φ t-enc (g(y))
[0072] Where y represents the input text information, E object The text features that represent the final output Here, N represents the total number of objects in the generated image, and d text This refers to the dimension of the text features. The purpose of the text encoder in S12 above is to generate corresponding text features for the target object, which will be used for subsequent feature alignment.
[0073] In some embodiments of the present invention, in S13, a mapping network is used to first map the input text features and image features to the same feature dimension d. e Text features are directly mapped, and the number N remains unchanged. For image features, linear expansion is first used to expand the features into N′×D′, and then mapping is performed. Here, N′ represents the number of image features, and D′ represents the dimension of the expanded image features.
[0074] The cross-modal feature alignment network is a network based on a transformer decoder. It uses the mapped text features as the query value and the mapped image features as the key and value, and outputs the final aligned representation.
[0075]
[0076] Among them, E aligned Φ represents the final aligned representation. transformer-D Represents a cross-modal feature alignment network. W Q W K W VThese represent the learnable mapping parameters corresponding to the query value, key value, and value, respectively.
[0077] The cross-modal feature alignment network in S13 above aligns text features and image features, thereby extracting the positional information of the target object from the image features. Most existing technologies use prior information of image features to obtain the positional information of the object, while the embodiments of the present invention use the alignment of text features and image features to obtain more accurate positional information.
[0078] Furthermore, in some preferred embodiments, the cross-modal feature alignment network also uses an attention-based mechanism, which can be implemented through a cross-attention structure.
[0079] Specifically, the cross-modal feature alignment network includes a cross-attention structure, which first utilizes the mapping parameter W Q Map text features to a query matrix Q = W Q ·E object Using mapping parameter W K W V The input image features are mapped into key and value matrices. Finally, the attention level is calculated:
[0080]
[0081] Where d is the dimension of the mapped matrix, and K T The key-value matrix is transposed, and softmax is the activation function. The core idea of the above attention mechanism is to calculate the similarity between the query matrix and the key-value matrix, and then recombine the values of the value matrix based on the calculation results. Using a cross-attention structure can better achieve this alignment operation. By aligning through the similarity between text features and image features, the barrier between the text feature domain and the image feature domain can be effectively broken down, thereby enabling the extraction of location information.
[0082] In some embodiments of the present invention, in S14, the output features obtained by the cross-modal feature alignment network are mapped to an image mask using a mask generation network:
[0083]
[0084] Where, Φ MLP This represents a mask generation network, composed of multiple perceptron networks connected in series, where the output of one perceptron network serves as the input to the next; Φ MLP (E aligned The output is a feature representation of the mask, which is compared with the output of the image feature extraction network. After performing the dot product calculation, the final mask result can be obtained. The mask result After processing with the sigmoid function, values greater than the first threshold are set to 1, representing the area where the target object is located, while values less than the first threshold are set to 0, representing the background area in the image. Specifically, in one embodiment, the first threshold can be set to 0.5, meaning values greater than 0.5 are set to 1, representing the area where the target object is located, and values less than 0.5 are set to 0, representing the background area in the image. The mask generation network in this embodiment can be implemented using existing technology.
[0085] In the above embodiments of the present invention, under a pre-designed loss constraint, an image feature extraction network, a text encoder, a cross-modal feature alignment network, and a mask generation network are trained to generate image location information. The loss function L is:
[0086]
[0087] in, is the real image mask used to supervise the image feature extraction network, text encoder, cross-modal feature alignment network, and mask generation network. N is the total number of training samples. This loss function is essentially a binomial cross-entropy loss. The final mask result is represented by σ, which stands for the Sigmoid function.
[0088] Based on the same technical concept described above, embodiments of the present invention also provide an image object localization system based on a generative model, such as... Figure 2 The diagram shown is a framework diagram of an image object localization system based on a generative model according to this embodiment. Specifically, it includes: an image feature extraction module, a text feature extraction module, a cross-modal feature alignment module, and a mask generation module. The image feature extraction module extracts image features from the process of synthesizing an image from noise and text prompts using the generative model. The text feature extraction module extracts corresponding text features for a specific object category in the text prompts. The cross-modal feature alignment module aligns the text features and image features to obtain the aligned representation. The mask generation module uses a mask generation network to map the aligned representation to an image mask, and obtains the image object localization result based on the image mask.
[0089] Specifically, the modules of the image object localization system based on the generative model in this embodiment can be implemented with reference to the technical implementation of each step in the image object localization method based on the generative model described above, and will not be repeated here.
[0090] The above embodiments of the present invention utilize feature representations in the natural image generation process to obtain object location information by aligning with text features of the corresponding object category. Simultaneously, the mask of the corresponding object is obtained while generating the image, thereby generating the object location.
[0091] The above embodiments of the present invention solve the object localization problem based on the generative model, and can be applied to various images, not limited to a specific image; the above embodiments of the present invention align image features and text features to obtain the position information of the corresponding object in the image, and finally generate a new image, thus solving the problem of generating the position information of the object in the image at the same time as generating the image.
[0092] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention. The above preferred features can be used in any combination without conflict.
Claims
1. An image object localization method based on a generative model, comprising the process of generating an image from noise and text prompts and generating the localization result, characterized in that, include: An image feature extraction network is used to obtain image features during the generation process of synthesizing images from noise and text prompts; For a specific object category in the text prompt, the corresponding text features are extracted using a text encoder; The text features and image features are aligned using a cross-modal feature alignment network to obtain the aligned representation; A mask generation network is used to map the aligned representation to an image mask, and the image object localization result is obtained based on the image mask. The image feature extraction network extracts image features from different locations in the generation model, aggregates the image features, and then uses a convolutional neural network to extract and reduce the dimensionality of the features through convolution operations to obtain more refined image features. The step of using an image feature extraction network to obtain image features during the generation process includes: Input features , ,in It is the height of the generated image. It is the width of the generated image. Features The dimension; The image feature extraction network first upsamples each feature, then performs a convolution operation using a convolutional neural network, and finally connects them together; The entire process of feature extraction by the image feature extraction network is represented as follows: ； in, This represents the image features finally extracted and processed by the image feature extraction network. This means that the features at this point maintain the same resolution as the original image. Represents the final feature dimension. This represents an image feature extraction network.
2. The image object localization method based on a generative model according to claim 1, characterized in that, The text encoder is used to extract corresponding text features for the text prompts in the generated model: First, extract the corresponding object keywords from the text prompts; Then, the object keywords are put into the template statement to form a complete sentence, and sent to the text encoder to extract the corresponding text features.
3. The image object localization method based on a generative model according to claim 2, characterized in that, The text encoder is used to extract corresponding text features. This is a pre-trained CLIP model that has a strong ability to map textual information to the feature domain, i.e. ； in, Represents the input text information. The text features that represent the final output ,here This represents the total number of objects in the generated image. Dimensions of text features.
4. The image object localization method based on a generative model according to claim 1, characterized in that, The step of aligning the text features and the image features using a cross-modal feature alignment network includes: The mapping network first maps the input text features and image features to the same feature dimension. ; Then, a cross-modal feature alignment network is used for alignment. This network is based on a transformer decoder, using the input text features as the query value and the input image features as the key and value, and outputting the final aligned representation. ； in, This represents the final aligned representation. Represents a cross-modal feature alignment network. , These represent the learnable mapping parameters corresponding to the query value, key value, and value, respectively. This represents the image features finally extracted and processed by the image feature extraction network.
5. The image object localization method based on a generative model according to claim 4, characterized in that, The cross-modal feature alignment network also uses a cross-attention structure to implement the attention mechanism, including: First, using mapping parameters Mapping text features to a query matrix ; Secondly, using mapping parameters The input image features are mapped into key and value matrices. ; Finally, the attention level is calculated: ； in, Let be the dimension of the mapped matrix. The transpose of the key-value matrix. As the activation function, the attention mechanism described above calculates the similarity between the query matrix and the key-value matrix, and then recombines the values of the value matrix based on the calculation results.
6. The image object localization method based on a generative model according to claim 1, characterized in that, The process of using a mask generation network to map the aligned representation to an image mask, and obtaining the image object localization result based on the image mask, includes: For the output features obtained by the cross-modal feature alignment network, a mask generation network is used to map them into an image mask: ； in, The representative mask generation network consists of multiple multilayer perceptron networks. The output is a feature representation of the mask, which is compared with the output of the image feature extraction network. After performing the dot product calculation, the final mask result is obtained. , ,in It is the height of the generated image. It is the width of the generated image; Set a first threshold, and the mask result After processing with the sigmoid function, the values of the portion greater than the first threshold are set to 1, representing the portion where the target object is located, and the values of the portion less than the first threshold are set to 0, representing the background area in the image.
7. The image object localization method based on a generative model according to claim 1, characterized in that, The loss function of the entire method for: ； in, The image mask is used to supervise all network structures except the generative model. Given the total number of training samples, this loss function is essentially a binomial cross-entropy loss.
8. An image object localization system based on a generative model, used for the generation process of synthesizing images from noise and text prompts using a generative model, characterized in that, include: The image feature extraction module uses an image feature extraction network to obtain image features from the process of generating an image synthesized from noise and text prompts; The text feature extraction module extracts the corresponding text features for a specific object category in the text prompt using a text encoder. The cross-modal feature alignment module uses a cross-modal feature alignment network to align the text features and the image features to obtain the aligned representation; The mask generation module uses a mask generation network to map the aligned representation to an image mask, and obtains the image object localization result based on the image mask. The image feature extraction module extracts image features from different locations in the generation model, aggregates the image features, and then uses a convolutional neural network to extract and reduce the dimensionality of the features through convolution operations to obtain more refined image features; wherein, the step of using the image feature extraction network to obtain the image features in the generation process includes: Input features , ,in It is the height of the generated image. It is the width of the generated image. Features The dimension; The image feature extraction network first upsamples each feature, then performs a convolution operation using a convolutional neural network, and finally connects them together; The entire process of feature extraction by the image feature extraction network is represented as follows: ； in, This represents the image features finally extracted and processed by the image feature extraction network. This means that the features at this point maintain the same resolution as the original image. Represents the final feature dimension. This represents an image feature extraction network.