Image processing method and device based on user intention, equipment and medium

Abstract

The invention provides a user intention-based image processing method and device, equipment and a medium. The method comprises the steps of obtaining a to-be-processed image and an intention description text of a user on the to-be-processed image; the intention description text indicates a target object in which a user is interested in at least one object of the to-be-processed image; based on the intention description text and the to-be-processed image, adopting a target multi-modal large model to generate a target class activation thermodynamic diagram corresponding to the to-be-processed image; the target class activation thermodynamic diagram is used for indicating the degree of interest of the user to each object; based on the intention description text, segmenting the to-be-processed image by adopting an image segmentation device to obtain candidate images corresponding to the objects; and based on the target class activation thermodynamic diagram, screening out a target image corresponding to the target object from the candidate images. The image of the object in which the user is interested is effectively obtained, the individual requirement of the user is met, the segmentation of the unseen target in the image instance segmentation is realized, and the problem that one model cannot be suitable for multiple scenes is solved.

Description

Technical Field

[0001] The present application relates to the field of large model technology, and in particular to an image processing method, device, equipment and medium based on user intention. Background Art

[0002] Remote sensing image instance segmentation refers to the technology of accurately segmenting each independent object (or instance) in a remote sensing image. This technology not only needs to distinguish objects of different categories, but also needs to separately label and segment each independent object in the same category.

[0003] With the continuous development of computer technology and artificial intelligence technology, remote sensing image instance segmentation technology has become one of the research hotspots in the field of remote sensing, and has been widely used in many fields, such as agriculture, forestry, urban planning, environmental monitoring, etc. In the field of agriculture, remote sensing image instance segmentation technology can be used to analyze remote sensing images to automatically extract information such as crop types, growth status, pests and diseases; in the field of urban planning, remote sensing image instance segmentation technology can be used to process remote sensing images to automatically classify and extract information such as urban buildings, roads, and green spaces; in the field of environmental monitoring, remote sensing image instance segmentation technology can be used to analyze remote sensing images to automatically discover and evaluate problems such as water pollution and land degradation. It can be seen that remote sensing image instance segmentation technology has great application value.

[0004] However, at present, remote sensing image instance segmentation methods need to be implemented in a pre-defined model framework. There is a problem that a single model cannot cover multiple remote sensing application scenarios, and the extracted object information cannot meet the personalized needs of users. Summary of the invention

[0005] The present application aims to solve one of the technical problems in the related art at least to some extent.

[0006] To this end, the purpose of the present application is to propose an image processing method, device, equipment and medium based on user intention, which can screen candidate images obtained by image segmentation with reference to the target class activation heat map, can effectively identify objects of interest to the user in the image, and can improve the operability and accuracy of the images of the objects of interest to the user, improve the image processing effect based on user intention, meet the personalized needs of users, and can realize the segmentation of unseen targets in image instance segmentation, solving the problem that one model cannot be applied to multiple scenarios.

[0007] To achieve the above-mentioned purpose, the image processing method based on user intention proposed in the first aspect of the present application includes:

[0008] Acquire an image to be processed and a user's intention description text for the image to be processed; wherein the intention description text describes at least one object of the image to be processed and indicates a target object of interest to the user in the at least one object;

[0009] Based on the intention description text and the image to be processed, a target multimodal large model is used to generate a target class activation heat map corresponding to the image to be processed; wherein the target class activation heat map is used to indicate the user's interest in each of the objects;

[0010] Based on the intention description text, segment the image to be processed using an image segmentor to obtain candidate images corresponding to each of the objects;

[0011] Based on the target class activation heat map, a target image corresponding to the target object is screened out from each of the candidate images.

[0012] To achieve the above-mentioned purpose, the image processing device based on user intention proposed in the second aspect of the present application includes:

[0013] An acquisition module, used to acquire an image to be processed and a user's intention description text for the image to be processed; wherein the intention description text describes at least one object of the image to be processed and indicates a target object of interest to the user in the at least one object;

[0014] A generation module, used to generate a target class activation heat map corresponding to the image to be processed based on the intention description text and the image to be processed, using a target multimodal large model; wherein the target class activation heat map is used to indicate the user's interest in each of the objects;

[0015] A segmentation module, configured to segment the image to be processed using an image segmentor based on the intention description text to obtain candidate images corresponding to each of the objects;

[0016] A screening module is used to screen out a target image corresponding to the target object from each of the candidate images based on the target class activation heat map.

[0017] The electronic device proposed in the third aspect embodiment of the present application includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the image processing method based on user intention proposed in the first aspect embodiment of the present application.

[0018] The fourth aspect embodiment of the present application proposes a non-temporary computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, it implements the image processing method based on user intention as proposed in the first aspect embodiment of the present application.

[0019] The fifth aspect embodiment of the present application proposes a computer program product. When the instructions in the computer program product are executed by a processor, the image processing method based on user intention proposed in the first aspect embodiment of the present application is executed.

[0020] The image processing method, device, equipment and medium based on user intention provided by the present application include at least the following beneficial effects:

[0021] Obtain an image to be processed and a user's intention description text for the image to be processed; wherein the intention description text describes at least one object of the image to be processed and indicates a target object of interest to the user in at least one object; based on the intention description text and the image to be processed, a target multimodal large model is used to generate a target class activation heat map corresponding to the image to be processed; wherein the target class activation heat map is used to indicate the user's interest in each object; based on the intention description text, an image segmenter is used to segment the image to be processed to obtain candidate images corresponding to each object; based on the target class activation heat map, a target image corresponding to the target object is screened from each candidate image. Thus, the candidate images obtained by image segmentation can be screened with reference to the target class activation heat map, the objects of interest to the user in the image can be effectively identified, and the operability and accuracy of the images of the objects of interest to the user can be improved, the image processing effect based on the user's intention is improved, the personalized needs of the user are met, and the segmentation of unseen targets in image instance segmentation can be realized, solving the problem that one model cannot be applied to multiple scenarios.

[0022] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0024] Figure 1 is a flowchart of an image processing method based on user intention proposed in one embodiment of the present application;

[0025] Figure 2 is a flowchart of an image processing method based on user intention proposed in another embodiment of the present application;

[0026] Figure 3 is a flowchart of an image processing method based on user intention proposed in another embodiment of the present application;

[0027] Figure 4is a flowchart of an image processing method based on user intention proposed in another embodiment of the present application;

[0028] Figure 5 is a flowchart of an image processing method based on user intention proposed in another embodiment of the present application;

[0029] Figure 6 is a flowchart of an image processing method based on user intention proposed in another embodiment of the present application;

[0030] Figure 7 It is a flowchart of the image processing method based on user intention provided by this application;

[0031] Figure 8 It is a schematic diagram of the processing flow for obtaining the target class activation heat map provided by this application;

[0032] Fig. 9 It is a schematic diagram of a processing flow for obtaining a candidate instance mask map corresponding to an entity of interest to a user provided in the present application;

[0033] Fig.10 is a structural diagram of image processing based on user intention proposed in another embodiment of the present application;

[0034] Fig.11 A block diagram of an exemplary electronic device suitable for implementing the embodiments of the present application is shown. DETAILED DESCRIPTION

[0035] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be construed as limiting the present application. On the contrary, the embodiments of the present application include all changes, modifications and equivalents that fall within the spirit and connotation of the appended claims.

[0036] At present, remote sensing image instance segmentation methods have made significant progress. However, in related technologies, remote sensing image instance segmentation methods need to be implemented in a predefined model framework. There is a single model that cannot cover multiple remote sensing application scenarios, which makes the extracted object information unable to meet the personalized needs of users and difficult to obtain labeled data sets. Specifically, first, most models need to clarify the category of objects in advance, and instance segmentation cannot be performed for undefined or new categories; second, the current model is a general instance segmentation method, which cannot extract the corresponding objects according to the user's actual intention; finally, most of the current instance segmentation methods rely on large-scale standard data sets, and it is very difficult to obtain large-scale labeled data in remote sensing scenes.

[0037] With the rapid development of multimodal large models, it uses the natural language input by the user as the intention description, allowing the user to clearly specify the object information that needs to be extracted. By using the instance segmentation model and the large language model in the open environment to carry out multimodal learning, the instance segmentation of remote sensing image objects can be realized. It can extract object types not covered in the training set, and can extract information from remote sensing images according to user needs. The use of natural language as the weak label information of objects in remote sensing images can help alleviate the problem of insufficient labeled data.

[0038] In response to at least one of the above problems, the present application proposes an image processing method, apparatus, device and medium based on user intent.

[0039] The following describes the image processing method, apparatus, device and medium based on user intention in embodiments of the present application with reference to the accompanying drawings.

[0040] Figure 1 It is a flowchart of an image processing method based on user intention proposed in one embodiment of the present application.

[0041] It should be noted that the executor of the image processing method based on user intention in this embodiment is an image processing device based on user intention, which can be implemented by software and / or hardware. The device can be configured in an electronic device, and the electronic device can include but is not limited to a terminal, a server, etc. For example, the terminal can be a mobile phone, a PDA, etc.

[0042] like Figure 1 As shown, the image processing method based on user intention may include the following steps:

[0043] Step S101, obtaining an image to be processed and a text describing the user's intention for the image to be processed.

[0044] The intention description text describes at least one object of the image to be processed and indicates a target object of interest to the user in the at least one object.

[0045] The image to be processed refers to, for example, a remote sensing image currently to be used in an image processing task based on user intention.

[0046] In an embodiment of the present application, the image to be processed may be acquired based on an image acquisition device (such as a remote sensing image acquisition device) for a preset area, or the target image may be acquired from an image database storing multiple images as the image to be processed, and there is no limitation on this.

[0047] In an embodiment of the present application, the intent description text may be obtained based on user input, wherein the user input includes but is not limited to voice input, handwriting input, keyboard input, and the like.

[0048] As an example, assume that the acquired image to be processed includes trees, basketball courts, and tennis courts, and the corresponding intent description text is "the basketball court to the right of the tennis court", wherein the objects in the image to be processed described by the intent description text include "tennis court" and "basketball court", and indicates the target object "basketball court" that the user is interested in among the "tennis court" and "basketball court".

[0049] Step S102, based on the intent description text and the image to be processed, a target multimodal large model is used to generate a target class activation heat map corresponding to the image to be processed.

[0050] It should be noted that the target class activation heat map can indicate the distribution of the user's interest in each area in the image to be processed, that is, it can be used to indicate the user's interest in each object in the image to be processed.

[0051] In an embodiment of the present application, the target multimodal large model identifies the area in the image to be processed corresponding to the object described in the intent description text through multimodal learning and deep learning, and outputs the target class activation heat map corresponding to the image to be processed.

[0052] As a possible implementation, the target multimodal large model may include a first text encoder, a visual encoder, and a target language model. For example, assuming that the target multimodal large model is an EarthGPT model, and the target language model is an LLaMA (Large Language Model Architecture) model.

[0053] It should be noted that the above examples of the target multimodal large model are only exemplary. In practical applications, the target multimodal large model may also be other, and this application does not impose any limitation on this.

[0054] Step S103 : Based on the intention description text, an image segmenter is used to segment the image to be processed to obtain candidate images corresponding to each object.

[0055] Among them, the image segmenter can be an open vocabulary instance segmentation model (also known as an open vocabulary domain instance segmentation model). For example, the image segmentation model can be but is not limited to a SAM (Segment Anything Model) model, an OpenSeeD model, etc., and this application does not impose any restrictions on this.

[0056] Therefore, the open vocabulary instance segmentation model is used for image segmentation. It can segment unseen targets in an open environment without being restricted to specific categories or objects, that is, it can segment any object in any image, improve the image segmentation capability and flexibility, and has significant generalization ability, which is suitable for solving image segmentation problems in different scenarios.

[0057] In the embodiment of the present application, the image segmentor can identify and obtain each object in the intended description text, and segment the image to be processed according to each object to obtain a candidate image corresponding to each object.

[0058] Step S104: based on the target class activation heat map, a target image corresponding to the target object is screened out from each candidate image.

[0059] It can be understood that since the target class activation heat map indicates the user's interest in each object in the processed image, it means that the target class activation heat map can indicate the object in the processed image that the user is most interested in, that is, the target object. Therefore, based on the target class activation heat map, the target image corresponding to the target object can be screened out from the candidate images corresponding to each object.

[0060] The image processing method of the user intention of the embodiment of the present application is obtained by obtaining an image to be processed and a user's intention description text for the image to be processed; wherein the intention description text describes at least one object of the image to be processed and indicates a target object of interest to the user in at least one object; based on the intention description text and the image to be processed, a target multimodal large model is used to generate a target class activation heat map corresponding to the image to be processed; wherein the target class activation heat map is used to indicate the user's interest in each object; based on the intention description text, an image segmenter is used to segment the image to be processed to obtain candidate images corresponding to each object; based on the target class activation heat map, the target image corresponding to the target object is screened from each candidate image. Thus, the candidate images obtained by image segmentation can be screened with reference to the target class activation heat map, the objects of interest to the user in the image can be effectively identified, and the operability and accuracy of the images of the objects of interest to the user can be improved, the image processing effect based on the user's intention is improved, the personalized needs of the user are met, and the segmentation of unseen targets in image instance segmentation can be realized, solving the problem that one model cannot be applied to multiple scenes.

[0061] In order to clearly explain how, in the above embodiments of the present application, a target class activation heat map is generated based on the intent description text and the image to be processed using a target multimodal large model, the present application also proposes an image processing method based on user intent.

[0062] Figure 2It is a flowchart of an image processing method based on user intention proposed in another embodiment of the present application.

[0063] like Figure 2 As shown, based on the above embodiment of the present application, the image processing method based on user intention may further include the following steps:

[0064] Step S201: input the intent description text into the first text encoder in the target multimodal large model to obtain the first text feature output by the first text encoder.

[0065] In an embodiment of the present application, after the intent description text is input into the first text encoder in the target multimodal large model, the first text encoder can output text features corresponding to the intent description text, which are recorded as first text features in the present application.

[0066] As an example, the first text encoder may include a word segmenter and an embedding model. For example, the word segmenter is a Tokenzier model and the embedding model is an Embedding model. After the intent description text is input into the first text encoder in the target multimodal large model, the word segmenter in the first text encoder segments the intent description text to obtain a discrete character unit token sequence. Then, each character in the character unit sequence is mapped into a vector through the embedding model to obtain the first text feature.

[0067] Step S202: input the image to be processed into a visual encoder in the target multimodal large model to obtain a first image feature output by the visual encoder.

[0068] In an embodiment of the present application, after the image to be processed is input into the visual encoder in the target multimodal large model, the visual encoder can output image features corresponding to the image to be processed, which are recorded as the first image features in the present application.

[0069] As an example, the visual encoder may include a first network layer and a second network layer in parallel. After the image to be processed is input into the visual encoder, the global features of the image to be processed are extracted through the first network layer, and the local features of the image to be processed are extracted through the second network layer. Finally, the global features and the local features are fused to obtain the first image features.

[0070] For example, assuming that the first network layer is a ViT (Vision Transformer) model and the second network layer is a CNN (Convolutional Neural Network) model, in the channel of the ViT model, first, the image to be processed is segmented to obtain multiple image slices (i.e., image tokens) of a fixed size (e.g., m×m), and then each image slice is converted into a corresponding feature vector. Finally, the feature vector corresponding to each image slice is input into the Transformer model to obtain global features; in the channel of the CNN, the image to be processed is directly used as input, and through multiple convolution and pooling layers, the hierarchical feature representation of the image is gradually extracted, i.e., local features are obtained; after obtaining the global features and the local features, the global features and the local features are fused to obtain the first image features.

[0071] Step S203, based on the first text feature and the first image feature, a target class activation heat map is obtained by adopting a target language model in the target multimodal large model.

[0072] It should be noted that in order to obtain the target class activation heat map, the target language model uses a fully connected layer.

[0073] Assuming that the target multimodal large model is the EarthGPT model, it is important to understand that the EarthGPT model is a remote sensing multimodal large model built on the basis of the LLaMA (Large Language Model Architecture) model. This model can implement multimodal tasks in the remote sensing field such as scene classification, image title generation, region-level title generation, visual question answering, visual localization, and instance segmentation through prompts. However, the EarthGPT model is mainly used for text generation. In order to accurately obtain the target class activation heat map, the large language model (i.e., LLaMA model) in the EarthGPT model can be modified.

[0074] Therefore, in order to clearly illustrate the construction process of the target language model using the fully connected layer, as a possible implementation method, the last classification head (or classification layer) in the original language model can be updated to a fully connected layer to obtain the target language model using the fully connected layer. For example, when the original language model is an LLaMA model, in the original construction of the model, the last layer is the classification head (i.e., the classification head, or classification layer). The classification head can be removed and a d×1 fully connected layer can be added to obtain the target language model using the fully connected layer.

[0075] The image processing method based on user intention in the embodiment of the present application inputs the intention description text into the first text encoder in the target multimodal large model to obtain the first text feature output by the first text encoder; inputs the image to be processed into the visual encoder in the target multimodal large model to obtain the first image feature output by the visual encoder; based on the first text feature and the first image feature, the target class activation heat map is obtained by adopting the target language model in the target multimodal large model. Thus, the target class activation heat map can be effectively obtained through deep learning and computer vision technology to indicate the user's interest in each object in the image to be processed.

[0076] In the case where the target language model includes multiple attention networks, in order to clearly illustrate how, in the above embodiment of the present application, the target class activation heat map corresponding to the image to be processed is obtained based on text features and first image features by adopting the target language model of the fully connected layer, the present application also proposes an image processing method based on user intention.

[0077] Figure 3 It is a flowchart of an image processing method based on user intention proposed in another embodiment of the present application.

[0078] like Figure 3 As shown, based on the above embodiment of the present application, the image processing method based on user intention may further include the following steps:

[0079] Step S301, concatenate the first text feature and the first image feature to obtain a concatenated feature.

[0080] As an example, assume that the first text feature is {t1, t2, …, t q}, where q represents the number of character unit tokens in the intended description text, and the first image feature is {v1,v2,…,v p}, where p represents the number of image tokens after the image to be processed is segmented; the first text feature and the first image feature are concatenated to obtain the concatenated feature X:

[0081] X=concat[v1,v2,…,v p ,t1,t2,……,t q ];(1)

[0082] Among them, p = m×m, where m represents the dimension of each image token.

[0083] Step S302, input the concatenated features into a target language model using a fully connected layer to obtain the similarity between the intent description text and the image to be processed, and extract the target attention weight tensor output by the last attention network among the multiple attention networks.

[0084] Still using the above example for illustration, assume that the target language model using the fully connected layer is the LLaMA-2 model, and the last layer of the classification head in the LLaMA-2 model is replaced with a fully connected layer, wherein the LLaMA-2 model is composed of L (for example, 32) transformer blocks, each of which includes a self-attention network (referred to as an attention network in this application), an RMS Norm (Root Mean Square Normalization) network, and a forward fully connected network. Among them, the attention weight tensor of the self-attention layer represents the degree of association between different tokens (including character unit tokens and image tokens). Assume that the input of the l-th self-attention layer is X l-1 , after three full connection transformations, we get three variables Q, K, and V, and Q, K, and V all belong to R N×(p+q)×d , where N represents the number of multiple attention heads in the self-attention network and d represents the dimension of the latent variable. The attention weight tensor of the l-th self-attention layer can be calculated as follows:

[0085]

[0086] Therefore, in an embodiment of the present application, the concatenated features can be input into a target language model using a fully connected layer, the scalar output by the target language model using a fully connected layer can be determined as the similarity between the intended description text and the image to be processed, and the output of the last attention network among the multiple attention networks in the target language model using a fully connected layer can be determined as the target attention weight tensor.

[0087] Step S303, clip the target attention weight tensor to obtain a cross attention weight tensor.

[0088] It can be understood that the target attention weight tensor may include the cross-attention weight between the first image feature and the first text feature, as well as the attention weights between the image tokens within the first image feature and the attention weights between the character unit tokens within the first text feature. Since the purpose of this application is to extract the area most relevant to the text feature in the image to be processed, what is required in this application is the cross-attention weight between the first image feature and the first text feature.

[0089] Therefore, in this application, the target attention weight tensor can be cropped to obtain the cross attention weight tensor between the first image feature and the first text feature.

[0090] Still using the above example to illustrate, assuming that the target attention weight tensor is AL , then the target attention weight tensor is cropped to obtain the cross attention weight tensor A between the first image feature and the first text feature L′ A L′ :

[0091] A L′ =A L [i,j]; (3)

[0092] Among them, 0<i<p, p<j<p+q, A L′ ∈R N×p×q , A L [i,j] can directly reflect the importance of the image features of the Lth self-attention layer to the text features.

[0093] It should be noted that when the attention network includes N attention heads, the target attention weight tensor A is obtained L It is composed of N attention weight tensor components Combination of which, Furthermore, each attention weight tensor component can be Crop to get the corresponding cross attention weight tensor components in, The cross attention weight tensor A is obtained by combining N cross attention weight tensor components. L′ .

[0094] Step S304: Generate a target class activation heat map corresponding to the image to be processed based on the similarity and the cross-attention weight tensor.

[0095] In an embodiment of the present application, a target class activation heat map corresponding to the image to be processed can be generated based on the similarity and cross-attention weight tensor.

[0096] The image processing method based on user intention of the embodiment of the present application obtains a spliced ​​feature by splicing the first text feature and the first image feature; the spliced ​​feature is input into the target language model using a fully connected layer to obtain the similarity between the intention description text and the image to be processed, and the target attention weight tensor output by the last attention network in multiple attention networks is extracted; the target attention weight tensor is cropped to obtain a cross-attention weight tensor; based on the similarity and the cross-attention weight tensor, a target class activation heat map corresponding to the image to be processed is generated. In this way, it is possible to effectively obtain the target class activation heat map corresponding to the image to be processed based on the first text feature and the first image feature.

[0097] When the last attention network includes N attention heads, correspondingly, the cross-attention weight tensor includes N cross-attention weight tensor components. At this time, in order to clearly illustrate how to generate the target class activation heat map corresponding to the image to be processed based on the similarity and the cross-attention weight tensor in the above embodiment of the present application, the present application also proposes an image processing method based on user intention.

[0098] Figure 4 It is a flowchart of an image processing method based on user intention proposed in another embodiment of the present application.

[0099] like Figure 4 As shown, based on the above embodiment of the present application, the image processing method based on user intention may further include the following steps:

[0100] Step S401, perform gradient calculation on the similarity and any cross-attention weight tensor component to obtain the gradient vector corresponding to the cross-attention weight tensor component.

[0101] Among them, the gradient vector may include the gradient value of the corresponding element in the corresponding cross-attention weight tensor component.

[0102] Step S402, average the gradient vectors corresponding to the N cross-attention weight tensor components to obtain the average gradient vector corresponding to the cross-attention weight tensor.

[0103] The average gradient vector may include the average of the gradient values ​​of the elements at the same position in each cross-attention weight tensor component.

[0104] As an example, suppose N is 2 and the cross attention weight tensor A is L′ The cross attention weight tensor components included are The gradient vectors corresponding to the two cross-attention weight tensor components are in:

[0105]

[0106] Then the gradient vectors corresponding to the above two cross-attention weight tensor components are averaged to obtain the average gradient vector corresponding to the cross-attention weight tensor for:

[0107]

[0108] Step S403, filtering the value of each element in the average gradient vector, and obtaining the target class activation heat map corresponding to the image to be processed based on the filtering result and the cross attention weight tensor.

[0109] As an example, assuming the similarity is S, the kth cross-attention weight tensor component is Among them, k∈[1,N]; for the kth cross-attention weight tensor component, the similarity S and the kth cross-attention weight tensor component are gradient calculated, that is, the gradient value of S for each element in the kth cross-attention weight tensor component is calculated, that is, the gradient vector corresponding to the cross-attention weight tensor component can be obtained Then, the gradient vectors corresponding to the N cross-attention weight tensor components are averaged, that is, the gradient values ​​of the elements at the same position of the N cross-attention weight tensor components are averaged to obtain the average gradient vector corresponding to the cross-attention weight tensor; finally, the values ​​of each element in the average gradient vector are filtered, and based on the filtering result and the cross-attention weight tensor, the target class activation heat map corresponding to the image to be processed can be obtained. For example, the values ​​of each element in the average gradient vector can be filtered according to the following formula:

[0110]

[0111] Among them, E N (·) represents the average of the gradient vectors corresponding to N cross-attention weights (represented as cross-attention weight tensor components in this application), max(·,0) means filtering out negative values ​​in each element and using 0 as the new value of the corresponding element, and W is the filtering result, that is, the average gradient vector after filtering.

[0112] As a possible implementation method, the process of obtaining the target class activation heat map corresponding to the image to be processed based on the filtering result and the cross attention weight tensor may include the following steps:

[0113] Step S4031, multiply the filtering result and the cross attention weight tensor to obtain an activation map of the image to be processed relative to the intent description text.

[0114] Step S4032, average the pixel values ​​of each row in the activation map to obtain a class activation heat map vector.

[0115] Step S4033: Generate a target class activation heat map corresponding to the image to be processed according to the class activation heat map vector.

[0116] Still using the above example for illustration, assume that the filtering result is First, the filtering result and the cross attention weight tensor A are L′ Perform multiplication to obtain the activation map φ of the image to be processed relative to the intended description text:

[0117]

[0118] Among them, φ∈R p×q , q represents the number of character unit tokens in the intended description text, p represents the number of image tokens after the image to be processed is segmented, p = m × m, where m represents the dimension of each image token;

[0119] Secondly, the pixel values ​​of each row in the activation map are averaged, that is, the class activation values ​​of each character unit token are averaged to obtain the class activation heat map vector H of the entire intent description text in the image to be processed:

[0120]

[0121] Among them, H∈R p ;

[0122] Finally, the target class activation heat map corresponding to the image to be processed can be generated according to the class activation heat map vector.

[0123] As an example, the class activation heat map vector can be reshaped into a matrix to obtain a first-class activation heat map; the first-class activation heat map is normalized and upsampled in turn to obtain a second-class activation heat map of the same size as the image to be processed; the second-class activation heat map and the image to be processed are superimposed to obtain a target class activation heat map corresponding to the image to be processed.

[0124] Still using the above example to illustrate, the size (or size) of the class activation heat map vector is p×1. At this time, the class activation heat map vector can be reshaped to obtain a first-class activation heat map with a size of m×m; secondly, the first-class activation heat map is normalized and upsampled to obtain a matrix with the same size as the image to be processed, that is, the second-class activation heat map is obtained; finally, in order to facilitate visual distinction and intuitively reflect the distribution of the user's intended interest level in the image to be processed, the second-class activation heat map and the image to be processed can be superimposed to obtain the target class activation heat map corresponding to the image to be processed.

[0125] The image processing method based on user intention in the embodiment of the present application obtains the gradient vector corresponding to the cross-attention weight tensor component by performing gradient calculation on the similarity and any cross-attention weight tensor component; averages the gradient vectors corresponding to N cross-attention weight tensor components to obtain the average gradient vector corresponding to the cross-attention weight tensor; filters the values ​​of each element in the average gradient vector, and obtains the target class activation heat map corresponding to the image to be processed based on the filtering result and the cross-attention weight tensor. In this way, it is possible to generate the target class activation heat map corresponding to the image to be processed based on the similarity and the cross-attention weight tensor.

[0126] In order to clearly explain how, in any of the above embodiments of the present application, an image segmentor is used to segment the image to be processed based on the intention description text to obtain candidate images corresponding to each object, the present application also proposes an image processing method based on user intention.

[0127] Figure 5 It is a flowchart of an image processing method based on user intention proposed in another embodiment of the present application.

[0128] like Figure 5 As shown, based on any of the above embodiments of the present application, the image processing method based on user intention may further include the following steps:

[0129] Step S501: input the intent description text into the second text encoder in the image segmenter to obtain an embedding vector of the entity noun in the intent description text.

[0130] Among them, entity nouns may include nouns corresponding to each object.

[0131] As an example, assume that the intended description text is "the basketball court under the two tennis courts", where the objects include "tennis court" and "basketball court". The above objects all have corresponding nouns in the intended description text, which are recorded as entity nouns in this application.

[0132] Among them, the second text encoder, for example, can be a CLIP (Contrastive Language-Image Pre-Training) model, a RemoteClip model, etc., and this application does not impose any restrictions on this.

[0133] In an embodiment of the present application, the intent description text may be input into a second text encoder in an image segmentor, and an embedding vector of the entity noun in the intent description text may be obtained based on the output of the second text encoder.

[0134] Still using the above example to illustrate, the intent description text "the basketball court under the two tennis courts" is input into the second text encoder in the image segmenter, and based on the output of the second text encoder, the embedding vector of the entity noun (tennis court, basketball court) in the intent description text is obtained.

[0135] Step S502: Use the image encoder in the image segmentor to perform feature extraction on the image to be processed to obtain a plurality of intermediate image features.

[0136] Among them, the sizes of any two intermediate image features are different.

[0137] As a possible implementation method, when the image encoder in the image segmenter includes multiple feature extraction network layers, the image encoder in the image segmenter is used to perform feature extraction on the processed image, and multiple target feature extraction network layers can be determined from multiple different feature extraction network layers; for any target feature extraction network layer, the image features output by the target feature extraction network layer are upsampled or downsampled to obtain corresponding intermediate image features.

[0138] As an example, assuming that the image encoder in the image segmenter includes 12 feature extraction network layers, when the image encoder in the image segmenter is used to extract features of the image to be processed, the 3rd, 6th, 9th and 12th feature extraction network layers can be determined as target feature extraction network layers; for the 3rd feature extraction network layer, the image features whose size is 1 / 16 of the size of the image to be processed output by the 3rd feature extraction network layer are upsampled to obtain intermediate image features whose size is 1 / 4 of the size of the image to be processed; for the 6th feature extraction network layer, the image features whose size is 1 / 16 of the size of the image to be processed output by the 6th feature extraction network layer are upsampled to obtain intermediate image features whose size is 1 / 4 of the size of the image to be processed; The image features with a size of 1 / 16 of the size of the image to be processed are upsampled to obtain intermediate image features with a size of 1 / 8 of the size of the image to be processed; for the 9th feature extraction network layer, the image features with a size of 1 / 16 of the size of the image to be processed output by the 9th feature extraction network layer are upsampled to obtain intermediate image features with a size of 1 / 16 of the size of the image to be processed; for the 12th feature extraction network layer, the image features with a size of 1 / 16 of the size of the image to be processed output by the 12th feature extraction network layer are upsampled to obtain intermediate image features with a size of 1 / 32 of the size of the image to be processed.

[0139] In this way, multiple intermediate image features can be effectively acquired.

[0140] Step S503, using a reparameterizable visual language path aggregation network RepVL-PAN, cross-modal information fusion is performed on each intermediate image feature and the embedding vector to obtain a second image feature corresponding to each intermediate image feature and a second text feature corresponding to the embedding vector.

[0141] Specifically, each intermediate image feature and embedding vector can be input into RepVL-PAN (Re-parameterizable Vision-Language Path Aggregation Network), and the representation of text and image can be enhanced by cross-modal fusion of intermediate image features and embedding vectors, so as to obtain the second image feature corresponding to each intermediate image feature and the second text feature corresponding to the embedding vector. For example, an image feature pyramid can be established by RepVL-PAN following the top-down and bottom-up paths, and the embedding vector can be fused with the multi-scale features in the image feature pyramid by T-CSPLayer (Text-guided Cross Stage Partial Layers) to obtain the updated second image feature of each intermediate image feature, so as to enhance the text representation of image features, and the image pooling attention (I-Pooling Attention) mechanism can be used to update the embedding vector to obtain the updated second text feature of the embedding vector, so as to enhance the image representation of the embedding vector.

[0142] Step S504: Based on each second image feature and second text feature, a mask decoder in the image segmentor is used to generate a candidate image corresponding to each object.

[0143] As an example, assume that the mask decoder is a pre-trained mask decoder in the SAM model, and the mask decoder includes a Transformer module and a dynamic mask prediction head. After each second image feature and second text feature are input into the above-mentioned mask decoder, the Transformer module weights and transforms the input features through a multi-layer self-attention structure to generate output features containing rich contextual information. The mask prediction head obtains the instance mask corresponding to each object (referred to as a candidate image in this application) based on the output features provided by the Transformer module. It should be noted that when obtaining the candidate image corresponding to each object, the image category to which each candidate image belongs and the corresponding confidence (referred to as a classification confidence score in this application) can also be obtained.

[0144] The image processing method based on user intention in the embodiment of the present application is to input the intention description text into the second text encoder in the image segmentor to obtain the embedding vector of the entity noun in the intention description text; wherein the entity noun includes the noun corresponding to each object; the image encoder in the image segmentor is used to extract the features of the processed image to obtain multiple intermediate image features; wherein the sizes of any two intermediate image features are different; the re-parameterized visual language path aggregation network RepVL-PAN is used to perform cross-modal information fusion on each intermediate image feature and the embedding vector to obtain the second image feature corresponding to each intermediate image feature, and the second text feature corresponding to the embedding vector; based on each second image feature and the second text feature, the mask decoder in the image segmentor is used to generate the candidate image corresponding to each object. Thus, the image to be processed can be segmented based on the intention description text, and the candidate image corresponding to each object can be effectively obtained.

[0145] In the case where any candidate image has a corresponding classification confidence score, in order to clearly illustrate how, in any of the above embodiments of the present application, the target image corresponding to the target object is screened out from each candidate image based on the target class activation heat map, the present application also proposes an image processing method based on user intent.

[0146] Figure 6 It is a flowchart of an image processing method based on user intention proposed in another embodiment of the present application.

[0147] like Figure 6 As shown, based on any of the above embodiments of the present application, the image processing method based on user intention may further include the following steps:

[0148] Step S601: for any candidate image, determine the class activation map score corresponding to any pixel in the candidate image in the target class activation heat map.

[0149] Among them, the class activation map score is determined based on the pixel value of the corresponding pixel in the target class activation heat map.

[0150] In an embodiment of the present application, for any pixel in any candidate image, the pixel value of the pixel in the target class activation heat map can be determined as the class activation map score corresponding to the pixel.

[0151] Step S602: Determine the corresponding target value based on the class activation map score corresponding to each pixel in the candidate image.

[0152] Specifically, the class activation map scores corresponding to each pixel in the candidate image may be averaged, and the average value may be determined as the target value corresponding to the candidate image.

[0153] As an example, assume that the candidate image Mk The pixel set is φ k , the average value U of the class activation map scores corresponding to each pixel in the candidate image can be calculated according to the following formula:

[0154]

[0155] Where n is the candidate image M k The total number of pixels of the candidate image M k The class activation map score corresponding to the pixel with the pixel coordinate (i, j) in ;

[0156] Finally, the average value U can be determined as the candidate image M k The corresponding target value r k ,Right now:

[0157] r k =U; (11)

[0158] It can be understood that since the target class activation heat map indicates the user's interest in each object, that is, it reflects the distribution of the user's interest in each object, the average value of the class activation map score corresponding to each pixel in the candidate image, that is, the target value corresponding to the candidate image, can be used to measure the similarity between the area corresponding to the candidate image and the user's intention.

[0159] Step S603: Update the classification confidence score corresponding to the candidate image according to the target value corresponding to the candidate image to obtain the target confidence score corresponding to the candidate image.

[0160] Still using the above example to illustrate, assume that the candidate image M k The corresponding target value r k , the classification confidence score corresponding to the candidate image is s k , the classification confidence score corresponding to the candidate image can be updated according to the following formula to obtain the target confidence score o corresponding to the candidate image k :

[0161]

[0162] Step S604: determining a target image corresponding to the target object from each candidate image according to the target confidence score corresponding to each candidate image.

[0163] As a possible implementation manner, a maximum value may be determined from the target confidence scores corresponding to the candidate images, and the candidate image corresponding to the maximum value may be determined as the target image corresponding to the target object.

[0164] As another possible implementation method, the target confidence score corresponding to any candidate image can be compared with a set threshold, and based on the comparison result, it is determined whether the candidate image is the target image corresponding to the target object. For example, when the comparison result is that the target confidence score corresponding to the candidate image is greater than the set threshold, the candidate image is determined to be the target image corresponding to the target object; when the comparison result is that the target confidence score corresponding to the candidate image is not greater than the set threshold, it is determined that the candidate image is not the target image corresponding to the target object.

[0165] It should be noted that the threshold value may be pre-set, and the threshold value may be determined based on a large number of test results, and the present application does not impose any restrictions on the value of the threshold value.

[0166] The image processing method based on user intention of the embodiment of the present application determines the class activation map score corresponding to any pixel in the candidate image in the target class activation heat map for any candidate image; wherein the class activation map score is determined based on the pixel value of the corresponding pixel in the target class activation heat map; based on the class activation map score corresponding to each pixel in the candidate image, the corresponding target value is determined; according to the target value corresponding to the candidate image, the classification confidence score corresponding to the candidate image is updated to obtain the target confidence score corresponding to the candidate image; according to the target confidence score corresponding to each candidate image, the target image corresponding to the target object is determined from each candidate image. Thus, in combination with the target class activation heat map, the classification confidence score corresponding to each candidate image is updated, so that the updated target confidence score is associated with the user's interest level, and then the candidate images are screened based on the target confidence score, which can improve the accuracy of obtaining the target image corresponding to the target object of interest to the user.

[0167] In order to clearly illustrate the above-mentioned image processing method based on user intention in the present application, a detailed description is given below with reference to examples.

[0168] As an example, the image processing method based on user intent is applied to remote sensing image processing, such as Figure 7 As shown, the image processing method based on user intention may include the following steps:

[0169] Step S701, using the Grad-CAM method and the features of the last layer of the remote sensing multimodal large language model (referred to as the target multimodal large language model in this application) to calculate the target class activation heat map to obtain the image area of ​​interest to the user.

[0170] It should be noted that the main purpose of generating the target class activation heat map is to align image features with text features through the remote sensing multimodal large language model, so as to mine the salient areas of the input text in the image.

[0171] This application adopts the EarthGPT model as the infrastructure of the remote sensing multimodal large language model. Among them, the EarthGPT model is a remote sensing multimodal large language model built based on the open source large language model LLaMA, which can realize multimodal tasks in the remote sensing field such as scene classification, image title generation, regional title generation, visual question answering, visual positioning, and instance segmentation through the prompt Prompt. Since the EarthGPT model is mainly used for text generation, and this application hopes to use it to measure the similarity between the input image and the input text, this application has made a simple modification to the EarthGPT model, removing the last classification head of the large language model in the EarthGPT model and adding a fully connected layer to obtain a scalar representing the similarity between the input image and the input text. In order to train the last fully connected layer, this application freezes all other networks of the EarthGPT model and trains the last fully connected layer using the remote sensing image-text constructed in the RemoteCLIP model.

[0172] The EarthGPT model includes a text encoder (referred to as the first text encoder in this application), a visual encoder, and a large language model (referred to as the target language model in this application), where:

[0173] 1. Similar to most natural language processing models, the text encoder in the EarthGPT model first uses the text Tokenzier tool to divide the user description information (referred to as intent description text in this application) into a discrete sequence of character unit Tokens, and then uses the Embedding method to continue mapping each character unit Token into a real number vector to obtain the first text feature.

[0174] 2. The visual encoder in the EarthGPT model uses two parallel ViT models and CNN models as the backbone network, and extracts features from the remote sensing image (referred to as the image to be processed in this application) at the same time, and connects the two types of features extracted to obtain the final first image feature of the remote sensing image. Among them, the global information of the remote sensing image can be modeled through the ViT model, and the local detail information of the remote sensing image can be described through the CNN model.

[0175] Specifically, one channel of the visual encoder divides the remote sensing image of size h×w into m×m image slices and inputs each image slice into the ViT model. The other channel inputs the remote sensing image into the CNN model. Through the feature extraction of the above two networks, the feature representation of m×m image tokens is finally obtained, that is, the final first image feature of the remote sensing image is obtained.

[0176] 3. The large language model in the EarthGPT model is the LLaMA-2 model. By fine-tuning some parameters and instructions on the LAION-400M (Large-scale Artificial Intelligence Open Network-400M) dataset and the MMRS-1M dataset, the LLaMA-2 model has basic visual understanding capabilities and achieves visual language alignment.

[0177] In order to enable the EarthGPT model to generate the target class activation heat map, this application can obtain the gradient of the last attention map of LLaMA-2 by using the Grad-CAM model, assuming that the first image feature obtained by the visual encoder is {v1,v2,…,v p}, where p represents the number of image tokens, and the first text feature obtained by the text encoder is {t1, t2, …, t q}, where q represents the number of character unit Tokens; as shown in formula (1), by connecting the first image feature and the first text feature, the input of the large language model (i.e., the LLaMA-2 model), i.e., the concatenated feature, is obtained.

[0178] It should be understood that the LLaMA-2 model consists of L Transformer Blocks, each of which includes a self-attention network (referred to as an attention network in this application), an RMS Norm network, and a forward fully connected network. The attention weight tensor A of the self-attention network represents the degree of association between different tokens (including character unit tokens and image tokens). Assume that the input of the lth self-attention network is X l-1 , after three full connection transformations, we get three variables Q, K, and V, and Q, K, and V all belong to R N×(p+q)×d , where N represents the number of multiple attention heads in the self-attention network, d represents the dimension of the latent variable, and the attention weight tensor of the l-th self-attention layer can be obtained by formula (2), where l∈[1,L].

[0179] Obviously, the target attention weight tensor A output by the Lth self-attention network is LIt not only includes the cross attention weight between the first image feature and the first text feature, but also includes the attention weight between each image token within the first image feature and the attention weight between each character unit token within the first text feature. Since the purpose of this application is to extract the area image most relevant to the first text feature corresponding to the user description information from the remote sensing image, that is, to extract the area image that the user is most interested in, this application only needs to use the cross attention weight between the first image feature and the first text feature. Specifically, the target attention weight tensor A can be L After cropping, according to formula (3), the cross attention weight tensor between the first image feature and the first text feature is A L′ .

[0180] Among them, the cross attention weight tensor is A L′ It can directly reflect the importance of the image features output by the Lth self-attention network to the text features. Therefore, the class activation heat map can be visualized according to the attention weight to locate the key areas of the user's attention in the remote sensing image, or the image areas of interest to the user.

[0181] In this application, the Grad-CAM method is used to calculate the cross attention weight tensor A output by the last self-attention network of the LLaMA-2 model. L′ Gradient calculation is performed to obtain the degree of association between each image token and the user description information, and then these association degrees are mapped back to the spatial position of the original image to form a heat map. Specifically, after the remote sensing image and user description information are input into the trained EarthGPT model, the EarthGPT model outputs the similarity scalar S (denoted as similarity in this application) between the remote sensing image and the user description information; secondly, the similarity scalar S is calculated for the cross-attention weight tensor A. L The gradient of each element in And take the gradient average of the multiple attention heads and add the gradient average to the cross attention weight tensor A L By multiplying, we can obtain the activation map φ of each image token of the remote sensing image relative to the character unit token of each user description information according to formula (8);

[0182] Furthermore, by averaging the class activation values ​​of all character unit tokens, that is, averaging the pixel values ​​of each row in the activation map, the class activation heat map vector H of the entire user description information in the remote sensing image is obtained according to formula (9);

[0183] Finally, the class activation heatmap vector is transformed into a first-class activation heatmap of size m×m; the first-class activation heatmap is normalized and upsampled in turn to obtain a matrix of the same size as the original remote sensing image, that is, the second-class activation heatmap is obtained, which represents the user's intention to pay attention to different areas of the image; in order to facilitate visual distinction and intuitively reflect the distribution of the user's interest level in the image to be processed, the second-class activation heatmap and the remote sensing image can be superimposed to obtain the target class activation heatmap corresponding to the remote sensing image.

[0184] As an example, Figure 8 The remote sensing image 81 to be processed and the user description information 82 “a basketball court under two tennis courts” are displayed. Based on the remote sensing image 81 and the user description information 82, the EarthGPT model described above is used to finally obtain the target class activation heat map 83.

[0185] Step S702, using an open vocabulary domain instance segmenter (referred to as an image segmenter in this application), takes the remote sensing image and user description information as input, implements instance segmentation guided by the user description information, and obtains a candidate instance mask map corresponding to each instance (referred to as a candidate image in this application).

[0186] It should be noted that although weak supervision information such as class activation heat map can also be used to extract objects from remote sensing images, it is likely to cause a lot of information about the object to be missed. For example, assuming that there is a dog in the remote sensing image, for the extraction of a dog, since the class activation heat map pays more attention to the position of the dog's head, the area extracted using the class activation heat map will lose the dog's body parts, making the segmentation result very unsatisfactory. With the rapid development of multimodal large language model technology, great progress has been made in using basic multimodal models to achieve instance segmentation of open vocabulary domains. Therefore, this application selects an open vocabulary domain instance segmentation method, which extracts possible object instances in the image by taking remote sensing images and user description information as input, thereby constructing a candidate set for image instance segmentation.

[0187] This application uses the SAM model as an open vocabulary domain instance segmentation model. The SAM model is a general image segmentation model that generates segmentation results based on given prompt words (such as foreground / background points, bounding boxes, or masks), and segments arbitrary objects without pre-defining image categories through visual-language modeling and pre-training on large-scale datasets.

[0188] The architecture of the SAM model mainly includes three core components: image encoder, cue word encoder and mask decoder. The SAM model uses the ViT model pre-trained by MAE (Masked Autoencoders) as the image encoder to extract image features; the cue word encoder is used to encode various types of cue words into cue word features; the mask decoder uses the cross-attention mechanism to learn the correlation between image features and cue word features, and finally outputs the segmentation mask map of the object of interest to the user.

[0189] It should be noted that the original SAM model can output segmentation results based on multiple prompt words. Since the present application takes user intent into consideration, when only user description information is used as user intent prompt words, the prompt word encoder in the present application is a sparse text encoder.

[0190] It should also be noted that according to a large number of studies, the segmentation ability of the SAM model in the field of remote sensing images is not significant, which may be because the SAM model is designed for the segmentation of natural images. In order to apply the SAM model to remote sensing image scenes, this application partially transforms and fine-tunes the original SAM model.

[0191] 1. At the model level, the original SAM model uses the pre-trained CLIP model as a text encoder, while in this application, the RemoteClip model is used to replace the CLIP model and used as a text encoder. The RemoteClip model uses a large-scale remote sensing paired image-text dataset to fine-tune the pre-trained CLIP model, which can better capture the visual-semantic features in remote sensing scenes. Given the input user description information T, the RemoteClip model is used as a text encoder TextEncoder(·), and the embedding vector corresponding to the entity noun in the user description information can be obtained.

[0192] 2. For the image encoder, the original SAM model uses the ViT model pre-trained by MAE. However, the ViT model only outputs image features of one scale, which is difficult to meet the problem of large scale changes of objects in remote sensing image scenes. Therefore, in order to obtain image features of multiple different scales, this application uses the ViTAE transformer pre-trained on the large-scale remote sensing dataset MillionAID as the image encoder, wherein the ViTAE transformer includes 12 feature extraction network layers, and the size of the image features output by each layer is 1 / 16 of the input image. By upsampling or downsampling the image features of the 3rd, 6th, 9th and 12th layers respectively, 4 scales of features of 1 / 4, 1 / 8, 1 / 16 and 1 / 32 are obtained. Then, through the RepVL-PAN model, a multi-scale image feature pyramid is constructed following the top-down and bottom-up paths, and the multi-scale image features of the remote sensing image and the embedded vector of the user description information are used as the input of the RepVL-PAN model. The representation of text and image is enhanced by cross-modal fusion of image features and embedded vectors, and the updated multi-scale image features and updated embedded vectors are output.

[0193] 3. For the mask decoder, this application still uses the pre-trained mask decoder in the SAM model, wherein the mask decoder includes a Transformer module and a dynamic mask prediction head, wherein the Transformer module is used to associate multimodal features, and the mask prediction head is used to generate a detailed segmentation mask map.

[0194] After the open vocabulary domain instance segmenter is constructed using the above method, in order to apply the constructed open vocabulary domain instance segmenter to the remote sensing image instance segmentation scenario, this application will fine-tune the constructed open vocabulary domain instance segmentor based on the remote sensing image visual positioning dataset DIOR-RSVG. Among them, the DIOR-RSVG dataset is a visual positioning benchmark dataset constructed using the existing DIOR dataset as basic data. Since DIOR-RSVG only contains the rectangular bounding box of the object and does not contain clear segmentation edges, the DIOR-RSVG dataset needs to be segmented. Specifically, by using the rectangular bounding box as a prompt word and using the SAM model to segment the image, a multimodal instance segmentation dataset is constructed.

[0195] Finally, by inputting the remote sensing image and user description text into the constructed open vocabulary domain instance segmentor, a set of candidate instance mask maps (referred to as candidate images in this application) are obtained, as well as the image category corresponding to each candidate instance mask map and the corresponding classification confidence score.

[0196] As an example, Fig. 9 The remote sensing image 91 and the user description information 91 to be processed are shown. Based on the remote sensing image 91 and the user description information 92, the above-constructed open vocabulary domain instance segmenter is used to obtain the instance segmentation result as shown in FIG. Fig. 9 As shown in Figure 91.

[0197] Step S703 , screening the segmentation results based on the target class activation heat map and the candidate instance mask map obtained by the open vocabulary domain instance segmentor.

[0198] After obtaining the target class activation heat map and a set of candidate instance mask maps, it is necessary to determine the candidate instance mask map corresponding to the entity that the user is most interested in from the set of candidate instance mask maps based on the target class activation heat map. In this application, the entity that the user is most interested in is recorded as the target object. Generally speaking, an image may contain many objects, including a lot of noise. Therefore, it is difficult to determine the entity that the user is most interested in based on the user description information. In this application, the target class activation heat map can be used to filter the obtained set of candidate instance mask maps, wherein the target class activation heat map of the remote sensing image, the candidate instance mask map generated by the open vocabulary domain instance segmentor, and the corresponding image category and classification confidence score are used as input, and the process of screening the candidate instance mask map includes the following steps:

[0199] 1. Calculate the mean score of the class activation map. The classification confidence only reflects the similarity between the object segmentation mask map and the corresponding category of the specific input text, and cannot fully reflect the degree of association with the entire user intent. This application proposes to use the average value of the class activation map scores corresponding to all pixels in the candidate instance mask map as a measure of the association between the candidate instance mask map and the user intent (referred to as the target value in this application). Specifically, for the candidate instance mask map M k , calculate the average class activation map scores of all pixels in the candidate instance mask map according to formulas (10) and (11), and determine the average value as the target value r k ;

[0200] 2. Recalculate the confidence score corresponding to the candidate instance mask map. The final score of the candidate instance mask map (referred to as the target confidence score in this application) must consider both the correlation between the image area and the specific category and the correlation between the image area and the user description information. Therefore, the inventor of this application uses formula (12) to define the final score of each candidate instance mask map.

[0201] 3. Filter the candidate instance mask maps to obtain the candidate instance mask maps corresponding to the entities that the user is most interested in.

[0202] Since the open vocabulary domain instance segmenter can output redundant object segmentation masks (i.e., candidate instance mask maps) based on user description information, for example, Fig. 9 As shown, the open vocabulary domain instance segmenter outputs a candidate instance mask map that is not corresponding to the "tennis court" that the user is interested in. The candidate instance mask map corresponding to the "tennis court" is a redundant object segmentation mask. Therefore, this application directly uses the confidence score of the candidate instance mask map to filter all candidate instance mask maps. Specifically, a set threshold can be set, and the target confidence score corresponding to any candidate instance mask map can be compared with the set threshold. Based on the comparison result, it is determined whether the candidate instance mask map is the candidate instance mask map corresponding to the entity that the user is most interested in.

[0203] In summary, according to step S701 to step S703, open environment instance segmentation of remote sensing images can be achieved in the case of zero samples based on the actual needs of the user.

[0204] At present, the instance segmentation method of remote sensing images has the problem that a single model cannot cover multiple remote sensing application scenarios, and all of them are segmented for predefined object categories, which cannot be applied to new scenarios and object categories. There is also a problem of lack of labeled data. However, this application extracts the objects that the user needs to segment through user intention, and uses the remote sensing multimodal large language model to generate the class activation heat map of the image; at the same time, based on the user intention text data and remote sensing images, the open vocabulary domain instance segmentation method is used to generate candidate instance mask maps for remote sensing images; finally, the class activation heat map of the remote sensing image and the candidate instance mask map are combined to obtain the instance mask map corresponding to the object of interest to the user.

[0205] The method of the present application can satisfy instance segmentation of any scene and any object type according to user intention, and has wide applicability. At the same time, it can achieve instance segmentation under zero-sample conditions by utilizing the powerful capabilities of the current multimodal large language model and open vocabulary domain instance segmentation method.

[0206] Fig.10 It is a structural diagram of an image processing device based on user intention proposed in another embodiment of the present application.

[0207] like Fig.10 As shown, the image processing device 1000 based on user intention includes:

[0208] The acquisition module 1010 is used to acquire the image to be processed and the user's intention description text for the image to be processed; wherein the intention description text describes at least one object of the image to be processed and indicates the target object that the user is interested in among the at least one object.

[0209] The generation module 1020 is used to generate a target class activation heat map corresponding to the image to be processed based on the intention description text and the image to be processed, using a target multimodal large model; wherein the target class activation heat map is used to indicate the user's interest level in each of the objects.

[0210] The segmentation module 1030 is used to segment the image to be processed by using an image segmentor based on the intention description text to obtain candidate images corresponding to each of the objects.

[0211] The screening module 1040 is used to screen out a target image corresponding to the target object from each of the candidate images based on the target class activation heat map.

[0212] In a possible implementation of an embodiment of the present application, the target multimodal large model includes a first text encoder, a visual encoder and a target language model, wherein the target language model adopts a fully connected layer; the generation module 1020 is used to: input the intent description text into the first text encoder to obtain a first text feature output by the first text encoder; input the image to be processed into the visual encoder to obtain a first image feature output by the visual encoder; based on the first text feature and the first image feature, obtain the target class activation heat map corresponding to the image to be processed by adopting the target language model of the fully connected layer.

[0213] In a possible implementation of an embodiment of the present application, the target language model includes multiple attention networks; the generation module 1020 is used to: splice the first text feature and the first image feature to obtain a spliced ​​feature; input the spliced ​​feature into the target language model using a fully connected layer to obtain the similarity between the intent description text and the image to be processed, and extract the target attention weight tensor output by the last attention network in the multiple attention networks; crop the target attention weight tensor to obtain a cross-attention weight tensor; based on the similarity and the cross-attention weight tensor, generate a target class activation heat map corresponding to the image to be processed.

[0214] In a possible implementation of an embodiment of the present application, the last attention network includes N attention heads, and correspondingly, the cross-attention weight tensor includes N cross-attention weight tensor components; the generation module 1020 is used to: perform gradient calculation on the similarity and any of the cross-attention weight tensor components to obtain a gradient vector corresponding to the cross-attention weight tensor component; average the gradient vectors corresponding to the N cross-attention weight tensor components to obtain an average gradient vector corresponding to the cross-attention weight tensor; filter the values ​​of each element in the average gradient vector, and based on the filtering result and the cross-attention weight tensor, obtain a target class activation heat map corresponding to the image to be processed.

[0215] In a possible implementation of an embodiment of the present application, the generation module 1020 is used to: multiply the filtering result by the cross-attention weight tensor to obtain an activation map of the image to be processed relative to the intent description text; average the pixel values ​​of each row in the activation map to obtain a class activation heat map vector; and generate a target class activation heat map corresponding to the image to be processed based on the class activation heat map vector.

[0216] In a possible implementation of an embodiment of the present application, the generation module 1020 is used to: perform matrix reshaping on the class activation heat map vector to obtain a first class activation heat map; normalize and upsample the first class activation heat map in sequence to obtain a second class activation heat map having the same size as the image to be processed; and superimpose the second class activation heat map and the image to be processed to obtain a target class activation heat map corresponding to the image to be processed.

[0217] In a possible implementation of the embodiment of the present application, the image processing device 1000 based on user intention may further include:

[0218] A construction module is used to update the last classification head in the original language model to a fully connected layer to obtain the target language model using the fully connected layer.

[0219] In a possible implementation of an embodiment of the present application, the image segmenter includes a second text encoder, an image encoder and a mask decoder; the segmentation module 1030 is used to: input the intent description text into the second text encoder to obtain an embedding vector of the entity noun in the intent description text; wherein the entity noun includes a noun corresponding to each of the objects; use the image encoder to extract features of the image to be processed to obtain multiple intermediate image features; wherein the sizes of any two of the intermediate image features are different; use a reparameterizable visual language path aggregation network RepVL-PAN to perform cross-modal information fusion on each of the intermediate image features and the embedding vector to obtain a second image feature corresponding to each of the intermediate image features, and a second text feature corresponding to the embedding vector; based on each of the second image features and the second text features, use the mask decoder to generate a candidate image corresponding to each of the objects.

[0220] In a possible implementation of an embodiment of the present application, any of the candidate images has a corresponding classification confidence score; the screening module 1040 is used to: for any of the candidate images, determine the class activation map score corresponding to any pixel in the candidate image in the target class activation heat map; wherein the class activation map score is determined based on the pixel value of the corresponding pixel in the target class activation heat map; based on the class activation map score corresponding to each pixel in the candidate image, determine the corresponding target value; according to the target value corresponding to the candidate image, update the classification confidence score corresponding to the candidate image to obtain the target confidence score corresponding to the candidate image; according to the target confidence score corresponding to each of the candidate images, determine the target image corresponding to the target object from each of the candidate images.

[0221] The image processing device based on user intention of the embodiment of the present application obtains the image to be processed and the user's intention description text for the image to be processed; wherein the intention description text describes at least one object of the image to be processed and indicates the target object of interest to the user in at least one object; based on the intention description text and the image to be processed, a target multimodal large model is used to generate a target class activation heat map corresponding to the image to be processed; wherein the target class activation heat map is used to indicate the user's interest in each object; based on the intention description text, an image segmenter is used to segment the image to be processed to obtain candidate images corresponding to each object; based on the target class activation heat map, the target image corresponding to the target object is screened from each candidate image. Thus, the candidate images obtained by image segmentation can be screened with reference to the target class activation heat map, the objects of interest to the user in the image can be effectively identified, and the operability and accuracy of the images of the objects of interest to the user can be improved, the image processing effect based on user intention can be improved, and the segmentation of unseen targets in image instance segmentation can be realized, solving the problem that one model cannot be applied to multiple scenes.

[0222] Fig.11 A block diagram of an exemplary electronic device suitable for implementing the embodiments of the present application is shown. Fig.11 The electronic device 12 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0223] like Fig.11 As shown, the electronic device 12 is in the form of a general purpose computing device. The components of the electronic device 12 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 that connects various system components (including the system memory 28 and the processing unit 16).

[0224] The bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor or a local bus using any of a variety of bus structures. For example, these architectures include but are not limited to Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus and Peripheral Component Interconnection (PCI) bus.

[0225] The electronic device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device 12, including volatile and non-volatile media, removable and non-removable media.

[0226] The memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The electronic device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be used to read and write non-removable, non-volatile magnetic media ( Fig.11 Not shown, often called a "hard drive").

[0227] although Fig.11 Not shown, a disk drive for reading and writing a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing a removable non-volatile optical disk (e.g., a compact disc read only memory (CD-ROM), a digital versatile disc read only memory (DVD-ROM), or other optical media) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present application.

[0228] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in the memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules 42 generally perform the functions and / or methods of the embodiments described herein.

[0229] The electronic device 12 may also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.), may also communicate with one or more devices that enable the human body to interact with the electronic device 12, and / or communicate with any device that enables the electronic device 12 to communicate with one or more other computing devices (e.g., network card, modem, etc.). Such communication may be performed through an input / output (I / O) interface 22. In addition, the electronic device 12 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 20. As shown, the network adapter 20 communicates with other modules of the electronic device 12 through a bus 18. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0230] The processing unit 16 executes various functional applications and data processing by running programs stored in the system memory 28, such as implementing the image processing method based on user intention mentioned in the above embodiment.

[0231] In order to implement the above embodiments, the present application also proposes a non-temporary computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the image processing method based on user intention proposed in the above embodiments of the present application is implemented.

[0232] In order to implement the above embodiments, the present application also proposes a computer program product. When the instruction processor in the computer program product is executed, the image processing method based on user intention proposed in the above embodiments of the present application is executed.

[0233] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the inventions claimed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary techniques in the art that are not claimed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0234] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

[0235] It should be noted that, in the description of this application, the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of this application, unless otherwise specified, the meaning of "plurality" is two or more.

[0236] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0237] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0238] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0239] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0240] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0241] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0242] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in the field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. An image processing method based on user intention, characterized in that: The method comprises: Acquire an image to be processed and a user's intention description text for the image to be processed; wherein the intention description text describes at least one object of the image to be processed and indicates a target object of interest to the user in the at least one object; Based on the intention description text and the image to be processed, a target multimodal large model is used to generate a target class activation heat map corresponding to the image to be processed; wherein the target class activation heat map is used to indicate the user's interest in each of the objects; Based on the intention description text, segment the image to be processed using an image segmentor to obtain candidate images corresponding to each of the objects; Based on the target class activation heat map, a target image corresponding to the target object is screened out from each of the candidate images.

2. The method according to claim 1, characterized in that The target multimodal large model includes a first text encoder, a visual encoder and a target language model, wherein the target language model adopts a fully connected layer; The method of generating a target class activation heat map corresponding to the image to be processed by using a target multimodal large model based on the intention description text and the image to be processed includes: Inputting the intention description text into the first text encoder to obtain a first text feature output by the first text encoder; Inputting the image to be processed into the visual encoder to obtain a first image feature output by the visual encoder; Based on the first text feature and the first image feature, a target class activation heat map corresponding to the image to be processed is obtained by adopting the target language model of the fully connected layer.

3. The method according to claim 2, characterized in that The target language model includes multiple attention networks; The step of obtaining a target class activation heat map corresponding to the image to be processed by using the target language model of the fully connected layer based on the first text feature and the first image feature includes: splicing the first text feature and the first image feature to obtain a spliced ​​feature; Inputting the concatenated features into a target language model using a fully connected layer, obtaining the similarity between the intent description text and the image to be processed, and extracting a target attention weight tensor output by a last attention network among the multiple attention networks; Clipping the target attention weight tensor to obtain a cross attention weight tensor; Based on the similarity and the cross-attention weight tensor, a target class activation heat map corresponding to the image to be processed is generated.

4. The method according to claim 3, characterized in that: The last attention network includes N attention heads, and correspondingly, the cross-attention weight tensor includes N cross-attention weight tensor components; The generating, based on the similarity and the cross attention weight tensor, a target class activation heat map corresponding to the image to be processed, comprises: Performing gradient calculation on the similarity and any of the cross-attention weight tensor components to obtain a gradient vector corresponding to the cross-attention weight tensor component; averaging the gradient vectors corresponding to the N cross-attention weight tensor components to obtain an average gradient vector corresponding to the cross-attention weight tensor; The value of each element in the average gradient vector is filtered, and based on the filtering result and the cross-attention weight tensor, a target class activation heat map corresponding to the image to be processed is obtained.

5. The method according to claim 4, characterized in that The step of obtaining a class activation heat map corresponding to the image to be processed based on the filtering result and the cross attention weight tensor includes: Performing a multiplication operation on the filtering result and the cross attention weight tensor to obtain an activation map of the image to be processed relative to the intent description text; Averaging the pixel values ​​of each row in the activation map to obtain a class activation heat map vector; Generate a target class activation heat map corresponding to the image to be processed according to the class activation heat map vector.

6. The method according to claim 5, characterized in that The step of generating a target class activation heat map corresponding to the image to be processed according to the class activation heat map vector includes: Reshaping the class activation heat map vector to obtain a first class activation heat map; Normalizing and upsampling the first type of activation heat map in sequence to obtain a second type of activation heat map having the same size as the image to be processed; The second type activation heat map and the image to be processed are superimposed to obtain a target type activation heat map corresponding to the image to be processed.

7. The method according to claim 2, characterized in that: The process of constructing the target language model using the fully connected layer includes: The last classification head in the original language model is updated to a fully connected layer to obtain the target language model using the fully connected layer.

8. The method according to claim 1, characterized in that: The image segmenter includes a second text encoder, an image encoder and a mask decoder; The step of segmenting the image to be processed based on the intention description text using an image segmenter to obtain candidate images corresponding to each of the objects includes: Inputting the intention description text into the second text encoder to obtain an embedding vector of entity nouns in the intention description text; wherein the entity nouns include nouns corresponding to each of the objects; Using the image encoder to extract features from the image to be processed to obtain a plurality of intermediate image features; wherein the sizes of any two of the intermediate image features are different; Using a reparameterizable visual language path aggregation network RepVL-PAN, cross-modal information fusion is performed on each of the intermediate image features and the embedding vector to obtain a second image feature corresponding to each of the intermediate image features and a second text feature corresponding to the embedding vector; Based on each of the second image features and the second text features, the mask decoder is used to generate a candidate image corresponding to each of the objects.

9. The method according to any one of claims 1 to 8, characterized in that: Any of the candidate images has a corresponding classification confidence score; The step of selecting a target image corresponding to the target object from each of the candidate images based on the target class activation heat map includes: For any of the candidate images, determining a class activation map score corresponding to any pixel in the candidate image in the target class activation heat map; wherein the class activation map score is determined based on a pixel value of the corresponding pixel in the target class activation heat map; Determining a corresponding target value based on a class activation map score corresponding to each pixel in the candidate image; According to the target value corresponding to the candidate image, the classification confidence score corresponding to the candidate image is updated to obtain the target confidence score corresponding to the candidate image; According to the target confidence scores corresponding to the candidate images, a target image corresponding to the target object is determined from the candidate images.

10. An image processing device based on user intention, characterized in that: The device comprises: An acquisition module, used to acquire an image to be processed and a user's intention description text for the image to be processed; wherein the intention description text describes at least one object of the image to be processed and indicates a target object of interest to the user in the at least one object; A generation module, used to generate a target class activation heat map corresponding to the image to be processed based on the intention description text and the image to be processed, using a target multimodal large model; wherein the target class activation heat map is used to indicate the user's interest in each of the objects; A segmentation module, configured to segment the image to be processed using an image segmentor based on the intention description text to obtain candidate images corresponding to each of the objects; A screening module is used to screen out a target image corresponding to the target object from each of the candidate images based on the target class activation heat map.

11. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 9.

12. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: in, The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 9.

13. A computer program product, characterized in that The invention comprises a computer program which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 9.

Images