Image understanding method, device, equipment and storage medium

Abstract

The present application discloses an image understanding method, apparatus, device and storage medium, the method comprising obtaining an image to be understood and a task token; obtaining multi-scale image features, multi-scale image coding features and target area location information of the image to be understood; determining local fine-grained image features based on the multi-scale image features and the target area location information, and determining semantic features based on the multi-scale image coding features and the target area location information; and determining a task processing result for the image to be understood based on the semantic features, the local fine-grained image features and the task token. The present application improves the accuracy of understanding the local area image by using the target area location information as area indication information to focus on local understanding of the local area image corresponding to the target area location information, thereby improving the accuracy of the task processing result.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an image understanding method, apparatus, device and storage medium. Background Art

[0002] Artificial intelligence is widely used in the field of open-world visual understanding, using language models to understand the entire image of unseen images, such as Qwen2-VL and InterNumberVL. Existing open-world visual understanding models generally focus on understanding the entire image. Because these models focus on understanding the entire image, they are unable to accurately collect the visual features and contextual information of local images within the overall image (especially complex image regions). This can lead to deviations in the understanding of these local regions, and in turn, unsatisfactory results in task processing based on local image information.

[0003] Therefore existing technology still needs to be improved and improved. Summary of the Invention

[0004] The technical problem to be solved by this application is to provide an image understanding method, device, equipment and storage medium to address the shortcomings of the existing technology.

[0005] In order to solve the above technical problems, the first aspect of the present application provides an image understanding method, wherein the image understanding method specifically includes:

[0006] Get the image to be understood and the task token;

[0007] Acquire multi-scale image features, multi-scale image coding features, and target area position information of the image to be understood;

[0008] Determining local fine-grained image features based on the multi-scale image features and the target region position information, and determining semantic features based on the multi-scale image coding features and the target region position information;

[0009] A task processing result of the image to be understood is determined based on the semantic features, the local fine-grained image features, and the task token.

[0010] The image understanding method, wherein the multi-scale image features and the multi-scale image coding features of the image to be understood are obtained through a preset target detection model, the target detection model includes a backbone network and an encoder, and the acquisition process specifically includes:

[0011] Inputting the image to be understood into the backbone network, and outputting multi-scale image features through the backbone network;

[0012] The multi-scale image features are input into the encoder, and the encoder outputs the multi-scale image coding features.

[0013] The image understanding method, wherein the target detection model also includes a decoder, and the target area position information is output by the decoder based on the multi-scale image coding features.

[0014] The image understanding method, wherein the target area location information is determined based on the annotation box carried in the image to be understood; or, is determined based on interactive information input by the user, wherein the interactive information includes information on the area selected by the user in the image to be understood or text prompt information input by the user.

[0015] In the image understanding method, the task token is used to reflect the understanding task performed on the image to be understood, and when the understanding task is a visual question-answering task, the task token carries the question text.

[0016] In the image understanding method, the question text is proposed for the image area corresponding to the target area position information, and the target area position information is determined based on the question text.

[0017] In the image understanding method, the image area corresponding to the target area position information includes one or more objects to be understood.

[0018] The image understanding method, wherein determining local fine-grained image features based on the multi-scale image features and the target area position information specifically includes:

[0019] Inputting the target area position information and the multi-scale image features into a local fine-grained feature extractor, and selecting a target image feature from the multi-scale image features by the local fine-grained feature extractor;

[0020] The local fine-grained feature extractor extracts the local image features corresponding to the target area position information from the target image features to obtain local fine-grained image features.

[0021] In the image understanding method, when selecting a target image feature from the multi-scale image features, the feature is selected based on the area or size ratio of the local area image corresponding to the target area position information.

[0022] The image understanding method, wherein determining the semantic features based on the multi-scale image coding features and the target area position information specifically includes:

[0023] Initializing a content coding vector and obtaining a position coding vector corresponding to the target area position information;

[0024] A semantic feature is determined based on the content coding vector, the position coding vector, and the multi-scale image coding feature.

[0025] In the image understanding method, determining the semantic feature based on the content coding vector, the position coding vector, and the multi-scale image coding feature specifically includes:

[0026] The content coding vector, the position coding vector, and the multi-scale image coding feature are input into a semantic feature encoder, and the semantic feature encoder outputs the semantic feature. The semantic feature encoder includes several coding layers, wherein the input items of each coding layer include the updated content coding vector output by the previous coding layer, the position coding vector, and the multi-scale image coding feature, and the output items of each coding layer include the updated content coding vector. The content coding vector included in the input item of the first coding layer is a randomly initialized content coding vector, and the updated content coding vector output by the last coding layer is used as the semantic feature.

[0027] In the image understanding method, wherein the encoding layer includes a self-attention module and a deformable cross-attention module, the process of determining the updated content encoding vector specifically includes:

[0028] Constructing a first query vector, a first value vector, and a first key vector based on the updated content encoding vector output by the previous encoding layer;

[0029] concatenating the first query vector and the position encoding vector to obtain a second query vector, and concatenating the first key vector and the position encoding vector to obtain a second key vector;

[0030] Outputting a third query vector through the self-attention module based on the second query vector, the first value vector, and the second key vector;

[0031] concatenating the third query vector and the position encoding vector to obtain a fourth query vector, and constructing a second value vector based on the multi-scale image encoding feature;

[0032] Based on the fourth query vector and the second value vector, an updated content encoding vector is output through the deformable cross attention module.

[0033] The image understanding method, wherein determining the task processing result of the image to be understood based on the semantic features, the local fine-grained image features, and the task token specifically includes:

[0034] aligning feature dimensions of the semantic features and the local fine-grained image features with feature dimensions of a task token;

[0035] The aligned semantic features, the aligned local fine-grained image features and the task token are input into a text decoder, and the task processing result is output through the text decoder.

[0036] A second aspect of the present application provides an image understanding device, wherein the image understanding device specifically includes:

[0037] Content acquisition module, used to obtain the image to be understood and the task token;

[0038] An image feature acquisition module, configured to acquire multi-scale image features, multi-scale image coding features, and target area location information of the image to be understood;

[0039] a semantic feature determination module, configured to determine local fine-grained image features based on the multi-scale image features and the target region position information, and to determine semantic features based on the multi-scale image coding features and the target region position information;

[0040] A processing module is used to determine a task processing result of the image to be understood based on the semantic features, the local fine-grained image features and the task token.

[0041] A third aspect of the present application provides a computer-readable storage medium, which stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in any of the image understanding methods described above.

[0042] A fourth aspect of the present application provides a terminal device, comprising: a processor and a memory;

[0043] The memory stores a computer-readable program executable by the processor;

[0044] When the processor executes the computer-readable program, the steps in any one of the above-described image understanding methods are implemented.

[0045] Beneficial effects:

[0046] 1. This application uses the target area location information as area indication information to focus on in-depth understanding of the local area image corresponding to the target area location information, thereby improving the accuracy of understanding the local area image.

[0047] 2. This application simultaneously extracts local fine-grained image features and semantic features of local area images, provides shallow information of local area images through local fine-grained image features, provides feature representation for semantic understanding through semantic features, and then determines the task processing results based on local fine-grained image features and semantic features, which can ensure the accuracy of the task processing results.

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A flowchart of the image understanding method provided in an embodiment of the present application.

[0050] Figure 2 A flowchart illustrating a specific example of the image understanding method provided in the embodiments of the present application.

[0051] Figure 3 This is a flowchart of the coding layer principle provided in the embodiment of the present application.

[0052] Figure 4 This is a block diagram of the principles of the image understanding device provided in an embodiment of the present application.

[0053] Figure 5 This is a block diagram of the principles of the terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] The embodiments of this application provide an image understanding method, apparatus, device, and storage medium. To clarify the purpose, technical solution, and effects of this application, the present application is further described below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are intended only to explain this application and are not intended to limit this application.

[0055] It will be understood by those skilled in the art that, unless expressly stated otherwise, the singular forms "a", "an", "said" and "the" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, "connected" or "coupled" as used herein may include wireless connections or wireless couplings. The term "and / or" used herein includes all or any units and all combinations of one or more associated listed items.

[0056] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with their meanings in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0057] It should be understood that the sequence numbers and sizes of the steps in this embodiment do not imply the order of execution. The order of execution of each process is determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of this application.

[0058] The application content will be further explained below through description of embodiments in conjunction with the accompanying drawings.

[0059] This embodiment provides an image understanding method, such as Figure 1 and Figure 2 As shown, the image understanding method specifically includes:

[0060] S10: Obtain the image to be understood and the task token.

[0061] Specifically, the task token is used to reflect the understanding task to be performed on the image to be understood. In other words, the task to be performed on the image to be understood is determined based on the task token. There can be one task token or multiple task tokens. When there are multiple task tokens, each of the multiple task tokens corresponds to a single understanding task. The multiple task tokens can include task tokens reflecting the same understanding task or task tokens reflecting different understanding tasks.

[0062] The image to be understood can be a real-time image, a video frame in a video, an image sent by an external device, a locally stored image, or an image uploaded by a user. The understanding task represented by the task token can be an object description task, an object recognition task, a character recognition task, or a visual question answering task. When the understanding task represented by the task token is an object description task, the task token can be represented as <cap>; When the understanding task represented by the task token is an object recognition task, the task token can be represented as <roc>; When the understanding task represented by the task token is a character recognition task, the task token can be represented as <ocr>; When the understanding task represented by the task token is a visual question answering task, the task token can be represented as <vqa>, and the visual question answering task token carries a question text, wherein the question text can be input by the user or a default question text. For example, the understanding task represented by the task token is a visual question answering task, and the task token is <vqa>Is the cat standing?

[0063] It should be noted that the understanding task represented by the task token can also be other tasks, and when the understanding task represented by the task token is a visual question-answering task, the language of the question text carried by the task token is not limited, for example, the language of the question text is Chinese, English, etc.

[0064] S20: Acquire multi-scale image features, multi-scale image coding features, and target area position information of the image to be understood.

[0065] Specifically, the multi-scale image features and multi-scale image coding features are obtained by extracting features from the image to be understood, wherein the multi-scale image features and multi-scale image coding features can be obtained by extracting features from the image to be understood separately, or the multi-scale image features can be obtained by directly extracting features from the image to be understood, and the multi-scale image coding features can be obtained by re-extracting features from the multi-scale image features. In addition, the number of image features contained in each image feature and the number of image features contained in the multi-scale image coding features can be the same or different; and the image scale of each image feature in the multi-scale image feature and the image scale of each image coding feature in the multi-scale image coding features can also be set according to actual needs, without specific restrictions here.

[0066] Exemplarily, the multi-scale image features and the multi-scale image coding features of the image to be understood are obtained through a preset target detection model, wherein the target detection model includes a backbone network and an encoder, the backbone network is connected to the encoder, the input item of the backbone network is the image to be understood, and the output item of the backbone network is the multi-scale image features; the input item of the encoder is the multi-scale image features, and the output item is the multi-scale image coding features. Based on this, the process of obtaining the multi-scale image features and the multi-scale image coding features specifically includes:

[0067] Inputting the image to be understood into the backbone network, and outputting multi-scale image features through the backbone network;

[0068] The multi-scale image features are input into the encoder, and the encoder outputs the multi-scale image coding features.

[0069] Specifically, after acquiring the image to be interpreted, it is fed into the object detection model. The backbone network in the object detection model extracts multi-scale image features. Each layer of the backbone network extracts image features at a different scale, resulting in different dimensions for each image feature. Accordingly, the multi-scale image features are fed into the encoder, which extracts multi-scale image encoding features.

[0070] The target region position information is the position information of a local region image within the image to be understood, and the local region image corresponding to the target region position information is the image region to be understood within the image to be understood. In other words, the target region position information may be the position coordinates of a bounding box of the image region to be understood within the image to be understood, and the image region to be understood may be determined using this target region position information. Of course, the image region to be understood may include an object to be understood, and the image region to be understood may contain one or more objects to be understood.

[0071] In one embodiment, one task token may correspond to one target area location information, or one target area location information may correspond to multiple task tokens. That is to say, multiple task tokens may simultaneously understand multiple local area images in the image to be understood, or may understand one local area image in the image to be understood. When the image area to be understood includes an object to be understood, one task token may correspond to one target area location information; if the image area to be understood includes multiple objects to be understood, one target area location information may correspond to multiple task tokens, that is, each object to be understood corresponds to one task token. At this time, multiple task tokens corresponding to one target area location information may execute tasks synchronously.

[0072] Furthermore, the target area location information may be output by a decoder in a target detection model, or may be pre-labeled by a user, or may be determined based on interactive information input by a user, or may be determined based on the question text in the task token when the understanding task represented by the task token is a visual question-answering task. Specifically, if the target area location information is output by a decoder in a target detection model, the target area location information may be determined by inputting the multi-scale image coding features into a decoder after obtaining the multi-scale image coding features, and performing target detection through the decoder to obtain the target area location information. If the target area location information is pre-labeled by a user, the target area location information may be determined by reading the position coordinates of the labeling box in the image to be understood to obtain the target area location information. If the target area location information is determined based on interactive information input by a user, the target area location information may be determined by interacting with the user during the image understanding process, and the interactive information may include information on the user's area selection in the image to be understood or text prompt information based on the user's input. If the target area location information is determined based on the question text in the task token, the prompt text can be used as a prompt word and the target area location information can be obtained through detection by a target detection model, or the question text can contain the target area location information. For example, the question text includes four-dimensional coordinate information consisting of the center point coordinates of the annotation box and the width and height of the annotation box, or four-dimensional coordinate information consisting of the coordinates of the diagonal points of the annotation box, etc.

[0073] It should be noted that when the understanding task represented by the task token is a visual question-answering task, in addition to determining the target area location information based on the question text in the task token, the target area location information can also be used to determine the image area targeted by the question text in the task token. Specifically, after obtaining the target area location information, the question text posed for the image area corresponding to the target area location information is received. That is to say, when the understanding task represented by the task token is a visual question-answering task, the target area location information is first determined, and then the question text for the image area corresponding to the target area location information is received, that is, the question text in the task token is posed for the image area corresponding to the target area location information; or, the question text in the task token is first obtained, and then the target area location information is determined based on the question text in the task token, that is, the target area location information is determined based on the question text.

[0074] S30: Determine local fine-grained image features based on the multi-scale image features and the target region position information, and determine semantic features based on the multi-scale image coding features and the target region position information.

[0075] Specifically, local fine-grained image features are used to reflect the shallow information of the local area image corresponding to the target area position information, and semantic features are used to reflect the deep semantic features of the local area image corresponding to the target area position information. By combining local fine-grained image features and semantic features to perform task processing, more accurate task processing results can be obtained.

[0076] Exemplarily, the determining of the local fine-grained image features based on the multi-scale image features and the target area position information specifically includes:

[0077] Inputting the target area position information and the multi-scale image features into a local fine-grained feature extractor, and selecting a target image feature from the multi-scale image features by the local fine-grained feature extractor;

[0078] The local fine-grained feature extractor extracts the local image features corresponding to the target area position information from the target image features to obtain local fine-grained image features.

[0079] Specifically, the local fine-grained feature extractor is used to extract local fine-grained image features. That is, multi-scale image features and target area position information can be input into the local fine-grained feature extractor, and local fine-grained image features can be extracted by the local fine-grained feature extractor. Among them, the local fine-grained feature extractor can adopt operators such as RoIAlign or RoIPoo l ing.

[0080] Furthermore, when selecting a target image feature from the multi-scale image features, it can be selected based on the area or size ratio of the local area image corresponding to the target area position information. For example, the area ratio of the local area image to the image area of ​​multiple image features can be calculated, and then the image feature with the area ratio closest to the preset area ratio is selected as the target image feature, or the width ratio of the width of the local area image to the image width of multiple image features, and the height ratio of the height of the local area image to the image height of multiple image features are calculated, and then the target image feature is selected from the multi-scale image features based on the width ratio and height ratio (for example, the width ratio or height ratio closest to the preset ratio is selected, or the average of the width ratio and height ratio closest to the preset ratio is selected, etc.). Of course, in actual applications, one can randomly select one from the multi-scale image features as the target image feature, or the target image feature can be selected from the multi-scale image features through interaction with the user.

[0081] After selecting the target image features, the local area image corresponding to the target position area information is mapped to the target image features, and the local feature area corresponding to the local area image in the target image features is used as the local fine-grained image features, that is, the local image features corresponding to the local area image are extracted on the target image features, and the extracted local image features are used as the local fine-grained image features. The local fine-grained image features in this application are obtained by extracting the shallow image features extracted by the backbone network, which can play an important role in the understanding of shallow information (such as color, material, texture, etc.) and character recognition.

[0082] Exemplarily, determining the semantic feature based on the multi-scale image coding feature and the target area position information specifically includes:

[0083] Initializing a content coding vector and obtaining a position coding vector corresponding to the target area position information;

[0084] A semantic feature is determined based on the content coding vector, the position coding vector, and the multi-scale image coding feature.

[0085] Specifically, the position encoding vector can be a 4-dimensional coordinate vector obtained by position encoding the target area position information. For example, the target area position information includes the width and height of the annotation box and the coordinates of the center point of the annotation box, that is, the target area position information is expressed as {x, y, w, h}, then the position encoding vector is a 4-dimensional vector including x, y, w, h, that is, the position encoding vector can be expressed as [x, y, w, h]. In another embodiment, the target area position information includes information of the diagonal vertices of the annotation box, that is, [x1, y1, x2, y2]. Of course, in actual applications, other methods can also be used to represent the position encoding vector, for example, a 4-dimensional coordinate vector and a one-dimensional area vector can be used.

[0086] Specifically, the content coding vector can be randomly generated in advance, that is, the content coding vector is a random initialization vector, which is used to construct a query vector with the position coding vector. The query vector interacts with the multi-scale image coding features to update the content coding vector and obtain an updated content coding vector.

[0087] Based on this, determining the semantic feature based on the content coding vector, the position coding vector, and the multi-scale image coding feature specifically includes:

[0088] The content coding vector, the position coding vector and the multi-scale image coding feature are input into a semantic feature encoder, and the semantic feature is output through the semantic feature encoder.

[0089] Specifically, the semantic feature decoder is used to extract semantic features. The input items of the semantic feature decoder include a query vector constructed based on the content coding vector and the position coding vector. The content coding vector in the query vector is updated based on the multi-scale image coding feature. The semantic feature encoder includes several coding layers. The input items of each coding layer include the updated content coding vector output by the previous coding layer, the position coding vector and the multi-scale image coding feature. The output items of each coding layer include the updated content coding vector. The content coding vector included in the input items of the first coding layer is a randomly initialized content coding vector. The updated content coding vector output by the last coding layer is used as the semantic feature. In other words, the updated content coding vector output by the first coding layer of two adjacent coding layers will be used as the content coding vector of the next coding layer, and then the query vector of the next coding layer is constructed based on the content coding vector and the position coding vector.

[0090] Furthermore, in one implementation, Figure 3 As shown, the encoding layer can include a self-attention module and a variable cross-attention module. The self-attention module is used to perform self-attention learning on the content coding vector and the position coding vector. The variable cross-attention module is used to construct a query vector using the content coding vector and the position coding vector learned through self-attention, and interactively learn with the multi-scale image coding features to obtain an updated content coding vector. Based on this, the process of determining the updated content coding vector specifically includes:

[0091] Constructing a first query vector, a first value vector, and a first key vector based on the updated content encoding vector output by the previous encoding layer;

[0092] concatenating the first query vector and the position encoding vector to obtain a second query vector, and concatenating the first key vector and the position encoding vector to obtain a second key vector;

[0093] Outputting a third query vector through a self-attention module based on the second query vector, the first value vector, and the second key vector;

[0094] concatenating the third query vector and the position encoding vector to obtain a fourth query vector, and constructing a second value vector based on the multi-scale image encoding feature;

[0095] Based on the fourth query vector and the second value vector, an updated content encoding vector is output through a deformable cross-attention module.

[0096] Specifically, the output item of each coding layer is the updated content coding vector, and the output item of the previous coding layer of the two adjacent coding layers is the input item of the next coding layer, and the input item of the front coding layer is the initialized content coding vector. Here, the process of determining the updated content coding vector of the non-front coding layer is used as an example to illustrate. To this end, the updated content coding vector output by the previous coding layer is first obtained, and then the first query vector, the first value vector and the first key vector are constructed based on the updated content coding vector. After the first query vector, the first value vector and the first key vector are constructed, the position coding vector is spliced ​​with the first query vector and the first key vector respectively, so that the spliced ​​second query vector and the second key vector both include the updated content coding vector and the position coding vector.

[0097] After obtaining the second query vector, the first value vector, and the second key vector, the self-attention mechanism in the self-attention module performs self-attention learning on the second query vector, the first value vector, and the second key vector to obtain a third query vector. The third query vector and the position encoding vector are then spliced ​​together so that the spliced ​​fourth query vector includes the third query vector and the position encoding vector. Finally, the fourth query vector and the second value vector are cross-attention learned by the cross-attention mechanism in the deformable cross-attention module to obtain an updated content encoding vector. In particular, when constructing the second value vector based on the multi-scale image coding feature, the multi-scale image coding feature can be first flattened to form a flattened image feature, and then the flattened image feature is used as the second value vector.

[0098] S40: Determine a task processing result of the image to be understood based on the semantic features, the local fine-grained image features, and the task token.

[0099] Specifically, after acquiring semantic features and local fine-grained image features, the semantic features and local fine-grained image features are used as feature information of the image model, and the task token is used as feature information of the text modality. Based on the feature information of the image model and the feature information of the text modality, the image to be understood is understood to obtain the task processing result. This not only combines multimodal information for image understanding, but also focuses on the visual information and contextual information of the local area of ​​the image to be understood, thereby improving the accuracy of the task processing result.

[0100] Exemplarily, determining the task processing result of the image to be understood based on the semantic features, the local fine-grained image features, and the task token specifically includes:

[0101] aligning the feature dimensions of the semantic features and the local fine-grained image features with the feature dimensions of the text vector;

[0102] The aligned semantic features, the aligned local fine-grained image features and the task token are input into a text decoder, and the task processing result is output through the text decoder.

[0103] Specifically, the semantic features and local fine-grained image features are each passed through a linear layer, and the feature dimensions of the semantic features and local fine-grained image features are aligned to the feature dimensions of the text vector through the linear layer. Then, they are input into the text decoder together with the task token, and the task processing result is output by the text decoder. The feature dimension of the text vector is determined based on the task token, and the task token can control the task processing result output by the text decoder. In addition, the text decoder can be, but is not limited to, any autoregressive text decoder, such as OPT, GPT, etc.

[0104] In summary, this embodiment provides an image understanding method, which includes obtaining an image to be understood and a task token; obtaining multi-scale image features, multi-scale image coding features and target area location information of the image to be understood; determining local fine-grained image features based on the multi-scale image features and the target area location information, and determining semantic features based on the multi-scale image coding features and the target area location information; and determining the task processing result of the image to be understood based on the semantic features, the local fine-grained image features and the task token. This application improves the accuracy of understanding the local area image by focusing on the local understanding of the local area image corresponding to the target area location information using the target area location information as the area indication information. At the same time, this application simultaneously extracts the local fine-grained image features and semantic features of the local area image, provides shallow information of the local area image through the local fine-grained image features, provides feature representation for semantic understanding through the semantic features, and then determines the task processing result based on the local fine-grained image features and the semantic features, which can ensure the accuracy of the task processing result.

[0105] Based on the above image understanding method, this embodiment provides an image understanding device, such as Figure 4 As shown, the image understanding device specifically includes:

[0106] Content acquisition module 100, used to acquire the image to be understood and the task token;

[0107] An image feature acquisition module 200 is configured to acquire multi-scale image features, multi-scale image coding features, and target region location information of the image to be understood;

[0108] a semantic feature determination module 300 for determining local fine-grained image features based on the multi-scale image features and the target region position information, and determining semantic features based on the multi-scale image coding features and the target region position information;

[0109] The task processing result determination module 400 is configured to determine the task processing result of the image to be understood based on the semantic features, the local fine-grained image features, and the task token.

[0110] Based on the above-mentioned image understanding method, this embodiment provides a computer-readable storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the steps in the image understanding method as described in the above-mentioned embodiment.

[0111] Based on the above image understanding method, this application also provides a terminal device, such as Figure 5 As shown, it includes at least one processor 20; a display screen 21; and a memory 22. It may also include a communication interface 23 and a bus 24. The processor 20, display screen 21, memory 22, and communication interface 23 can communicate with each other via bus 24. The display screen 21 is configured to display a preset user guidance interface in the initial setup mode. The communication interface 23 can transmit information. The processor 20 can call logic instructions in the memory 22 to execute the method in the above embodiment.

[0112] In addition, the logic instructions in the memory 22 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product.

[0113] The memory 22, as a computer-readable storage medium, can be configured to store software programs or computer-executable programs, such as program instructions or modules corresponding to the methods in the embodiments of the present disclosure. The processor 20 executes the software programs, instructions, or modules stored in the memory 22 to perform functional applications and data processing, thereby implementing the methods in the above embodiments.

[0114] The memory 22 may include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function; the data storage area may store data created based on the use of the terminal device. In addition, the memory 22 may include high-speed random access memory and non-volatile memory. For example, various media that can store program code, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, may also be transient storage media.

[0115] In addition, the specific process of loading and executing the multiple instructions in the storage medium and the processor in the terminal device has been described in detail in the above method and will not be described here one by one.

[0116] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.< / vqa> < / vqa> < / ocr> < / roc> < / cap>

Claims

1. An image understanding method, characterized in that: The image understanding method specifically includes: Get the image to be understood and the task token; Acquire multi-scale image features, multi-scale image coding features, and target area position information of the image to be understood; Determining local fine-grained image features based on the multi-scale image features and the target region position information, and determining semantic features based on the multi-scale image coding features and the target region position information; Determining a task processing result of the image to be understood based on the semantic features, the local fine-grained image features, and the task token; The determining of the local fine-grained image features based on the multi-scale image features and the target area position information specifically includes: Inputting the target area position information and the multi-scale image features into a local fine-grained feature extractor, and selecting a target image feature from the multi-scale image features by the local fine-grained feature extractor; Extracting local image features corresponding to the target area position information from the target image features by the local fine-grained feature extractor to obtain local fine-grained image features; The determining of the semantic feature based on the multi-scale image coding feature and the target area position information specifically includes: Initializing a content coding vector and obtaining a position coding vector corresponding to the target area position information; A semantic feature is determined based on the content coding vector, the position coding vector, and the multi-scale image coding feature.

2. The image understanding method according to claim 1, wherein: The multi-scale image features and the multi-scale image coding features of the image to be understood are obtained through a preset target detection model, the target detection model includes a backbone network and an encoder, and the acquisition process specifically includes: Inputting the image to be understood into the backbone network, and outputting multi-scale image features through the backbone network; The multi-scale image features are input into the encoder, and the encoder outputs the multi-scale image coding features.

3. The image understanding method according to claim 2, wherein: The target detection model also includes a decoder, and the target area position information is output by the decoder based on the multi-scale image coding features.

4. The image understanding method according to claim 1 or 3, characterized in that: The target area location information is determined based on a labeling box carried in the image to be understood; or, is determined based on interactive information input by a user, wherein the interactive information includes information on area selection by the user in the image to be understood or text prompt information input by the user.

5. The image understanding method according to claim 1, wherein: The task token is used to reflect the understanding task performed on the image to be understood, and when the understanding task is a visual question answering task, the task token carries a question text.

6. The image understanding method according to claim 5, characterized in that The question text is proposed for the image area corresponding to the target area position information; or, the target area position information is determined based on the question text.

7. The image understanding method according to claim 1, wherein: The image area corresponding to the target area position information includes one or more objects to be understood.

8. The image understanding method according to claim 1, wherein: When selecting a target image feature from the multi-scale image features, the feature is selected based on the area or size ratio of the local region image corresponding to the target region position information.

9. The image understanding method according to claim 1, wherein: The determining of the semantic feature based on the content coding vector, the position coding vector, and the multi-scale image coding feature specifically includes: The content coding vector, the position coding vector, and the multi-scale image coding feature are input into a semantic feature encoder, and the semantic feature is output through the semantic feature encoder, wherein the semantic feature encoder includes several coding layers, the input items of the coding layers include the updated content coding vector output by the previous coding layer, the position coding vector, and the multi-scale image coding feature, the output items of the coding layers of each layer include the updated content coding vector, the content coding vector included in the input item of the frontmost coding layer is a randomly initialized content coding vector, and the updated content coding vector output by the last coding layer is used as the semantic feature.

10. The image understanding method according to claim 9, characterized in that: The encoding layer includes a self-attention module and a deformable cross-attention module. The process of determining the updated content encoding vector specifically includes: Constructing a first query vector, a first value vector, and a first key vector based on the updated content encoding vector output by the previous encoding layer; concatenating the first query vector and the position encoding vector to obtain a second query vector, and concatenating the first key vector and the position encoding vector to obtain a second key vector; Outputting a third query vector through the self-attention module based on the second query vector, the first value vector, and the second key vector; concatenating the third query vector and the position encoding vector to obtain a fourth query vector, and constructing a second value vector based on the multi-scale image encoding feature; Based on the fourth query vector and the second value vector, an updated content encoding vector is output through the deformable cross attention module.

11. The image understanding method according to claim 1, wherein: The determining of the task processing result of the image to be understood based on the semantic feature, the local fine-grained image feature, and the task token specifically includes: aligning feature dimensions of the semantic features and the local fine-grained image features with feature dimensions of a task token; The aligned semantic features, the aligned local fine-grained image features and the task token are input into a text decoder, and the task processing result is output through the text decoder.

12. An image understanding device, characterized in that: The image understanding device specifically includes: Content acquisition module, used to obtain the image to be understood and the task token; An image feature acquisition module, configured to acquire multi-scale image features, multi-scale image coding features, and target area location information of the image to be understood; a semantic feature determination module, configured to determine local fine-grained image features based on the multi-scale image features and the target region position information, and to determine semantic features based on the multi-scale image coding features and the target region position information; a task processing result determination module, configured to determine a task processing result of the image to be understood based on the semantic features, the local fine-grained image features, and the task token; The determining of the local fine-grained image features based on the multi-scale image features and the target area position information specifically includes: Inputting the target area position information and the multi-scale image features into a local fine-grained feature extractor, and selecting a target image feature from the multi-scale image features by the local fine-grained feature extractor; Extracting local image features corresponding to the target area position information from the target image features by the local fine-grained feature extractor to obtain local fine-grained image features; The determining of the semantic feature based on the multi-scale image coding feature and the target area position information specifically includes: Initializing a content coding vector and obtaining a position coding vector corresponding to the target area position information; A semantic feature is determined based on the content coding vector, the position coding vector, and the multi-scale image coding feature.

13. A computer-readable storage medium, characterized in that The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the image understanding method according to any one of claims 1 to 11.

14. A terminal device, characterized in that: include: processor and memory; The memory stores a computer-readable program executable by the processor; When the processor executes the computer-readable program, the steps of the image understanding method according to any one of claims 1 to 11 are implemented.

Images