Image processing method and device and storage medium

Abstract

The invention relates to an image processing method and device and a storage medium, and the method comprises the steps: obtaining an original image and a demand expression for the original image, and carrying out the cutting of the original image, and obtaining a plurality of local image blocks; the resolution of the original image is adjusted to obtain a target image, and the resolution of the target image is the same as the resolution of each local image block; coding the plurality of local image blocks by adopting a cross-block relation strategy to obtain a coding result; and processing the coding result, the target image and the demand expression to obtain an answer expression corresponding to the demand expression. According to the method, a cross-block relation strategy is adopted, and the local image blocks are coded, so that the problems of semantic splitting and space splitting can be effectively relieved, and a high-resolution image can be well processed.

Description

Technical Field

[0001] The present application relates to the field of information processing technology, and in particular to an image processing method, device and storage medium. Background Art

[0002] The big model is a large-scale deep learning model. Its core idea is to fuse different information (such as text, images, audio and video, etc.) and achieve more intelligent information processing by learning the relationship between different information. In the big model, different information is input into a big model after preprocessing, and finally outputs the corresponding results after multiple layers of feature extraction and fusion.

[0003] Currently, in the large model, the input images are uniformly adjusted to a fixed resolution, which will cause the resolution of some high-resolution images to be severely compressed when input into the large model, resulting in loss of visual information. For this reason, a cropping strategy is adopted to crop the image into several image blocks and input them into the large model.

[0004] However, this cropping strategy may cause semantic and spatial fragmentation, resulting in inaccurate reasoning results when the large model processes related tasks. For example, for "Backup", the cropping strategy may inadvertently split it into "Back" and "up", causing semantic fragmentation, making the semantics incoherent, and "Backup" is separated in space, resulting in inaccurate reasoning results when the large model processes related tasks. Summary of the invention

[0005] In order to solve the technical problem that the above-mentioned cropping strategy may cause semantic fragmentation and spatial fragmentation, resulting in inaccurate results of large model reasoning, the embodiment of the present application provides an image processing method, device, electronic device and storage medium. The specific technical solution is as follows:

[0006] In a first aspect of an embodiment of the present application, an image processing method is first provided, the method comprising:

[0007] Acquire an original image and a requirement statement for the original image, and slice the original image to obtain a plurality of local image blocks;

[0008] Adjusting the resolution of the original image to obtain a target image, wherein the resolution of the target image is the same as the resolution of each of the local image blocks;

[0009] Adopting a cross-block connection strategy, encoding the plurality of local image blocks to obtain encoding results;

[0010] The encoding result, the target image and the requirement statement are processed to obtain an answer statement corresponding to the requirement statement.

[0011] In an optional implementation, the slicing of the original image to obtain a plurality of local image blocks includes:

[0012] Determining a scaling resolution corresponding to the original image, and performing scaling processing on the original image so that the resolution of the scaled original image reaches the scaling resolution;

[0013] The scaled original image is cut into blocks to obtain a plurality of local image blocks.

[0014] In an optional implementation, determining the scaling resolution corresponding to the original image includes:

[0015] Acquire a predefined resolution set, wherein the predefined resolution set includes a plurality of predefined resolutions, and each of the predefined resolutions is a multiple of a preset resolution;

[0016] For any of the predefined resolutions in the predefined resolution set, determining an intersection-over-union ratio of the predefined resolution and a resolution of the original image;

[0017] According to the intersection-over-union ratio, a target predefined resolution in the predefined resolution set is screened, and the target predefined resolution is determined as the scaled resolution corresponding to the original image.

[0018] In an optional implementation, the cross-slice connection strategy is adopted to encode the plurality of local image blocks to obtain encoding results, including:

[0019] Inputting the plurality of local image blocks into a pre-trained first visual encoder for encoding processing to obtain a plurality of local visual features;

[0020] Among them, the pre-trained first visual encoder adopts a sliding window attention mechanism to encode multiple local image blocks and output multiple local visual features.

[0021] In an optional implementation, the encoding result, the target image, and the requirement statement are processed to obtain an answer statement corresponding to the requirement statement, including:

[0022] Encoding the requirement statement to obtain text encoding features;

[0023] Inputting the target image into a pre-trained second visual encoder for encoding to obtain a global visual feature;

[0024] Inputting the plurality of local visual features and the global visual features into a pre-trained visual resampling model to obtain a plurality of resampled local visual features and a resampled global visual features;

[0025] Input the multiple resampled local visual features, the resampled global visual features, and the text encoding features into a pre-trained large-scale language model to obtain an answer statement corresponding to the demand statement.

[0026] In an optional implementation, the inputting of the plurality of the resampled local visual features, the resampled global visual features, and the text encoding features into a pre-trained large language model to obtain an answer statement corresponding to the demand statement includes:

[0027] Selecting a target number of the resampled local visual features from the plurality of the resampled local visual features;

[0028] The target number of the resampled local visual features, the resampled global visual features, and the text encoding features are input into a pre-trained large-scale language model to obtain an answer statement corresponding to the demand statement.

[0029] In an optional implementation, selecting a target number of the resampled local visual features from the plurality of the resampled local visual features comprises:

[0030] Sampling from a preset compression probability set to obtain a target compression probability, and obtaining a characteristic length corresponding to the scaled resolution of the original image;

[0031] Determining the number of targets according to the target compression probability and the characteristic length;

[0032] The target number of the resampled local visual features is selected from the plurality of the resampled local visual features.

[0033] In an optional implementation, selecting the target number of the resampled local visual features from the plurality of the resampled local visual features comprises:

[0034] For any of the resampled local visual features, determining an importance score of the resampled local visual feature;

[0035] According to the importance scores, the plurality of resampled local visual features are sorted to obtain a resampled local visual feature sorting result;

[0036] According to the ranking result of the resampled local visual features, the target number of the resampled local visual features are selected.

[0037] In an optional implementation, determining the importance score of any of the resampled local visual features includes:

[0038] For any of the resampled local visual features, determining a similarity between the resampled local visual feature and any of the remaining resampled local visual features;

[0039] An importance score of the resampled local visual feature is determined based on the multiple similarities.

[0040] In an optional implementation, determining the importance score of the resampled local visual feature according to the multiple similarities includes:

[0041] For any of the similarities, obtaining a difference between a preset value and the similarity;

[0042] A weighted sum is performed on the obtained multiple differences to obtain an importance score of the resampled local visual feature.

[0043] In an optional implementation, the inputting of the target number of the resampled local visual features, the resampled global visual features, and the text encoding features into a pre-trained large language model to obtain an answer statement corresponding to the demand statement includes:

[0044] Determine a first initial feature order corresponding to each of the target number of the resampled local visual features, and sort the target number of the resampled local visual features according to the first initial feature order to obtain a first local visual feature sequence;

[0045] Determine a second initial feature order corresponding to each of the remaining resampled local visual features, and sort the remaining resampled local visual features according to the second initial feature order to obtain a second local visual feature sequence;

[0046] Aggregating the first local visual feature sequence and the second local visual feature sequence to obtain a plurality of aggregated local visual features;

[0047] The plurality of aggregated local visual features, the resampled global visual features, and the text encoding features are input into a pre-trained large-scale language model to obtain an answer statement corresponding to the demand statement.

[0048] In an optional implementation, the inputting of the plurality of the aggregated local visual features, the resampled global visual features, and the text encoding features into a pre-trained large language model to obtain an answer statement corresponding to the requirement statement includes:

[0049] Input the plurality of aggregated local visual features, the resampled global visual features, and the text encoding features into a pre-trained large language model, and the pre-trained large language model performs the following processing:

[0050] Generate an attention distribution map of visual features according to the plurality of aggregated local visual features, the resampled global visual features, and the text encoding features;

[0051] According to the attention distribution map, selecting a preset proportion of target visual features from the plurality of aggregated local visual features and the resampled global visual features;

[0052] According to the target visual features of the preset proportion, an answer statement corresponding to the demand statement is generated.

[0053] In an optional embodiment, the attention distribution map includes the degree of attention of the text encoding feature to each of the aggregated local visual features and the resampled global visual features;

[0054] The step of selecting a preset proportion of target visual features from the plurality of aggregated local visual features and the resampled global visual features according to the attention distribution map includes:

[0055] According to the attention degree, sorting the plurality of aggregated local visual features and the resampled global visual features to obtain a visual feature sorting result;

[0056] A preset proportion of target visual features is selected from the visual feature sorting results.

[0057] In an optional implementation, generating an answer statement corresponding to the demand statement according to the target visual features of the preset proportion includes:

[0058] For any of the target visual features, searching for adjacent visual features of the target visual feature;

[0059] An answer statement corresponding to the demand statement is generated according to the preset ratio of target visual features and adjacent visual features of each of the target visual features.

[0060] In an optional implementation, before executing the image processing method, the method further includes:

[0061] Obtaining a sample image and a sample requirement statement for the sample image, and determining a sample answer statement corresponding to the sample requirement statement;

[0062] Performing block processing on the sample image to obtain a plurality of sample local image blocks, and adjusting the resolution of the sample image to obtain a sample target image;

[0063] Inputting the plurality of sample local image blocks into a first visual encoder for encoding processing to obtain a plurality of predicted local visual features;

[0064] Encoding the sample demand statement to obtain sample text encoding features, and inputting the sample target image into a second visual encoder to obtain predicted global visual features;

[0065] Inputting the plurality of predicted local visual features and the predicted global visual features into a visual resampling model to obtain a plurality of resampled predicted local visual features and a resampled predicted global visual features;

[0066] Inputting a plurality of the resampled predicted local visual features, the resampled predicted global visual features, and the sample text encoding features into a large language model to obtain a predicted answer expression;

[0067] Determine a representation loss between the predicted answer representation and the sample answer representation, and perform model training on the first visual encoder, the second visual encoder, the visual resampling model, and the large language model according to the representation loss;

[0068] When the representation loss converges, the model training is stopped to obtain a pre-trained first visual encoder, a pre-trained second visual encoder, a pre-trained visual resampling model, and a pre-trained large language model.

[0069] In an optional implementation, the slicing of the sample image to obtain a plurality of sample local image blocks includes:

[0070] Determining a sample scaling resolution corresponding to the sample image;

[0071] Scaling the sample image so that the resolution of the scaled sample image reaches the sample scaling resolution;

[0072] The scaled sample image is cut into blocks to obtain a plurality of sample local image blocks.

[0073] In an optional implementation, determining the sample scaling resolution corresponding to the sample image includes:

[0074] Acquire a predefined resolution set, wherein the predefined resolution set includes a plurality of predefined resolutions, and each of the predefined resolutions is a multiple of a preset resolution;

[0075] For any of the predefined resolutions in the predefined resolution set, determining a sample intersection-over-union ratio of the predefined resolution to a resolution of the sample image;

[0076] Filtering the sample predefined resolutions in the predefined resolution set according to the sample intersection-over-union ratio;

[0077] The sample predefined resolution is determined as the sample scaling resolution corresponding to the sample image.

[0078] In an optional implementation, the inputting of the plurality of the resampled predicted local visual features, the resampled predicted global visual features, and the sample text encoding features into a large language model to obtain a predicted answer representation includes:

[0079] Selecting the resampled predicted local visual features of the number of samples from the plurality of resampled predicted local visual features;

[0080] The resampled predicted local visual features, the resampled predicted global visual features, and the sample text encoding features of the sample quantity are input into a large language model to obtain a predicted answer expression.

[0081] In an optional implementation, the selecting the resampled predicted local visual features of the number of samples from the plurality of resampled predicted local visual features comprises:

[0082] Sampling is performed from a preset compression probability set to obtain a sample compression probability, and a sample feature length corresponding to the sample scaling probability of the sample image is obtained;

[0083] Determining the number of samples according to the sample compression probability and the sample characteristic length;

[0084] The resampled predicted local visual features of the number of samples are selected from the multiple resampled predicted local visual features.

[0085] In an optional implementation, selecting the resampled predicted local visual features of the sample quantity from the plurality of resampled predicted local visual features comprises:

[0086] For any of the resampled predicted local visual features, determining a sample importance score of the resampled predicted local visual feature;

[0087] Sorting the plurality of resampled predicted local visual features according to the sample importance scores to obtain a resampled predicted local visual feature sorting result;

[0088] According to the ranking result of the resampled predicted local visual features, the resampled predicted local visual features of the number of samples are selected.

[0089] In an optional implementation, determining the sample importance score of any of the resampled predicted local visual features comprises:

[0090] For any of the resampled predicted local visual features, determining a sample similarity between the resampled predicted local visual feature and any of the remaining resampled predicted local visual features;

[0091] According to the plurality of sample similarities, a sample importance score of the resampled predicted local visual feature is determined.

[0092] In an optional implementation, determining the sample importance score of the resampled predicted local visual feature according to the plurality of sample similarities includes:

[0093] For any of the sample similarities, obtaining a sample difference between a preset value and the sample similarity;

[0094] A weighted sum is performed on the obtained multiple sample differences to obtain a sample importance score for the resampled predicted local visual feature.

[0095] In an optional embodiment, the resampled predicted local visual features, the resampled predicted global visual features, and the sample text encoding features are input into a large language model to obtain a predicted answer expression, including:

[0096] Determine the first sample initial feature order corresponding to each of the resampled predicted local visual features of the sample number, and sort the resampled predicted local visual features of the sample number according to the first sample initial feature order to obtain a first sample local visual feature sequence;

[0097] Determine the second sample initial feature order corresponding to each of the remaining resampled predicted local visual features, and sort the remaining resampled predicted local visual features according to the second sample initial feature order to obtain a second sample local visual feature sequence;

[0098] Aggregating the first sample local visual feature sequence and the second sample local visual feature sequence to obtain a plurality of predicted aggregated local visual features;

[0099] The plurality of predicted aggregated local visual features, the resampled predicted global visual features, and the sample text encoding features are input into a large language model to obtain a predicted answer representation.

[0100] In an optional implementation, the step of inputting the plurality of predicted aggregated local visual features, the resampled predicted global visual features, and the sample text encoding features into a large language model to obtain a predicted answer representation includes:

[0101] Input the plurality of predicted aggregated local visual features, the resampled predicted global visual features, and the sample text encoding features into a large language model, and the large language model performs the following processing:

[0102] Generate a predicted attention distribution map of visual features according to the plurality of predicted aggregated local visual features, the resampled predicted global visual features, and the sample text encoding features;

[0103] According to the predicted attention distribution map, selecting a preset proportion of sample visual features from the plurality of predicted aggregated local visual features and the resampled predicted global visual features;

[0104] A predicted answer statement is generated based on the preset proportion of sample visual features.

[0105] In an optional embodiment, the predicted attention distribution map includes the predicted attention degree of the sample text encoding feature to each of the predicted aggregated local visual features and the resampled predicted global visual features;

[0106] The step of selecting a preset proportion of sample visual features from the plurality of predicted aggregated local visual features and the resampled predicted global visual features according to the predicted attention distribution map comprises:

[0107] According to the predicted attention degree, sorting the plurality of predicted aggregated local visual features and the resampled predicted global visual features to obtain a sample visual feature sorting result;

[0108] From the sample visual feature sorting results, sample visual features of a preset proportion are selected.

[0109] In an optional implementation, generating a predicted answer statement according to the preset proportion of sample visual features includes:

[0110] For any of the predicted visual features, searching for sample adjacent visual features of the predicted visual feature;

[0111] A predicted answer statement is generated based on the preset proportion of sample visual features and sample adjacent visual features of each predicted visual feature.

[0112] In a second aspect of the embodiments of the present application, an image processing device is further provided, the device comprising:

[0113] An image and description acquisition module, used to acquire an original image and a required description for the original image;

[0114] An image cutting module is used to cut the original image into pieces to obtain multiple local image blocks;

[0115] An image adjustment module, used for adjusting the resolution of the original image to obtain a target image, wherein the resolution of the target image is the same as the resolution of each of the local image blocks;

[0116] An image block encoding module, used for adopting a cross-slice connection strategy to encode the plurality of local image blocks to obtain an encoding result;

[0117] The expression generation module is used to process the encoding result, the target image and the requirement expression to obtain an answer expression corresponding to the requirement expression.

[0118] In a third aspect of the embodiments of the present application, there is further provided an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0119] Memory, used to store computer programs;

[0120] The processor is used to implement any image processing method described in the first aspect when executing the program stored in the memory.

[0121] In a fourth aspect of the embodiments of the present application, a storage medium is further provided, wherein instructions are stored in the storage medium, and when the storage medium is run on a computer, the computer executes the image processing method described in any one of the first aspects above.

[0122] In a fifth aspect of the embodiments of the present application, a computer program product comprising instructions is also provided, which, when executed on a computer, enables the computer to execute any of the above-mentioned image processing methods.

[0123] The technical solution provided in the embodiment of the present application obtains an original image and a demand statement for the original image, and blocks the original image to obtain multiple local image blocks, adjusts the resolution of the original image to obtain a target image, wherein the resolution of the target image is the same as the resolution of each local image block, adopts a cross-block connection strategy, encodes the multiple local image blocks to obtain an encoding result, processes the encoding result, the target image and the demand statement to obtain an answer statement corresponding to the demand statement.

[0124] The original image is cut into blocks to obtain multiple local image blocks, the resolution of the original image is adjusted to obtain a target image, and a cross-block connection strategy is adopted to encode the multiple local image blocks to obtain an encoding result. The encoding result, the target image and the demand statement are processed to obtain an answer statement corresponding to the demand statement. In this way, the problem of semantic fragmentation and spatial fragmentation can be effectively alleviated by adopting a cross-block connection strategy to encode the multiple local image blocks, thereby ensuring that high-resolution images are well processed. BRIEF DESCRIPTION OF THE DRAWINGS

[0125] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0126] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0127] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0128] Figure 1 A schematic diagram of an implementation flow of an image processing method shown in an embodiment of the present application;

[0129] Figure 2 It is a schematic diagram of an implementation flow of another image processing method shown in an embodiment of the present application;

[0130] Figure 3 A schematic diagram of the structure of a pre-trained large model shown in an embodiment of the present application;

[0131] Figure 4 A schematic diagram of an implementation flow of a method for generating an answer statement shown in an embodiment of the present application;

[0132] Figure 5 A schematic diagram of an implementation flow of a visual feature dynamic compression method shown in an embodiment of the present application;

[0133] Figure 6 A schematic diagram of an implementation flow of another answer statement generation method shown in an embodiment of the present application;

[0134] Figure 7A schematic diagram of an implementation flow of a model training method shown in an embodiment of the present application;

[0135] Figure 8 is a structural schematic diagram of an image processing device shown in an embodiment of the present application;

[0136] Fig. 9 It is a schematic diagram of the structure of an electronic device shown in an embodiment of the present application. DETAILED DESCRIPTION

[0137] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0138] The disclosure below provides many different embodiments or examples to realize the different structures of the present application. In order to simplify the disclosure of the present application, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present application. In addition, the present application can repeat reference numbers and / or letters in different examples. This repetition is for the purpose of simplification and clarity, and does not itself indicate the relationship between the various embodiments and / or settings discussed.

[0139] like Figure 1 FIG. 1 is a schematic diagram of an implementation flow of an image processing method provided in an embodiment of the present application. The method is applied to an electronic device and may specifically include the following steps:

[0140] S101, obtaining an original image and a requirement statement for the original image, and slicing the original image to obtain a plurality of local image blocks.

[0141] In an embodiment of the present application, an original image and a demand statement for the original image may be obtained, where the demand statement is a demand statement described in natural language (which may be understood as a question raised by a user).

[0142] For example, an original image and a requirement statement for the original image are obtained. The requirement statement may be a requirement statement described in natural language, namely, "what is the name mentioned in the resume", "what is the brand of the basketball in the image", "who is the person mentioned in the article", etc.

[0143] In addition, for the original image, the original image may be cut into blocks to obtain a plurality of local image blocks, wherein each local image block is a corresponding local area in the original image, which means that the local area corresponding to the original image is cut into blocks to obtain the local image block.

[0144] It should be noted that, for multiple local image blocks, their resolutions are the same, which means that the resolution of each local image block is a certain resolution, for example, the resolution of each local image block is 448×448, and the embodiment of the present application does not limit this.

[0145] For example, for the original image, the original image can be cut into blocks to obtain 6 local image blocks, namely local image block 1, local image block 2, local image block 3, local image block 4, local image block 5, and local image block 6. The resolution of these 6 local image blocks is 448×448.

[0146] S102, adjusting the resolution of the original image to obtain a target image, wherein the resolution of the target image is the same as the resolution of each local image block.

[0147] In the embodiment of the present application, for the original image, the resolution of the original image can be adjusted to obtain the target image, wherein the resolution of the target image is the same as the resolution of each local image block, which means that the resolution of the original image is adjusted to the same resolution as the local image block.

[0148] For example, for an original image, the resolution of the original image can be adjusted to obtain a target image, the resolution of the target image is 448×448, and the resolution of the target image is the same as the resolution of each local image block, that is, for the original image, its resolution is adjusted to the same resolution as the local image block, 448×448.

[0149] S103, adopting a cross-slice connection strategy to perform encoding processing on multiple local image blocks to obtain encoding results.

[0150] In the embodiment of the present application, for multiple local image blocks, a cross-block connection strategy can be adopted to encode the multiple local image blocks to obtain an encoding result. The cross-block connection strategy refers to linking different local image blocks together for encoding, identifying and fusing the information corresponding to each local image block.

[0151] It should be noted that for the cross-block connection strategy, different local image blocks can be linked together for encoding processing, so as to effectively identify and fuse the information corresponding to each local image block, enhance the understanding of spatial and contextual relationships, and reduce the semantic fragmentation and spatial fragmentation problems caused by blocking.

[0152] S104, processing the encoding result, the target image and the requirement statement to obtain an answer statement corresponding to the requirement statement.

[0153] In an embodiment of the present application, the encoding results of multiple local image blocks, the target image and the demand statement can be processed to obtain an answer statement corresponding to the demand statement (which can be understood as the answer to the question raised by the user).

[0154] Through the above description of the technical solution provided in the embodiment of the present application, an original image and a demand statement for the original image are obtained, and the original image is cut into blocks to obtain multiple local image blocks. The resolution of the original image is adjusted to obtain a target image, wherein the resolution of the target image is the same as the resolution of each local image block. A cross-block connection strategy is adopted to encode the multiple local image blocks to obtain an encoding result. The encoding result, the target image and the demand statement are processed to obtain an answer statement corresponding to the demand statement.

[0155] The original image is cut into blocks to obtain multiple local image blocks, the resolution of the original image is adjusted to obtain a target image, and a cross-block connection strategy is adopted to encode the multiple local image blocks to obtain an encoding result. The encoding result, the target image and the demand statement are processed to obtain an answer statement corresponding to the demand statement. In this way, the problem of semantic fragmentation and spatial fragmentation can be effectively alleviated by adopting a cross-block connection strategy to encode the multiple local image blocks, thereby ensuring that high-resolution images are well processed.

[0156] In addition, in the embodiment of the present application, the method of slicing the original image is essentially to expand the ability of the pre-trained large model to process high-resolution images and complex scene images, and can achieve good processing of high-resolution images and complex scene images. Figure 2 FIG. 1 is a schematic diagram of an implementation flow of another image processing method provided in an embodiment of the present application. The method is applied to an electronic device and may specifically include the following steps:

[0157] S201, obtaining an original image and a requirement statement for the original image, and determining a scaling resolution corresponding to the original image.

[0158] In an embodiment of the present application, an original image and a requirement statement for the original image are obtained. For the original image, a scaling resolution corresponding to the original image is determined. The scaling resolution corresponding to the original image can be determined according to the resolution of the original image to facilitate subsequent block processing, and such processing is to ensure that the original image is not distorted during the scaling process.

[0159] Specifically, a predefined resolution set is obtained, where the predefined resolution set includes multiple predefined resolutions, and each predefined resolution is a multiple of a preset resolution (e.g., 448×448). For any predefined resolution in the predefined resolution set, an intersection-and-union ratio of the predefined resolution to the resolution of the original image is determined, and based on the intersection-and-union ratio, a target predefined resolution in the predefined resolution set is screened, and the target predefined resolution is determined as the scaled resolution corresponding to the original image.

[0160] Among them, for the intersection-and-union ratio, the largest intersection-and-union ratio can be selected, and then the predefined resolution corresponding to the largest intersection-and-union ratio is found, and the predefined resolution corresponding to the largest intersection-and-union ratio is used as the target predefined resolution, which means that the predefined resolution corresponding to the largest intersection-and-union ratio is used as the scaled resolution corresponding to the original image.

[0161] S202: Scaling the original image so that the resolution of the scaled original image reaches the scaling resolution.

[0162] S203, performing block processing on the scaled original image to obtain multiple local image blocks.

[0163] In an embodiment of the present application, after the above steps, the scaling resolution corresponding to the original image is determined, and the original image is scaled so that the resolution of the scaled original image reaches the scaling resolution (i.e., the predefined resolution corresponding to the maximum intersection-over-union ratio), and then the scaled original image is cut into blocks to obtain multiple local image blocks.

[0164] Among them, for the original image, a dynamic resolution strategy is adopted. According to the resolution of the original image, its corresponding scaling resolution is determined. The scaling resolution is a predefined resolution corresponding to the maximum intersection-over-union ratio, and the predefined resolution is a multiple of the preset resolution (for example, 448×448). In this way, the original image is scaled according to the scaling resolution to ensure that the original image is not distorted during the scaling process.

[0165] S204, adjusting the resolution of the original image to obtain a target image, wherein the resolution of the target image is the same as the resolution of each local image block.

[0166] In the embodiment of the present application, this step is similar to the above-mentioned step S102, and the embodiment of the present application will not be described one by one here.

[0167] S205, inputting the multiple local image blocks into a pre-trained first visual encoder for encoding processing to obtain multiple local visual features.

[0168] In the embodiment of the present application, the pre-trained large model generally mainly includes a pre-trained first visual encoder, a pre-trained second visual encoder, a pre-trained visual resampling model, and a pre-trained large language model, such as Figure 3 shown.

[0169] Among them, the pre-trained first visual encoder includes a sliding window attention mechanism, which can link different local image blocks together for encoding processing, thereby effectively identifying and fusing the information corresponding to each local image block, enhancing the understanding of spatial and contextual relationships, and reducing the semantic fragmentation and spatial fragmentation problems caused by cutting.

[0170] Based on this, for multiple local image blocks, the multiple local image blocks are input into the pre-trained first visual encoder for encoding processing to obtain multiple local visual features. Among them, the pre-trained first visual encoder adopts a sliding window attention mechanism to encode the multiple local image blocks and output multiple local visual features.

[0171] S206, encoding the requirement statement to obtain text encoding features.

[0172] In an embodiment of the present application, for the demand expression corresponding to the original image, the demand expression can be encoded to obtain text encoding features.

[0173] S207, input the target image into a pre-trained second visual encoder for encoding processing to obtain global visual features.

[0174] In an embodiment of the present application, the pre-trained second visual encoder reuses the visual encoder in the current large model, which means that the pre-trained second visual encoder does not include a sliding window attention mechanism.

[0175] To this end, for the target image (ie, the original image after adjusting the resolution), the target image can be input into the pre-trained second visual encoder for encoding processing to obtain global visual features.

[0176] S208, inputting the multiple local visual features and the global visual features into a pre-trained visual resampling model to obtain multiple resampled local visual features and resampled global visual features.

[0177] In the embodiment of the present application, for multiple local visual features and global visual features, the multiple local visual features and global visual features can be input into a pre-trained visual resampling model to obtain multiple resampled local visual features and resampled global visual features. Among them, for the pre-trained visual resampling model, it reuses the visual resampling model in the current large model.

[0178] It should be noted that, for the pre-trained visual resampling model, the number can be 1, which means that multiple local visual features and global visual features will be input into a common pre-trained visual resampling model for visual reconstruction and compression to obtain multiple compressed resampled local visual features and resampled global visual features.

[0179] In addition, for the pre-trained visual resampling model, an attention mechanism-based approach is adopted to compress and reconstruct multiple local visual features and global visual features by using a fixed number of query vectors to generate compressed multiple resampled local visual features and resampled global visual features. These fixed query vectors are a set of parameterized vectors learned during the training process, which are used to perform weighted aggregation on each of the above visual features.

[0180] S209, input multiple resampled local visual features, resampled global visual features, and text encoding features into a pre-trained large language model to obtain an answer statement corresponding to the demand statement.

[0181] In an embodiment of the present application, the multiple resampled local visual features, resampled global visual features, and text encoding features obtained in the above steps can be input into a pre-trained large language model to obtain an answer expression corresponding to the demand expression. For the pre-trained large language model, the large language model in the current large model is reused.

[0182] In addition, the method of slicing the original image will cause semantic fragmentation, spatial fragmentation and other problems, resulting in inaccurate reasoning results when the large model processes related tasks. On the other hand, it will also cause a significant increase in feature length, affecting the reasoning speed of the large model. To this end, the embodiment of the present application proposes a dynamic compression mechanism to compress visual features, and a method of using the attention mechanism of a large language model to screen visual features to shorten the feature length and avoid affecting the reasoning speed of the large model.

[0183] Based on this, the above method of inputting multiple resampled local visual features, resampled global visual features, and text encoding features into the pre-trained large language model to obtain the answer expression corresponding to the demand expression can be referred to as follows Figure 4 As shown in the method. Figure 4 FIG. 1 is a schematic diagram of an implementation flow of a method for generating an answer statement provided in an embodiment of the present application. The method is applied to an electronic device and may specifically include the following steps:

[0184] S401, selecting a target number of resampled local visual features from a plurality of resampled local visual features.

[0185] In an embodiment of the present application, for a plurality of resampled local visual features, a target number of resampled local visual features may be selected from the plurality of resampled local visual features, wherein the target number is dynamically determined, and dynamic compression of the resampled local visual features may be achieved.

[0186] Based on this, sampling is performed from the preset compression probability set to obtain the target compression probability, and the feature length corresponding to the scaled resolution of the original image is obtained. According to the target compression probability and the feature length, the number of targets is determined, and the target number of resampled local visual features is selected from multiple resampled local visual features. Here, the feature length corresponds to the scaled resolution of the original image, and the scaled resolution is determined by the resolution of the original image. Therefore, the feature length will change with the change of the resolution of the original image, so that the number of targets is dynamic.

[0187] For example, 20 discrete compression probabilities are pre-defined, uniformly distributed from 30% to 100%, to form a preset compression probability set. Sampling is performed from the preset compression probability set to obtain the target compression probability, and the feature length corresponding to the above-mentioned scaled resolution of the original image is obtained. According to the target compression probability and the feature length, the target number is calculated, so that the target number of resampled local visual features can be selected.

[0188] It should be noted that for the target compression probability and the feature length, the product of the two can be calculated, thereby obtaining the target quantity. In addition, a learnable parameter is defined in the large model, which represents the selection probability of the compression probability of the preset compression probability set. Through this parameter, sampling can be performed from the preset compression probability set to obtain the target compression probability. For learnable parameters, in an embodiment provided in the present application, Gumbel-Softmax (a technique for making discrete selection processes differentiable in deep learning) can be used for training. Among them, Gumbel-Softmax can be controlled by a temperature parameter so that the sampling process is differentiable, so that the learnable parameters can be directly trained in gradient descent.

[0189] In addition, the selection of a target number of resampled local visual features from a plurality of resampled local visual features is generally based on importance, that is, the most important resampled local visual features of a target number are selected from a plurality of resampled local visual features. Based on this, the above method of selecting a target number of resampled local visual features from a plurality of resampled local visual features can be parameterized Figure 5 As shown in the method. Figure 5 FIG. 1 is a schematic diagram of an implementation flow of a method for dynamic compression of visual features provided in an embodiment of the present application. The method is applied to an electronic device and may specifically include the following steps:

[0190] S501 : For any resampled local visual feature, determine an importance score of the resampled local visual feature.

[0191] In the embodiment of the present application, for the above-mentioned multiple resampled local visual features, for any resampled local visual feature, the importance score of the resampled local visual feature is determined. The importance score is generally obtained by the similarity between the features, and the similarity is generally calculated using, for example, cosine distance.

[0192] Based on this, for any resampled local visual feature, the similarity between the resampled local visual feature and any resampled local visual feature in the remaining resampled local visual features is determined, so that multiple similarities can be obtained, and thus the importance score of the resampled local visual feature can be determined based on the multiple similarities.

[0193] For any similarity, the difference between a preset value (e.g., 1) and the similarity can be obtained, so that multiple differences can be obtained, and the obtained multiple differences can be weighted and summed to obtain the importance score of the resampled local visual feature. In this way, the importance score of the resampled local visual feature can be determined according to multiple similarities.

[0194] S502, sorting the multiple resampled local visual features according to the importance scores to obtain a resampled local visual feature sorting result.

[0195] S503 , selecting a target number of resampled local visual features according to the sorting results of the resampled local visual features.

[0196] In an embodiment of the present application, for the importance scores, multiple resampled local visual features are sorted according to the importance scores to obtain the resampled local visual feature sorting result, and based on the resampled local visual feature sorting result, a target number (for example, r) of resampled local visual features are selected.

[0197] It should be noted that for the importance score, generally the smaller the importance score, the more important the corresponding resampled local visual feature is. Therefore, according to the importance score, multiple resampled local visual features can be sorted from small to large to obtain the resampled local visual feature sorting result, and then the top N resampled local visual features are selected, where N is the same as the number of targets.

[0198] S402, inputting the target number of resampled local visual features, resampled global visual features, and text encoding features into a pre-trained large language model to obtain an answer statement corresponding to the demand statement.

[0199] In an embodiment of the present application, after the above steps, a target number of resampled local visual features are selected from multiple resampled local visual features, thereby inputting the target number of resampled local visual features, resampled global visual features, and text encoding features into a pre-trained large language model to obtain an answer statement corresponding to the demand statement.

[0200] Among them, for multiple resampled local visual features, a target number of resampled local visual features are selected from them, and there are still remaining resampled local visual features. In order to avoid the information loss caused by directly discarding the remaining resampled local visual features, important resampled local visual features can be used as query vectors, and cross-attention can be used to further aggregate all resampled local visual features.

[0201] Based on this, Figure 6 FIG. 1 is a schematic diagram of an implementation flow of another answer statement generation method provided in an embodiment of the present application. The method is applied to an electronic device and may specifically include the following steps:

[0202] S601, determining a first initial feature order corresponding to each of the target number of resampled local visual features, and sorting the target number of resampled local visual features according to the first initial feature order to obtain a first local visual feature sequence.

[0203] In an embodiment of the present application, for a target number of resampled local visual features, a first initial feature order corresponding to each of the target number of resampled local visual features can be determined, and the target number of resampled local visual features can be sorted according to the first initial feature order to obtain a first local visual feature sequence.

[0204] Among them, the first initial feature order refers to the initial order of the target number of resampled local visual features in the above-mentioned multiple resampled local visual features. Therefore, the target number of resampled local visual features are sorted according to the initial order to obtain the first local visual feature sequence.

[0205] S602, determining a second initial feature order corresponding to each of the remaining resampled local visual features, and sorting the remaining resampled local visual features according to the second initial feature order to obtain a second local visual feature sequence.

[0206] In an embodiment of the present application, for the remaining resampled local visual features other than the target number of resampled local visual features among multiple resampled local visual features, a second initial feature order corresponding to each of the remaining resampled local visual features is determined, and the remaining resampled local visual features are sorted according to the second initial feature order to obtain a second local visual feature sequence.

[0207] Among them, the second initial feature order refers to the initial order of the remaining resampled local visual features in the above-mentioned multiple resampled local visual features. Therefore, the remaining resampled local visual features are sorted according to the initial order to obtain the second local visual feature sequence.

[0208] S603: Aggregate the first local visual feature sequence and the second local visual feature sequence to obtain a plurality of aggregated local visual features.

[0209] In an embodiment of the present application, for the first local visual feature sequence and the second local visual feature sequence obtained in the above steps, the first local visual feature sequence and the second local visual feature sequence can be aggregated using a cross-attention mechanism to obtain multiple aggregated local visual features.

[0210] S604, input multiple aggregated local visual features, resampled global visual features, and text encoding features into a pre-trained large language model to obtain an answer statement corresponding to the demand statement.

[0211] In an embodiment of the present application, the multiple aggregated local visual features, resampled global visual features, and text encoding features obtained in the above steps can be input into a pre-trained large-scale language model to obtain an answer statement corresponding to the demand statement.

[0212] Among them, multiple aggregated local visual features, resampled global visual features, and text encoding features are input into a pre-trained large language model, and the pre-trained large language model performs the following processing: an attention distribution map of visual features is generated according to the multiple aggregated local visual features, resampled global visual features, and text encoding features; according to the attention distribution map, a preset proportion of target visual features are selected from the multiple aggregated local visual features and the resampled global visual features; according to the preset proportion of the target visual features, an answer statement corresponding to the demand statement is generated.

[0213] It should be noted that the first two layers of the pre-trained large language model are responsible for generating an attention distribution map of visual features based on multiple aggregated local visual features, resampled global visual features, and text encoding features, and then screening multiple aggregated local visual features and resampled global visual features through the attention distribution map to obtain target visual features of a preset proportion. The remaining network of the pre-trained large language model is responsible for generating the answer statement corresponding to the demand statement based on the target visual features of the preset proportion.

[0214] Among them, the attention distribution map includes the degree of attention of the text encoding feature to each aggregated local visual feature and the resampled global visual feature. Therefore, according to the degree of attention, multiple aggregated local visual features and resampled global visual features can be sorted to obtain the visual feature sorting result; from the visual feature sorting result, a preset proportion (for example, 70%) of the target visual features are selected.

[0215] It should be noted that, for the degree of attention, the higher the value of the degree of attention, the more attention the text encoding feature pays to the corresponding aggregated local visual features or resampled global visual features. Therefore, according to the degree of attention, multiple aggregated local visual features and resampled global visual features can be sorted from large to small to obtain a visual feature sorting result, and the top 70% of target visual features are selected from the visual feature sorting result. The target visual features include aggregated local visual features and / or resampled global visual features.

[0216] In addition, considering the continuity of visual features, a saving window can be set, and several features before and after the target visual feature will be retained. For example, for the target visual feature, 5 features before and after can be retained from multiple aggregated local visual features and resampled global visual features. Based on this, for any target visual feature, the adjacent visual features of the target visual feature are searched (for example, the first 5 and last 5 visual features of the target visual feature, and the first 5 and last 5 visual features are searched from multiple aggregated local visual features and resampled global visual features), and the answer statement corresponding to the demand statement is generated according to the preset proportion of target visual features and the adjacent visual features of each target visual feature.

[0217] By adopting the cross-block connection strategy and encoding multiple local image blocks, the problems of semantic fragmentation and spatial fragmentation can be effectively alleviated, thus ensuring good processing of high-resolution images. In addition, a dynamic compression mechanism is proposed to compress visual features, and the attention mechanism of a large language model is used to screen visual features to shorten the feature length and avoid affecting the reasoning speed of the large model.

[0218] For the above-mentioned pre-trained large model, it needs to be trained before it can be obtained. Based on this, Figure 7 As shown, it is a schematic diagram of an implementation flow of a model training method provided in an embodiment of the present application. The method is applied to an electronic device and may specifically include the following steps:

[0219] S701, obtaining a sample image and a sample requirement statement for the sample image, and determining a sample answer statement corresponding to the sample requirement statement.

[0220] In the embodiment of the present application, a sample image and a sample requirement statement for the sample image may be obtained. In addition, for the sample requirement statement, a sample answer statement corresponding to the sample requirement statement also needs to be determined.

[0221] It should be noted that the sample requirement statement may correspond to different tasks, such as image description tasks, general visual question answering tasks, visual answering tasks that require answer locations, etc. For different tasks, the format of the sample requirement statement is different. For example, for a visual answering task that requires answer locations, the format of the sample requirement statement is generally "{Question}. Provide the location coordinates of the answer when answering the question.".

[0222] In addition, for different tasks corresponding to the sample requirement statements, the preset data sets corresponding to the different tasks can be determined. The preset data sets contain sample images and the annotation statements corresponding to the sample images. Thus, the annotation statements of the sample images can be found from the preset data sets, and the annotation statements are determined as the sample answer statements corresponding to the sample requirement statements. In addition, for different tasks corresponding to the sample requirement statements, it is necessary to control the number of sample images in the preset data sets corresponding to the different tasks to ensure that different tasks can be balanced during the model training process.

[0223] It should be noted that for different tasks corresponding to sample requirement expressions, tasks can be divided into different scenarios according to the scenarios, and the corresponding preset data sets are different. For example, if the scenario corresponding to the task is a document scenario, then the corresponding preset data sets can be IIT-CDIP, DocVQA, ChartQA, InfoVQA, DeepForm, Kleister Charity and WikiTableQuestion datasets; if the scenario corresponding to the task is a text scenario, then the corresponding preset data sets can be COCOText, TextOCR, HierText, TextVQA and MLT datasets. In addition, in order to increase the utilization rate of sample images in the preset data sets, you can also manually add descriptions based on the original annotation descriptions for the visual features in the sample images.

[0224] It should be noted that for the visual answering task that requires answering the location, additional modifications have been made to the annotation expression, in which the answer to the location is added, and in order to ensure a close connection between the annotation expression and the answer to the location, the annotation expression is strictly kept before the answer to the location. In addition, in order to standardize images of different resolutions, a scale of (0,1000) is used to represent the location. Therefore, in an image with a resolution of H×W, the location is normalized to (x / H*1000, y / W*1000), where x represents the X-coordinate of the location and y represents the Y-axis coordinate of the location. The answer to the location given by the subsequent large model performs the inverse operation.

[0225] S702, performing block processing on the sample image to obtain a plurality of sample local image blocks, and adjusting the resolution of the sample image to obtain a sample target image.

[0226] In the embodiment of the present application, a block processing may be performed on the sample image to obtain a plurality of sample local image blocks, and the resolution of the sample image may be adjusted to obtain a sample target image.

[0227] Among them, the corresponding sample scaling resolution can be determined according to the resolution of the sample image, so as to facilitate the subsequent block cutting process, and such processing is to ensure that the sample image is not distorted during the scaling process. Based on this, the sample scaling resolution corresponding to the sample image is determined; the sample image is scaled so that the resolution of the scaled sample image reaches the sample scaling resolution; the scaled sample image is cut into blocks to obtain multiple sample local image blocks.

[0228] Among them, the determination of the sample scaling resolution corresponding to the sample image can be specifically: obtaining a predefined resolution set, the predefined resolution set includes multiple predefined resolutions, and each predefined resolution is a multiple of a preset resolution (for example, 448×448), for any predefined resolution in the predefined resolution set, determining the sample intersection-and-union ratio of the predefined resolution and the resolution of the sample image, and based on the sample intersection-and-union ratio, screening the sample predefined resolutions in the predefined resolution set, and determining the sample predefined resolution as the sample scaling resolution corresponding to the sample image.

[0229] It should be noted that for the sample intersection-and-union ratio, the largest sample intersection-and-union ratio can be selected, and then the predefined resolution corresponding to the largest sample intersection-and-union ratio can be found, and the predefined resolution corresponding to the largest sample intersection-and-union ratio can be used as the sample predefined resolution, which means that the predefined resolution corresponding to the largest intersection-and-union ratio can be used as the sample scaling resolution corresponding to the sample image.

[0230] S703: Input the multiple sample local image blocks into the first visual encoder for encoding processing to obtain multiple predicted local visual features.

[0231] In the embodiment of the present application, for a large model, it usually mainly includes a first visual encoder, a second visual encoder, a visual resampling model and a large language model. For the first visual encoder, a sliding window attention mechanism is included, which can link different local image blocks together for encoding processing, thereby effectively identifying and fusing the information corresponding to each local image block, enhancing the understanding of spatial and contextual relationships, and reducing the problems of semantic fragmentation and spatial fragmentation caused by block cutting.

[0232] Based on this, for multiple sample local image blocks, the multiple sample local image blocks are input into the first visual encoder for encoding processing to obtain multiple predicted local visual features. The first visual encoder adopts a sliding window attention mechanism to encode the multiple sample local image blocks and output multiple predicted local visual features.

[0233] S704, encode the sample demand statement to obtain sample text encoding features, input the sample target image into the second visual encoder to obtain predicted global visual features.

[0234] In an embodiment of the present application, for a sample requirement expression, the sample requirement expression may be encoded to obtain a sample text encoding feature.

[0235] In addition, for the second visual encoder, it reuses the visual encoder in the current large model, which means that for the second visual encoder, it does not include a sliding window attention mechanism. To this end, the sample target image is input into the second visual encoder to obtain the predicted global visual features.

[0236] S705 , inputting the plurality of predicted local visual features and the predicted global visual features into a visual resampling model to obtain a plurality of resampled predicted local visual features and a resampled predicted global visual features.

[0237] In an embodiment of the present application, for multiple predicted local visual features and predicted global visual features, the multiple predicted local visual features and predicted global visual features can be input into a visual resampling model to obtain multiple resampled predicted local visual features and resampled predicted global visual features.

[0238] It should be noted that for the visual resampling model, the number can be 1, which means that multiple predicted local visual features and predicted global visual features will be input into a common visual resampling model for visual reconstruction and compression to obtain multiple compressed resampled predicted local visual features and resampled predicted global visual features.

[0239] In addition, for the visual resampling model, an attention mechanism-based approach is adopted to compress and reconstruct multiple predicted local visual features and predicted global visual features by using a fixed number (e.g., 256) of query vectors to generate compressed multiple resampled predicted local visual features and resampled predicted global visual features. These fixed query vectors are a set of parameterized vectors learned during the training process and are used to perform weighted aggregation on each of the above visual features.

[0240] S706, input multiple resampled predicted local visual features, resampled predicted global visual features, and sample text encoding features into a large language model to obtain a predicted answer expression.

[0241] In an embodiment of the present application, the multiple resampled predicted local visual features, resampled predicted global visual features, and sample text encoding features obtained in the above steps can be input into a large language model to obtain a predicted answer expression.

[0242] In addition, the method of slicing sample images will cause semantic fragmentation, spatial fragmentation and other problems, resulting in inaccurate reasoning results when the large model processes related tasks. On the other hand, it will also cause a significant increase in feature length, affecting the reasoning speed of the large model. To this end, the embodiment of the present application proposes a dynamic compression mechanism to compress visual features, and a method of using the attention mechanism of a large language model to screen visual features to shorten the feature length and avoid affecting the reasoning speed of the large model.

[0243] Based on this, the above-mentioned inputting of multiple resampled predicted local visual features, resampled predicted global visual features, and sample text encoding features into a large language model to obtain a predicted answer expression can be specifically: selecting resampled predicted local visual features of a sample number from multiple resampled predicted local visual features; inputting the resampled predicted local visual features, resampled predicted global visual features, and sample text encoding features of a sample number into a large language model to obtain a predicted answer expression.

[0244] The number of samples is determined dynamically, and dynamic compression of the resampled predicted local visual features can be achieved. Based on this, sampling is performed from a preset compression probability set to obtain a sample compression probability, and a sample feature length corresponding to the sample scaling probability of the sample image is obtained; the number of samples is determined based on the sample compression probability and the sample feature length; and the resampled predicted local visual features of the sample number are selected from multiple resampled predicted local visual features.

[0245] It should be noted that for the sample compression probability and the sample feature length, the product of the two can be calculated, thereby obtaining the number of samples. In addition, a learnable parameter is defined in the large model, which represents the selection probability of the compression probability of the preset compression probability set. Through this parameter, sampling can be performed from the preset compression probability set to obtain the sample compression probability. For learnable parameters, in an embodiment provided in the present application, Gumbel-Softmax (a technique for making discrete selection processes differentiable in deep learning) can be used for training. Among them, Gumbel-Softmax can be controlled by a temperature parameter to make the sampling process differentiable, so that the learnable parameters can be directly trained in gradient descent.

[0246] In addition, for selecting the resampled predicted local visual features of the number of samples from multiple resampled predicted local visual features, it is generally based on importance, that is, among multiple resampled predicted local visual features, the most important resampled predicted local visual features of the number of samples are selected. To this end, for any resampled predicted local visual feature, the sample importance score of the resampled predicted local visual feature is determined; according to the sample importance score, the multiple resampled predicted local visual features are sorted to obtain the sorting result of the resampled predicted local visual features; according to the sorting result of the resampled predicted local visual features, the resampled predicted local visual features of the number of samples are selected.

[0247] The importance score is generally obtained by the similarity between features, and the similarity is generally calculated using, for example, cosine distance. Based on this, for any resampled predicted local visual feature, the sample similarity between the resampled predicted local visual feature and any resampled predicted local visual feature in the remaining resampled predicted local visual features is determined, so as to obtain multiple sample similarities, and the sample importance score of the resampled predicted local visual feature is determined based on the multiple sample similarities.

[0248] Specifically, for multiple sample similarities, they can be converted into sample importance scores. To this end, for any sample similarity, the sample difference between the preset value and the sample similarity is obtained, so that multiple sample differences can be obtained, and the obtained multiple sample differences are weighted and summed to obtain the sample importance score of the resampled predicted local visual features.

[0249] It should be noted that for the sample importance score, generally the smaller the sample importance score is, the more important the corresponding resampled predicted local visual feature is. Therefore, according to the sample importance score, multiple resampled predicted local visual features can be sorted from small to large to obtain the sorting result of the resampled sample local visual features, and then the top N resampled sample local visual features are selected, where N is the same as the number of samples.

[0250] In addition, for multiple resampled predicted local visual features, the resampled predicted local visual features of the sample number are selected from them, and there are still remaining resampled predicted local visual features. In order to avoid the information loss caused by directly discarding the remaining resampled predicted local visual features, important resampled predicted local visual features can be used as query vectors, and cross attention can be used to further aggregate all resampled predicted local visual features.

[0251] Based on this, the first sample initial feature order corresponding to each of the resampled predicted local visual features of the sample number is determined, and the resampled predicted local visual features of the sample number are sorted according to the first sample initial feature order to obtain a first sample local visual feature sequence; the second sample initial feature order corresponding to each of the remaining resampled predicted local visual features is determined, and the remaining resampled predicted local visual features are sorted according to the second sample initial feature order to obtain a second sample local visual feature sequence; the first sample local visual feature sequence and the second sample local visual feature sequence are aggregated using a cross-attention mechanism to obtain multiple predicted aggregated local visual features; the multiple predicted aggregated local visual features, the resampled predicted global visual features, and the sample text encoding features are input into a large language model to obtain a predicted answer expression.

[0252] Among them, multiple predicted aggregated local visual features, resampled predicted global visual features, and sample text encoding features are input into a large language model, and the large language model performs the following processing: based on the multiple predicted aggregated local visual features, resampled predicted global visual features, and sample text encoding features, a predicted attention distribution map of visual features is generated; based on the predicted attention distribution map, a preset proportion of sample visual features are selected from the multiple predicted aggregated local visual features and the resampled predicted global visual features; based on the preset proportion of sample visual features, a predicted answer statement is generated.

[0253] It should be noted that the first two layers of the large language model are responsible for generating a predicted attention distribution map of visual features based on multiple predicted aggregated local visual features, resampled predicted global visual features, and sample text encoding features. Then, through the predicted attention distribution map, multiple predicted aggregated local visual features and resampled predicted global visual features are screened to obtain sample visual features of a preset proportion. The remaining network of the large language model is responsible for generating a predicted answer statement based on the sample visual features of the preset proportion.

[0254] Among them, the predicted attention distribution map includes the predicted attention degree of the sample text encoding features to each predicted aggregated local visual feature and the resampled predicted global visual feature. Therefore, according to the predicted attention degree, multiple predicted aggregated local visual features and resampled predicted global visual features are sorted to obtain the sample visual feature sorting result; from the sample visual feature sorting result, a preset proportion of sample visual features are selected.

[0255] It should be noted that, for the predicted attention level, the higher the value of the predicted attention level, the more attention the sample text encoding feature pays to the corresponding predicted aggregated local visual features or resampled predicted global visual features. Therefore, according to the degree of attention, multiple predicted aggregated local visual features and resampled predicted global visual features can be sorted from large to small to obtain a sample visual feature sorting result, and the top 70% of the sample visual features are selected from the sample visual feature sorting result. The sample visual features include predicted aggregated local visual features and / or resampled predicted global visual features.

[0256] In addition, considering the continuity of visual features, a saving window can be set, and several features before and after the sample visual feature will be retained. For example, for the sample visual feature, 5 features before and after can be retained from multiple predicted aggregated local visual features and resampled predicted global visual features. Based on this, for any predicted visual feature, the sample adjacent visual features of the predicted visual feature are searched; based on the preset proportion of sample visual features and the sample adjacent visual features of each predicted visual feature, a predicted answer statement is generated.

[0257] S707, determining the expression loss between the predicted answer expression and the sample answer expression, and performing model training on the first visual encoder, the second visual encoder, the visual resampling model, and the large language model according to the expression loss.

[0258] S708, when the representation loss converges, stop model training and obtain a pre-trained first visual encoder, a pre-trained second visual encoder, a pre-trained visual resampling model, and a pre-trained large language model.

[0259] In an embodiment of the present application, the expression loss between the predicted answer expression and the sample answer expression can be determined, and the first visual encoder, the second visual encoder, the visual resampling model, and the large language model are trained according to the expression loss.

[0260] When the representation loss converges, the model training is stopped to obtain a pre-trained first visual encoder, a pre-trained second visual encoder, a pre-trained visual resampling model, and a pre-trained large language model.

[0261] It should be noted that during the model training process, the parameters in the second visual encoder can be frozen, while the parameters in the first visual encoder, the visual resampling model, and the large language model can be trained.

[0262] After the large model is trained in this way, the pre-trained first visual encoder in the large model uses a sliding window attention mechanism to encode multiple local image blocks, which can effectively alleviate the problems of semantic fragmentation and spatial fragmentation, thus ensuring good processing of high-resolution images. In addition, a dynamic compression mechanism is proposed to compress visual features, and the attention mechanism of a large language model is used to screen visual features to shorten the feature length and avoid affecting the reasoning speed of the large model.

[0263] Corresponding to the above method embodiment, the present application embodiment also provides an image processing device, such as Figure 8 As shown, the device may include: an image and description acquisition module 810 , an image cutting module 820 , an image adjustment module 830 , an image block encoding module 840 , and a description generation module 850 .

[0264] An image and description acquisition module 810 is used to acquire an original image and a required description for the original image;

[0265] An image cutting module 820 is used to cut the original image into pieces to obtain a plurality of local image blocks;

[0266] An image adjustment module 830, configured to adjust the resolution of the original image to obtain a target image, wherein the resolution of the target image is the same as the resolution of each of the local image blocks;

[0267] An image block encoding module 840 is used to adopt a cross-slice connection strategy to encode the plurality of local image blocks to obtain an encoding result;

[0268] The expression generation module 850 is used to process the encoding result, the target image and the requirement expression to obtain an answer expression corresponding to the requirement expression.

[0269] In an optional implementation, the image slicing module specifically includes:

[0270] A scaling resolution determination submodule, used to determine the scaling resolution corresponding to the original image;

[0271] An image scaling submodule, used for scaling the original image so that the resolution of the scaled original image reaches the scaling resolution;

[0272] The image cutting submodule is used to cut the scaled original image into blocks to obtain multiple local image blocks.

[0273] In an optional implementation, the scaling resolution determination submodule is specifically used to:

[0274] Acquire a predefined resolution set, wherein the predefined resolution set includes a plurality of predefined resolutions, and each of the predefined resolutions is a multiple of a preset resolution;

[0275] For any of the predefined resolutions in the predefined resolution set, determining an intersection-over-union ratio of the predefined resolution and a resolution of the original image;

[0276] According to the intersection-over-union ratio, a target predefined resolution in the predefined resolution set is screened, and the target predefined resolution is determined as the scaled resolution corresponding to the original image.

[0277] In an optional implementation, the image block encoding module is specifically used to:

[0278] Inputting the plurality of local image blocks into a pre-trained first visual encoder for encoding processing to obtain a plurality of local visual features;

[0279] Among them, the pre-trained first visual encoder adopts a sliding window attention mechanism to encode multiple local image blocks and output multiple local visual features.

[0280] In an optional implementation, the expression generation module specifically includes:

[0281] A statement encoding submodule is used to encode the requirement statement to obtain text encoding features;

[0282] A target image encoding submodule, used for inputting the target image into a pre-trained second visual encoder for encoding processing to obtain global visual features;

[0283] A feature resampling submodule, used for inputting the plurality of local visual features and the global visual features into a pre-trained visual resampling model to obtain a plurality of resampled local visual features and a resampled global visual feature;

[0284] The expression generation submodule is used to input the multiple resampled local visual features, the resampled global visual features, and the text encoding features into a pre-trained large language model to obtain an answer expression corresponding to the demand expression.

[0285] In an optional implementation, the expression generation submodule specifically includes:

[0286] A feature screening unit, configured to select a target number of the resampled local visual features from the plurality of the resampled local visual features;

[0287] The expression generation unit is used to input the target number of the resampled local visual features, the resampled global visual features, and the text encoding features into a pre-trained large language model to obtain an answer expression corresponding to the demand expression.

[0288] In an optional embodiment, the feature screening unit specifically includes:

[0289] A compression probability sampling subunit, used to sample from a preset compression probability set to obtain a target compression probability and obtain a characteristic length corresponding to the scaled resolution of the original image;

[0290] A target quantity determination subunit, used to determine the target quantity according to the target compression probability and the characteristic length;

[0291] The feature screening subunit is used to select the target number of the resampled local visual features from the multiple resampled local visual features.

[0292] In an optional embodiment, the feature screening subunit is specifically used for:

[0293] For any of the resampled local visual features, determining an importance score of the resampled local visual feature;

[0294] According to the importance scores, the plurality of resampled local visual features are sorted to obtain a resampled local visual feature sorting result;

[0295] According to the ranking result of the resampled local visual features, the target number of the resampled local visual features are selected.

[0296] In an optional embodiment, the feature screening subunit is specifically used for:

[0297] For any of the resampled local visual features, determining a similarity between the resampled local visual feature and any of the remaining resampled local visual features;

[0298] An importance score of the resampled local visual feature is determined based on the multiple similarities.

[0299] In an optional embodiment, the feature screening subunit is specifically used for:

[0300] For any of the similarities, obtaining a difference between a preset value and the similarity;

[0301] A weighted sum is performed on the obtained multiple differences to obtain an importance score of the resampled local visual feature.

[0302] In an optional implementation, the expression generating unit specifically includes:

[0303] A first visual feature sorting subunit is used to determine a first initial feature order corresponding to each of the target number of the resampled local visual features, and sort the target number of the resampled local visual features according to the first initial feature order to obtain a first local visual feature sequence;

[0304] A second visual feature sorting subunit is used to determine a second initial feature order corresponding to each of the remaining resampled local visual features, and sort the remaining resampled local visual features according to the second initial feature order to obtain a second local visual feature sequence;

[0305] a visual feature fusion subunit, configured to aggregate the first local visual feature sequence and the second local visual feature sequence to obtain a plurality of aggregated local visual features;

[0306] The expression generation subunit is used to input the multiple aggregated local visual features, the resampled global visual features, and the text encoding features into a pre-trained large language model to obtain an answer expression corresponding to the demand expression.

[0307] In an optional implementation, the expression generation subunit is specifically used for:

[0308] Input the plurality of aggregated local visual features, the resampled global visual features, and the text encoding features into a pre-trained large language model, and the pre-trained large language model performs the following processing:

[0309] Generate an attention distribution map of visual features according to the plurality of aggregated local visual features, the resampled global visual features, and the text encoding features;

[0310] According to the attention distribution map, selecting a preset proportion of target visual features from the plurality of aggregated local visual features and the resampled global visual features;

[0311] According to the target visual features of the preset proportion, an answer statement corresponding to the demand statement is generated.

[0312] In an optional implementation, the attention distribution map includes the degree of attention of the text encoding feature to each of the aggregated local visual features and the resampled global visual features; the representation generation subunit is specifically used for:

[0313] According to the attention degree, sorting the plurality of aggregated local visual features and the resampled global visual features to obtain a visual feature sorting result;

[0314] A preset proportion of target visual features is selected from the visual feature sorting results.

[0315] In an optional implementation, the expression generation subunit is specifically used for:

[0316] For any of the target visual features, searching for adjacent visual features of the target visual feature;

[0317] An answer statement corresponding to the demand statement is generated according to the preset ratio of target visual features and adjacent visual features of each of the target visual features.

[0318] In an optional embodiment, the device further comprises:

[0319] A sample image and sample requirement statement acquisition module, used to acquire a sample image and a sample requirement statement for the sample image, and determine a sample answer statement corresponding to the sample requirement statement;

[0320] A sample image cutting module is used to cut the sample image into blocks to obtain a plurality of sample local image blocks;

[0321] A sample image adjustment module, used to adjust the resolution of the sample image to obtain a sample target image;

[0322] A sample local image block encoding module, used for inputting a plurality of the sample local image blocks into a first visual encoder for encoding processing to obtain a plurality of predicted local visual features;

[0323] A sample target image encoding module is used to encode the sample requirement statement to obtain sample text encoding features, and input the sample target image into a second visual encoder to obtain predicted global visual features;

[0324] A visual feature resampling module, used for inputting the plurality of predicted local visual features and the predicted global visual features into a visual resampling model to obtain a plurality of resampled predicted local visual features and a resampled predicted global visual features;

[0325] A statement prediction module, used for inputting a plurality of the resampled predicted local visual features, the resampled predicted global visual features, and the sample text encoding features into a large language model to obtain a predicted answer statement;

[0326] A loss determination module, used to determine a statement loss between the predicted answer statement and the sample answer statement, and perform model training on the first visual encoder, the second visual encoder, the visual resampling model, and the large language model according to the statement loss;

[0327] The model training module is used to stop model training when the representation loss converges, and obtain a pre-trained first visual encoder, a pre-trained second visual encoder, a pre-trained visual resampling model, and a pre-trained large language model.

[0328] In an optional implementation, the sample image slicing module specifically includes:

[0329] A sample scaling resolution determination submodule, used to determine the sample scaling resolution corresponding to the sample image;

[0330] A sample image scaling submodule, used for scaling the sample image so that the resolution of the scaled sample image reaches the sample scaling resolution;

[0331] The sample image slicing submodule is used to slice the scaled sample image to obtain a plurality of sample local image blocks.

[0332] In an optional implementation, the sample scaling resolution determination submodule is specifically used to:

[0333] Acquire a predefined resolution set, wherein the predefined resolution set includes a plurality of predefined resolutions, and each of the predefined resolutions is a multiple of a preset resolution;

[0334] For any of the predefined resolutions in the predefined resolution set, determining a sample intersection-over-union ratio of the predefined resolution to a resolution of the sample image;

[0335] Filtering the sample predefined resolutions in the predefined resolution set according to the sample intersection-over-union ratio;

[0336] The sample predefined resolution is determined as the sample scaling resolution corresponding to the sample image.

[0337] In an optional implementation, the expression prediction module specifically includes:

[0338] A predicted visual feature screening submodule, used for selecting the resampled predicted local visual features of a sample number from a plurality of the resampled predicted local visual features;

[0339] The expression prediction submodule is used to input the resampled predicted local visual features, the resampled predicted global visual features, and the sample text encoding features of the sample quantity into a large language model to obtain a predicted answer expression.

[0340] In an optional embodiment, the predicted visual feature screening submodule specifically includes:

[0341] A compression probability sampling unit is used to sample from a preset compression probability set to obtain a sample compression probability and obtain a sample feature length corresponding to a sample scaling probability of a sample image;

[0342] A sample quantity determination unit, used to determine the sample quantity according to the sample compression probability and the sample characteristic length;

[0343] The predicted visual feature screening unit is used to select the resampled predicted local visual features of the sample quantity from the multiple resampled predicted local visual features.

[0344] In an optional embodiment, the predicted visual feature screening unit specifically includes:

[0345] A sample importance score determination subunit, used for determining the sample importance score of the resampled predicted local visual feature for any of the resampled predicted local visual features;

[0346] A predicted visual feature sorting subunit, used for sorting the plurality of resampled predicted local visual features according to the sample importance scores to obtain a resampled predicted local visual feature sorting result;

[0347] The predicted visual feature screening subunit is used to select the resampled predicted local visual features of the sample quantity according to the sorting result of the resampled predicted local visual features.

[0348] In an optional implementation, the sample importance score determination subunit is specifically used to:

[0349] For any of the resampled predicted local visual features, determining a sample similarity between the resampled predicted local visual feature and any of the remaining resampled predicted local visual features;

[0350] According to the plurality of sample similarities, a sample importance score of the resampled predicted local visual feature is determined.

[0351] In an optional implementation, the sample importance score determination subunit is specifically used to:

[0352] For any of the sample similarities, obtaining a sample difference between a preset value and the sample similarity;

[0353] A weighted sum is performed on the obtained multiple sample differences to obtain a sample importance score for the resampled predicted local visual feature.

[0354] In an optional implementation, the expression prediction submodule specifically includes:

[0355] A first predicted visual feature sorting unit is used to determine a first sample initial feature order corresponding to each of the resampled predicted local visual features of a sample number, and sort the resampled predicted local visual features of a sample number according to the first sample initial feature order to obtain a first sample local visual feature sequence;

[0356] A second predicted visual feature sorting unit is used to determine the second sample initial feature order corresponding to each of the remaining resampled predicted local visual features, and sort the remaining resampled predicted local visual features according to the second sample initial feature order to obtain a second sample local visual feature sequence;

[0357] A predicted visual feature aggregation unit, configured to aggregate the first sample local visual feature sequence and the second sample local visual feature sequence to obtain a plurality of predicted aggregated local visual features;

[0358] The expression prediction unit is used to input the plurality of predicted aggregated local visual features, the resampled predicted global visual features, and the sample text encoding features into a large language model to obtain a predicted answer expression.

[0359] In an optional implementation, the expression prediction unit specifically includes:

[0360] The feature input subunit is used to input the plurality of predicted aggregated local visual features, the resampled predicted global visual features, and the sample text encoding features into a large language model, and the large language model performs the following processing:

[0361] A distribution map generating subunit, configured to generate a predicted attention distribution map of visual features according to the plurality of predicted aggregated local visual features, the resampled predicted global visual features, and the sample text encoding features;

[0362] A feature screening subunit, configured to select a preset proportion of sample visual features from the plurality of predicted aggregated local visual features and the resampled predicted global visual features according to the predicted attention distribution map;

[0363] The expression generation subunit is used to generate a predicted answer expression based on the sample visual features of the preset proportion.

[0364] In an optional embodiment, the predicted attention distribution map includes the predicted attention degree of the sample text encoding feature to each of the predicted aggregated local visual features and the resampled predicted global visual features; the feature screening subunit is specifically used for:

[0365] According to the predicted attention degree, sorting the plurality of predicted aggregated local visual features and the resampled predicted global visual features to obtain a sample visual feature sorting result;

[0366] From the sample visual feature sorting results, sample visual features of a preset proportion are selected.

[0367] In an optional implementation, the expression generation subunit is specifically used for:

[0368] For any of the predicted visual features, searching for sample adjacent visual features of the predicted visual feature;

[0369] A predicted answer statement is generated based on the preset proportion of sample visual features and sample adjacent visual features of each predicted visual feature.

[0370] The present application also provides an electronic device, such as Fig. 9 As shown, it includes a processor 91, a communication interface 92, a memory 93 and a communication bus 94, wherein the processor 91, the communication interface 92, and the memory 93 communicate with each other through the communication bus 94.

[0371] Memory 93, used for storing computer programs;

[0372] The processor 91 is used to execute the program stored in the memory 93 to implement the following steps:

[0373] An original image and a requirement statement for the original image are obtained, and the original image is sliced ​​to obtain a plurality of local image blocks; the resolution of the original image is adjusted to obtain a target image, wherein the resolution of the target image is the same as the resolution of each of the local image blocks; a cross-slicing connection strategy is adopted to encode the plurality of local image blocks to obtain an encoding result; the encoding result, the target image and the requirement statement are processed to obtain an answer statement corresponding to the requirement statement.

[0374] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0375] The communication interface is used for communication between the above electronic device and other devices.

[0376] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0377] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0378] In another embodiment provided in the present application, a storage medium is provided. The storage medium stores instructions. When the instructions are executed on a computer, the computer executes the image processing method described in any one of the above embodiments.

[0379] In another embodiment provided in the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute the image processing method described in any one of the above embodiments.

[0380] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a storage medium, or transmitted from one storage medium to another storage medium, for example, the computer instructions may be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The storage medium may be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive Solid State Disk (SSD)), etc.

[0381] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0382] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0383] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire an original image and a requirement statement for the original image, and slice the original image to obtain a plurality of local image blocks; Adjusting the resolution of the original image to obtain a target image, wherein the resolution of the target image is the same as the resolution of each of the local image blocks; Adopting a cross-block connection strategy, encoding the plurality of local image blocks to obtain encoding results; The encoding result, the target image and the requirement statement are processed to obtain an answer statement corresponding to the requirement statement.

2. The method according to claim 1, characterized in that The block processing of the original image to obtain a plurality of local image blocks includes: Determine a scaling resolution corresponding to the original image, and perform scaling processing on the original image so that the resolution of the scaled original image reaches the scaling resolution; The scaled original image is cut into blocks to obtain a plurality of local image blocks.

3. The method according to claim 2, characterized in that The determining the scaling resolution corresponding to the original image includes: Acquire a predefined resolution set, wherein the predefined resolution set includes a plurality of predefined resolutions, and each of the predefined resolutions is a multiple of a preset resolution; For any of the predefined resolutions in the predefined resolution set, determining an intersection-over-union ratio of the predefined resolution and a resolution of the original image; According to the intersection-over-union ratio, a target predefined resolution in the predefined resolution set is screened, and the target predefined resolution is determined as the scaled resolution corresponding to the original image.

4. The method according to claim 1, characterized in that: The cross-slice connection strategy is adopted to encode the plurality of local image blocks to obtain encoding results, including: Inputting the plurality of local image blocks into a pre-trained first visual encoder for encoding processing to obtain a plurality of local visual features; Among them, the pre-trained first visual encoder adopts a sliding window attention mechanism to encode multiple local image blocks and output multiple local visual features.

5. The method according to claim 4, characterized in that Processing the encoding result, the target image, and the requirement statement to obtain an answer statement corresponding to the requirement statement includes: Encoding the requirement statement to obtain text encoding features; Inputting the target image into a pre-trained second visual encoder for encoding to obtain a global visual feature; Inputting the plurality of local visual features and the global visual features into a pre-trained visual resampling model to obtain a plurality of resampled local visual features and a resampled global visual features; Input the multiple resampled local visual features, the resampled global visual features, and the text encoding features into a pre-trained large-scale language model to obtain an answer statement corresponding to the demand statement.

6. The method according to claim 5, characterized in that The step of inputting the plurality of the resampled local visual features, the resampled global visual features, and the text encoding features into a pre-trained large language model to obtain an answer statement corresponding to the requirement statement includes: Selecting a target number of the resampled local visual features from the plurality of the resampled local visual features; The target number of the resampled local visual features, the resampled global visual features, and the text encoding features are input into a pre-trained large-scale language model to obtain an answer statement corresponding to the demand statement.

7. The method according to claim 6, characterized in that The step of selecting a target number of the resampled local visual features from the plurality of the resampled local visual features comprises: Sampling from a preset compression probability set to obtain a target compression probability, and obtaining a characteristic length corresponding to the scaled resolution of the original image; Determining the number of targets according to the target compression probability and the characteristic length; The target number of the resampled local visual features is selected from the plurality of the resampled local visual features.

8. The method according to claim 7, characterized in that The step of selecting the target number of the resampled local visual features from the plurality of the resampled local visual features comprises: For any of the resampled local visual features, determining an importance score of the resampled local visual feature; According to the importance scores, the plurality of resampled local visual features are sorted to obtain a resampled local visual feature sorting result; According to the ranking result of the resampled local visual features, the target number of the resampled local visual features are selected.

9. The method according to claim 8, characterized in that The step of determining the importance score of any of the resampled local visual features comprises: For any of the resampled local visual features, determining a similarity between the resampled local visual feature and any of the remaining resampled local visual features; An importance score of the resampled local visual feature is determined based on the multiple similarities.

10. The method according to claim 9, characterized in that Determining the importance score of the resampled local visual feature according to the multiple similarities includes: For any of the similarities, obtaining a difference between a preset value and the similarity; A weighted sum is performed on the obtained multiple differences to obtain an importance score of the resampled local visual feature.

11. The method according to claim 6, characterized in that The step of inputting the target number of the resampled local visual features, the resampled global visual features, and the text encoding features into a pre-trained large language model to obtain an answer statement corresponding to the requirement statement includes: Determine a first initial feature order corresponding to each of the target number of the resampled local visual features, and sort the target number of the resampled local visual features according to the first initial feature order to obtain a first local visual feature sequence; Determine a second initial feature order corresponding to each of the remaining resampled local visual features, and sort the remaining resampled local visual features according to the second initial feature order to obtain a second local visual feature sequence; Aggregating the first local visual feature sequence and the second local visual feature sequence to obtain a plurality of aggregated local visual features; The plurality of aggregated local visual features, the resampled global visual features, and the text encoding features are input into a pre-trained large-scale language model to obtain an answer statement corresponding to the demand statement.

12. The method according to claim 11, characterized in that The step of inputting the plurality of aggregated local visual features, the resampled global visual features, and the text encoding features into a pre-trained large language model to obtain an answer statement corresponding to the requirement statement includes: Input the plurality of aggregated local visual features, the resampled global visual features, and the text encoding features into a pre-trained large language model, and the pre-trained large language model performs the following processing: Generate an attention distribution map of visual features according to the plurality of aggregated local visual features, the resampled global visual features, and the text encoding features; According to the attention distribution map, selecting a preset proportion of target visual features from the plurality of aggregated local visual features and the resampled global visual features; According to the target visual features of the preset proportion, an answer statement corresponding to the demand statement is generated.

13. The method according to claim 12, characterized in that The attention distribution map includes the degree of attention paid by the text encoding feature to each of the aggregated local visual features and the resampled global visual features; The step of selecting a preset proportion of target visual features from the plurality of aggregated local visual features and the resampled global visual features according to the attention distribution map includes: According to the attention degree, sorting the plurality of aggregated local visual features and the resampled global visual features to obtain a visual feature sorting result; A preset proportion of target visual features is selected from the visual feature sorting results.

14. The method according to claim 12, characterized in that Generating an answer statement corresponding to the demand statement according to the target visual features of the preset proportion includes: For any of the target visual features, searching for adjacent visual features of the target visual feature; An answer statement corresponding to the demand statement is generated according to the preset ratio of target visual features and adjacent visual features of each of the target visual features.

15. The method according to claim 5, characterized in that Before executing the image processing method, the method further includes: Obtaining a sample image and a sample requirement statement for the sample image, and determining a sample answer statement corresponding to the sample requirement statement; Performing block processing on the sample image to obtain a plurality of sample local image blocks, and adjusting the resolution of the sample image to obtain a sample target image; Inputting the plurality of sample local image blocks into a first visual encoder for encoding processing to obtain a plurality of predicted local visual features; Encoding the sample demand statement to obtain sample text encoding features, and inputting the sample target image into a second visual encoder to obtain predicted global visual features; Inputting the plurality of predicted local visual features and the predicted global visual features into a visual resampling model to obtain a plurality of resampled predicted local visual features and a resampled predicted global visual features; Inputting a plurality of the resampled predicted local visual features, the resampled predicted global visual features, and the sample text encoding features into a large language model to obtain a predicted answer expression; Determine a representation loss between the predicted answer representation and the sample answer representation, and perform model training on the first visual encoder, the second visual encoder, the visual resampling model, and the large language model according to the representation loss; When the representation loss converges, the model training is stopped to obtain a pre-trained first visual encoder, a pre-trained second visual encoder, a pre-trained visual resampling model, and a pre-trained large language model.

16. An image processing device, characterized in that: The device comprises: An image and description acquisition module, used to acquire an original image and a required description for the original image; An image cutting module is used to cut the original image into pieces to obtain multiple local image blocks; An image adjustment module, used for adjusting the resolution of the original image to obtain a target image, wherein the resolution of the target image is the same as the resolution of each of the local image blocks; An image block encoding module, used for adopting a cross-slice connection strategy to encode the plurality of local image blocks to obtain an encoding result; The expression generation module is used to process the encoding result, the target image and the requirement expression to obtain an answer expression corresponding to the requirement expression.

17. A storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 15 is implemented.

Images