Image processing method and device

Abstract

The invention relates to an image processing method and device. The method comprises the following steps: acquiring an original image and a demand expression for the original image; the original image is cut into blocks, a plurality of local image blocks are obtained, and each local image block is a corresponding local area in the original image; the resolution of the original image is adjusted to obtain a target image, and the resolution of the target image is consistent with the resolution of each local image block; and processing the plurality of local image blocks, the target image and the demand expression to obtain an answer expression corresponding to the demand expression. According to the method, the capability of processing the high-resolution image can be improved by adopting the means of cutting the original image into blocks, and the high-resolution image can be well processed.

Description

Technical Field

[0001] The present application relates to the field of information processing technology, and in particular to an image processing method and device. Background Art

[0002] The big model is a large-scale deep learning model. Its core idea is to fuse different information (such as text, images, audio and video, etc.) and achieve more intelligent information processing by learning the relationship between different information. In the big model, different information is input into a big model after preprocessing, and finally outputs the corresponding results after multiple layers of feature extraction and fusion.

[0003] In the related technology, in the large model, for the input images, the images will be uniformly adjusted to a fixed resolution, and the size of this resolution is generally only 448×448. This will cause the resolution of some high-resolution images to be severely compressed when they are input into the large model, resulting in loss of visual information. Therefore, the large model faces challenges in processing high-resolution images and cannot process high-resolution images well. Summary of the invention

[0004] In order to solve the above problem, in the large model, for the input images, the images will be uniformly adjusted to a fixed resolution, and this resolution size is generally only 448×448, which will cause the resolution of some high-resolution images to be severely compressed when input into the large model, resulting in loss of visual information. Therefore, the large model faces challenges when processing high-resolution images and cannot process high-resolution images well. The embodiment of the present application provides an image processing method, device, electronic device and storage medium. The specific technical solution is as follows:

[0005] In a first aspect of an embodiment of the present application, an image processing method is first provided, the method comprising:

[0006] Acquire an original image and a requirement statement for the original image;

[0007] Performing block processing on the original image to obtain a plurality of local image blocks, wherein each of the local image blocks is a corresponding local area in the original image;

[0008] Adjusting the resolution of the original image to obtain a target image, wherein the resolution of the target image is consistent with the resolution of each of the local image blocks;

[0009] The plurality of local image blocks, the target image and the requirement statement are processed to obtain an answer statement corresponding to the requirement statement.

[0010] In an optional implementation, the slicing of the original image to obtain a plurality of local image blocks includes:

[0011] Determining whether the resolution of the original image is a preset resolution;

[0012] When the resolution of the original image is the preset resolution, the original image is sliced ​​to obtain a plurality of local image blocks.

[0013] In an optional embodiment, the method further comprises:

[0014] When the resolution of the original image is less than the preset resolution, adjusting the resolution of the original image to the preset resolution;

[0015] The original image after the resolution adjustment is cut into blocks to obtain a plurality of local image blocks.

[0016] In an optional embodiment, the method further comprises:

[0017] When the resolution of the original image is greater than the preset resolution, adjusting the resolution of the original image to the preset resolution;

[0018] The original image after the resolution adjustment is cut into blocks to obtain a plurality of local image blocks.

[0019] In an optional implementation, the processing of the plurality of local image blocks, the target image, and the requirement statement to obtain an answer statement corresponding to the requirement statement includes:

[0020] The plurality of local image blocks, the target image and the requirement statement are input into a pre-trained large model to obtain an answer statement corresponding to the requirement statement.

[0021] In an optional embodiment, the pre-trained large model includes a pre-trained feature extraction model and a pre-trained large language model;

[0022] The step of inputting the plurality of local image blocks, the target image, and the requirement statement into a pre-trained large model to obtain an answer statement corresponding to the requirement statement includes:

[0023] Inputting the plurality of local image blocks and the target image into the pre-trained feature extraction model to obtain a plurality of target local visual features and a target global visual feature;

[0024] Encoding the requirement statement to obtain text encoding features;

[0025] The text encoding feature, the target global visual feature and a plurality of the target local visual features are input into the pre-trained large language model to obtain an answer statement corresponding to the demand statement.

[0026] In an optional embodiment, the pre-trained feature extraction model includes a plurality of pre-trained visual fusion models, a pre-trained visual encoder and a pre-trained visual resampling model;

[0027] The step of inputting the plurality of local image blocks and the target image into the pre-trained feature extraction model to obtain a plurality of target local visual features and target global visual features includes:

[0028] For any of the local image blocks, input the local image block into the corresponding pre-trained visual fusion model to obtain the original local visual features corresponding to the local image block;

[0029] Inputting the original local visual features into the pre-trained visual resampling model to obtain the target local visual features corresponding to the local image block;

[0030] Inputting the target image into the pre-trained visual encoder to obtain original global visual features corresponding to the target image;

[0031] The original global visual features are input into the pre-trained visual resampling model to obtain target global visual features.

[0032] In an optional embodiment, the pre-trained visual fusion model includes a pre-trained visual encoder and a pre-trained visual bypass model;

[0033] Inputting the local image block into the corresponding pre-trained visual fusion model to obtain the original local visual features corresponding to the local image block includes:

[0034] Inputting the local image block into the corresponding pre-trained visual encoder to obtain a first local visual feature corresponding to the local image block;

[0035] Inputting the local image block into the corresponding pre-trained visual bypass model to obtain a second local visual feature corresponding to the local image block;

[0036] The first local visual feature and the second local visual feature are fused to obtain an original local visual feature corresponding to the local image block.

[0037] In an optional embodiment, before executing the method, the method further includes:

[0038] Obtaining a sample image and a sample requirement statement for the sample image, and determining a sample answer statement corresponding to the sample requirement statement;

[0039] Performing block processing on the sample image to obtain a plurality of sample local image blocks, wherein each of the sample local image blocks is a corresponding sample local area in the sample image;

[0040] Adjusting the resolution of the sample image to obtain a sample target image, wherein the resolution of the sample target image is consistent with the resolution of each of the sample local image blocks;

[0041] Inputting the plurality of sample local image blocks, the sample target image and the sample requirement statement into the large model to obtain a predicted answer statement corresponding to the sample requirement statement;

[0042] Determining a representation loss between the sample answer representation and the predicted answer representation, and training the large model based on the representation loss;

[0043] When the representation loss converges, the training is stopped to obtain a pre-trained large model.

[0044] In an optional implementation, the slicing of the sample image to obtain a plurality of sample local image blocks includes:

[0045] Determining whether the resolution of the sample image is a preset resolution;

[0046] When the resolution of the sample image is the preset resolution, the sample image is cut into blocks to obtain a plurality of sample local image blocks.

[0047] In an optional embodiment, the method further comprises:

[0048] When the resolution of the sample image is less than the preset resolution, adjusting the resolution of the sample image to the preset resolution;

[0049] The sample image after the resolution adjustment is cut into blocks to obtain a plurality of sample local image blocks.

[0050] In an optional embodiment, the method further comprises:

[0051] When the resolution of the sample image is greater than the preset resolution, adjusting the resolution of the sample image to the preset resolution;

[0052] The sample image after the resolution adjustment is cut into blocks to obtain a plurality of sample local image blocks.

[0053] In an optional embodiment, the large model includes a feature extraction model and a large language model;

[0054] The step of inputting the plurality of sample local image blocks, the sample target image and the sample requirement statement into the large model to obtain a predicted answer statement corresponding to the sample requirement statement includes:

[0055] Inputting the plurality of sample local image blocks and the sample target image into the feature extraction model to obtain a plurality of predicted target local visual features and predicted target global visual features;

[0056] Encoding the sample requirement statement to obtain sample text encoding features;

[0057] The sample text encoding features, the prediction target global visual features and a plurality of the prediction target local visual features are input into the large language model to obtain a prediction answer statement corresponding to the sample demand statement.

[0058] In an optional embodiment, the feature extraction model includes a plurality of visual fusion models, a visual encoder and a visual resampling model;

[0059] The step of inputting the plurality of sample local image blocks and the sample target image into the feature extraction model to obtain a plurality of predicted target local visual features and predicted target global visual features includes:

[0060] For any of the sample local image blocks, input the sample local image block into the corresponding visual fusion model to obtain the predicted original local visual features corresponding to the sample local image block;

[0061] Inputting the predicted original local visual features into the visual resampling model to obtain the predicted target local visual features corresponding to the sample local image block;

[0062] Inputting the sample target image into the visual encoder to obtain the predicted original global visual features corresponding to the sample target image;

[0063] The predicted original global visual features are input into the visual resampling model to obtain the predicted target global visual features corresponding to the sample target image.

[0064] In an optional embodiment, the visual fusion model includes a visual encoder and a visual bypass model;

[0065] The step of inputting the sample local image block into the corresponding visual fusion model to obtain the predicted original local visual features corresponding to the sample local image block includes:

[0066] Inputting the sample local image block into the corresponding visual encoder to obtain a predicted first local visual feature corresponding to the sample local image block;

[0067] Inputting the sample local image block into the corresponding visual bypass model to obtain a predicted second local visual feature corresponding to the sample local image block;

[0068] The predicted first local visual feature and the predicted second local visual feature are feature-fused to obtain the predicted original local visual feature corresponding to the sample local image block.

[0069] In an optional implementation, the determining the sample answer statement corresponding to the sample requirement statement includes:

[0070] Determine a task corresponding to the sample requirement statement, and if the task is not a target task, determine a preset data set corresponding to the task, wherein the preset data set includes the sample image;

[0071] From the preset data set, the annotated expression of the sample image is searched, and the annotated expression is determined as the sample answer expression corresponding to the sample requirement expression.

[0072] In an optional implementation, the determining the sample answer statement corresponding to the sample requirement statement includes:

[0073] Determine the task corresponding to the sample requirement statement, and if the task is a target task, perform the following processing:

[0074] Determine a preset data set corresponding to the task, the preset data set including the sample image;

[0075] Searching for the annotated expression of the sample image from the preset data set, and determining the annotated expression as the sample answer expression corresponding to the sample requirement expression;

[0076] or,

[0077] A detailed description statement of the sample image is generated, and the detailed description statement is determined as a sample answer statement corresponding to the sample requirement statement.

[0078] In an optional implementation, the generating a detailed description of the sample image includes:

[0079] Inputting the sample image into a pre-trained global description generation model to obtain a global description expression corresponding to the sample image;

[0080] Inputting the sample image into a pre-trained region description generation model to obtain a specific region in the sample image and a region description expression of the specific region;

[0081] Inputting the sample image into a pre-trained text detection model to obtain a text region in the sample image and text in the text region;

[0082] Inputting the sample image into a pre-trained object recognition model to obtain an object region in the sample image;

[0083] Inputting the object region in the sample image into a pre-trained object description generation model to obtain an object description expression corresponding to the object region in the sample image;

[0084] A detailed description statement of the sample image is generated according to the global description statement, the specific area, the area description statement, the text area, the text, the object area and the object description statement.

[0085] In an optional implementation, generating a detailed description of the sample image according to the global description, the specific area, the area description, the text area, the text, the object area and the object description includes:

[0086] Inputting the specific area and the area description into a pre-trained matching model to obtain an area matching score between the specific area and the area description;

[0087] According to the area matching score, the specific area and the area description are filtered to obtain a target specific area and a target area description;

[0088] Inputting the text region and the text into a pre-trained matching model to obtain a text matching score between the text region and the text;

[0089] Filtering the text area and the text according to the text matching score to obtain a target text area and a target text;

[0090] Inputting the object region and the object description into a pre-trained matching model to obtain an object matching score between the object region and the object description;

[0091] According to the object matching score, the object region and the object description are filtered to obtain a target object region and a target object description;

[0092] A detailed description statement of the sample image is generated according to the global description statement, the target area description, the target text and the target object description statement.

[0093] In an optional implementation, generating a detailed description of the sample image according to the global description, the target area description, the target text, and the target object description includes:

[0094] The global description, the target area description, the target text and the target object description are input into a pre-trained large language model to obtain a detailed description of the sample image.

[0095] In a second aspect of the embodiments of the present application, an image processing device is further provided, the device comprising:

[0096] An information acquisition module, used to acquire an original image and a requirement statement for the original image;

[0097] An image slicing module, used for slicing the original image to obtain a plurality of local image blocks, wherein each local image block is a corresponding local area in the original image;

[0098] A resolution adjustment module, used to adjust the resolution of the original image to obtain a target image, wherein the resolution of the target image is consistent with the resolution of each of the local image blocks;

[0099] The expression generation module is used to process the multiple local image blocks, the target image and the demand expression to obtain an answer expression corresponding to the demand expression.

[0100] In an optional implementation, the image slicing module is specifically used to:

[0101] Determining whether the resolution of the original image is a preset resolution;

[0102] When the resolution of the original image is the preset resolution, the original image is sliced ​​to obtain a plurality of local image blocks.

[0103] In an optional implementation, the image slicing module is further configured to:

[0104] When the resolution of the original image is less than the preset resolution, adjusting the resolution of the original image to the preset resolution;

[0105] The original image after the resolution adjustment is cut into blocks to obtain a plurality of local image blocks.

[0106] In an optional implementation, the image slicing module is further configured to:

[0107] When the resolution of the original image is greater than the preset resolution, adjusting the resolution of the original image to the preset resolution;

[0108] The original image after the resolution adjustment is cut into blocks to obtain a plurality of local image blocks.

[0109] In an optional implementation, the expression generation module is specifically used to:

[0110] The plurality of local image blocks, the target image and the requirement statement are input into a pre-trained large model to obtain an answer statement corresponding to the requirement statement.

[0111] In an optional embodiment, the pre-trained large model includes a pre-trained feature extraction model and a pre-trained large language model; the expression generation module specifically includes:

[0112] An image processing submodule, used for inputting the plurality of local image blocks and the target image into the pre-trained feature extraction model to obtain a plurality of target local visual features and a target global visual feature;

[0113] A statement encoding submodule is used to encode the requirement statement to obtain text encoding features;

[0114] The expression generation submodule is used to input the text encoding features, the target global visual features and multiple target local visual features into the pre-trained large language model to obtain the answer expression corresponding to the demand expression.

[0115] In an optional embodiment, the pre-trained feature extraction model includes a plurality of pre-trained visual fusion models, a pre-trained visual encoder and a pre-trained visual resampling model; the image processing submodule specifically includes:

[0116] An image block input unit, used for inputting any of the local image blocks into the corresponding pre-trained visual fusion model to obtain an original local visual feature corresponding to the local image block;

[0117] An original local visual feature input unit, used for inputting the original local visual feature into the pre-trained visual resampling model to obtain a target local visual feature corresponding to the local image block;

[0118] A target image input unit, used to input the target image into the pre-trained visual encoder to obtain the original global visual features corresponding to the target image;

[0119] The original global visual feature input unit is used to input the original global visual feature into the pre-trained visual resampling model to obtain the target global visual feature.

[0120] In an optional embodiment, the pre-trained visual fusion model includes a pre-trained visual encoder and a pre-trained visual bypass model; and the image block input unit is specifically used for:

[0121] Inputting the local image block into the corresponding pre-trained visual encoder to obtain a first local visual feature corresponding to the local image block;

[0122] Inputting the local image block into the corresponding pre-trained visual bypass model to obtain a second local visual feature corresponding to the local image block;

[0123] The first local visual feature and the second local visual feature are fused to obtain an original local visual feature corresponding to the local image block.

[0124] In an optional embodiment, the device further comprises:

[0125] A representation and image acquisition module, used to acquire sample images and sample requirement representations for the sample images;

[0126] A statement determination module, used to determine a sample answer statement corresponding to the sample requirement statement;

[0127] A sample image slicing module, used for slicing the sample image to obtain a plurality of sample local image blocks, wherein each of the sample local image blocks is a corresponding sample local area in the sample image;

[0128] A sample image adjustment module, used for adjusting the resolution of the sample image to obtain a sample target image, wherein the resolution of the sample target image is consistent with the resolution of each of the sample local image blocks;

[0129] An image and statement input module, used to input a plurality of the sample local image blocks, the sample target image and the sample demand statement into the large model to obtain a predicted answer statement corresponding to the sample demand statement;

[0130] A loss determination module, configured to determine a statement loss between the sample answer statement and the predicted answer statement;

[0131] The model training module is used to train the large model according to the representation loss, and stop the training when the representation loss converges to obtain the pre-trained large model.

[0132] In an optional implementation, the sample image slicing module is specifically used to:

[0133] Determining whether the resolution of the sample image is a preset resolution;

[0134] When the resolution of the sample image is the preset resolution, the sample image is cut into blocks to obtain a plurality of sample local image blocks.

[0135] In an optional implementation, the sample image slicing module is further configured to:

[0136] When the resolution of the sample image is less than the preset resolution, adjusting the resolution of the sample image to the preset resolution;

[0137] The sample image after the resolution adjustment is cut into blocks to obtain a plurality of sample local image blocks.

[0138] In an optional implementation, the sample image slicing module is further configured to:

[0139] When the resolution of the sample image is greater than the preset resolution, adjusting the resolution of the sample image to the preset resolution;

[0140] The sample image after the resolution adjustment is cut into blocks to obtain a plurality of sample local image blocks.

[0141] In an optional implementation, the large model includes a feature extraction model and a large language model; the image and expression input module specifically includes:

[0142] An image input submodule, used for inputting the plurality of sample local image blocks and the sample target image into the feature extraction model to obtain a plurality of predicted target local visual features and predicted target global visual features;

[0143] A sample requirement expression encoding submodule is used to encode the sample requirement expression to obtain sample text encoding features;

[0144] The feature input submodule is used to input the sample text encoding feature, the prediction target global visual feature and multiple prediction target local visual features into the large language model to obtain the predicted answer statement corresponding to the sample demand statement.

[0145] In an optional embodiment, the feature extraction model includes multiple visual fusion models, a visual encoder and a visual resampling model; the image input submodule specifically includes:

[0146] A sample local image block input unit, used for inputting any of the sample local image blocks into the corresponding visual fusion model to obtain a predicted original local visual feature corresponding to the sample local image block;

[0147] A predicted original local visual feature input unit, used for inputting the predicted original local visual feature into the visual resampling model to obtain a predicted target local visual feature corresponding to the sample local image block;

[0148] A sample target image input unit, used to input the sample target image into the visual encoder to obtain the predicted original global visual features corresponding to the sample target image;

[0149] The predicted original global visual feature input unit is used to input the predicted original global visual feature into the visual resampling model to obtain the predicted target global visual feature corresponding to the sample target image.

[0150] In an optional implementation, the visual fusion model includes a visual encoder and a visual bypass model; the sample local image block input unit is specifically used for:

[0151] Inputting the sample local image block into the corresponding visual encoder to obtain a predicted first local visual feature corresponding to the sample local image block;

[0152] Inputting the sample local image block into the corresponding visual bypass model to obtain a predicted second local visual feature corresponding to the sample local image block;

[0153] The predicted first local visual feature and the predicted second local visual feature are feature fused to obtain the predicted original local visual feature corresponding to the sample local image block.

[0154] In an optional implementation, the expression determination module is specifically used to:

[0155] Determine a task corresponding to the sample requirement statement, and if the task is not a target task, determine a preset data set corresponding to the task, wherein the preset data set includes the sample image;

[0156] From the preset data set, the annotated expression of the sample image is searched, and the annotated expression is determined as the sample answer expression corresponding to the sample requirement expression.

[0157] In an optional implementation, the expression determination module specifically includes:

[0158] The task determination submodule is used to determine the task corresponding to the sample requirement statement, and when the task is a target task, perform the following processing:

[0159] A data set determination submodule, used to determine a preset data set corresponding to the task, wherein the preset data set includes the sample image;

[0160] A first submodule for determining a statement is used to search for a labeled statement of the sample image from the preset data set, and determine the labeled statement as a sample answer statement corresponding to the sample requirement statement;

[0161] or,

[0162] A detailed description generation submodule, used to generate a detailed description of the sample image;

[0163] The second submodule for determining the expression is used to determine the detailed description expression as the sample answer expression corresponding to the sample requirement expression.

[0164] In an optional implementation, the detailed description statement generation submodule specifically includes:

[0165] A global description generation unit, used for inputting the sample image into a pre-trained global description generation model to obtain a global description corresponding to the sample image;

[0166] A region description generation unit, used for inputting the sample image into a pre-trained region description generation model to obtain a specific region in the sample image and a region description of the specific region;

[0167] A text generation unit, used for inputting the sample image into a pre-trained text detection model to obtain a text region in the sample image and text in the text region;

[0168] An object region generating unit, used for inputting the sample image into a pre-trained object recognition model to obtain an object region in the sample image;

[0169] An object description generation unit, configured to input the object region in the sample image into a pre-trained object description generation model to obtain an object description corresponding to the object region in the sample image;

[0170] A detailed description statement generating unit is used to generate a detailed description statement of the sample image according to the global description statement, the specific area, the area description statement, the text area, the text, the object area and the object description statement.

[0171] In an optional implementation, the detailed description statement generating unit specifically includes:

[0172] A region matching score generating subunit, configured to input the specific region and the region description into a pre-trained matching model to obtain a region matching score between the specific region and the region description;

[0173] A region description filtering subunit, configured to filter the specific region and the region description according to the region matching score to obtain a target specific region and a target region description;

[0174] A text matching score generating subunit, used for inputting the text region and the text into a pre-trained matching model to obtain a text matching score between the text region and the text;

[0175] A text filtering subunit, used for filtering the text area and the text according to the text matching score to obtain a target text area and a target text;

[0176] an object matching score generating subunit, configured to input the object region and the object description into a pre-trained matching model to obtain an object matching score between the object region and the object description;

[0177] An object description filtering unit, configured to filter the object region and the object description according to the object matching score to obtain a target object region and a target object description;

[0178] The detailed description statement generating subunit is used to generate a detailed description statement of the sample image according to the global description statement, the target area description, the target text and the target object description statement.

[0179] In an optional implementation, the detailed description statement generating subunit is specifically used for:

[0180] The global description, the target area description, the target text and the target object description are input into a pre-trained large language model to obtain a detailed description of the sample image.

[0181] In a third aspect of the embodiments of the present application, there is further provided an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0182] Memory, used to store computer programs;

[0183] The processor is used to implement any image processing method described in the first aspect when executing the program stored in the memory.

[0184] In a fourth aspect of the embodiments of the present application, a storage medium is further provided, wherein instructions are stored in the storage medium, and when the storage medium is run on a computer, the computer executes the image processing method described in any one of the first aspects above.

[0185] In a fifth aspect of the embodiments of the present application, a computer program product comprising instructions is also provided, which, when executed on a computer, enables the computer to execute any of the above-mentioned image processing methods.

[0186] The technical solution provided by the embodiment of the present application obtains an original image and a demand statement for the original image, slices the original image to obtain multiple local image blocks, wherein each local image block is a corresponding local area in the original image, adjusts the resolution of the original image to obtain a target image, wherein the resolution of the target image is consistent with the resolution of each local image block, processes the multiple local image blocks, the target image and the demand statement, and obtains an answer statement corresponding to the demand statement.

[0187] By slicing the original image to obtain multiple local image blocks, the resolution of the original image is adjusted to obtain a target image, and the multiple local image blocks, the target image and the demand statement for the original image are processed to obtain an answer statement corresponding to the demand statement. In this way, slicing the original image can expand the ability to process high-resolution images and achieve good processing of high-resolution images. BRIEF DESCRIPTION OF THE DRAWINGS

[0188] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0189] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0190] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0191] Figure 1 A schematic diagram of an implementation flow of an image processing method shown in an embodiment of the present application;

[0192] Figure 2It is a schematic diagram of an implementation flow of another image processing method shown in an embodiment of the present application;

[0193] Figure 3 A schematic diagram of the structure of a pre-trained large model shown in an embodiment of the present application;

[0194] Figure 4 A schematic diagram of the structure of another pre-trained large model shown in an embodiment of the present application;

[0195] Figure 5 A schematic diagram of an implementation flow of a model training method shown in an embodiment of the present application;

[0196] Figure 6 A schematic diagram of an implementation flow of a detailed description statement generation method shown in an embodiment of the present application;

[0197] Figure 7 A schematic diagram of the structure of an image processing device shown in an embodiment of the present application;

[0198] Figure 8 It is a schematic diagram of the structure of an electronic device shown in an embodiment of the present application. DETAILED DESCRIPTION

[0199] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0200] The disclosure below provides many different embodiments or examples to realize the different structures of the present application. In order to simplify the disclosure of the present application, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present application. In addition, the present application can repeat reference numbers and / or letters in different examples. This repetition is for the purpose of simplification and clarity, and does not itself indicate the relationship between the various embodiments and / or settings discussed.

[0201] like Figure 1 FIG. 1 is a schematic diagram of an implementation flow of an image processing method provided in an embodiment of the present application. The method is applied to an electronic device and may specifically include the following steps:

[0202] S101, obtaining an original image and a requirement statement for the original image.

[0203] In an embodiment of the present application, an original image and a demand statement for the original image may be obtained, where the demand statement is a demand statement described in natural language (which may be understood as a question raised by a user).

[0204] For example, an original image and a requirement statement for the original image are obtained. The requirement statement may be a requirement statement described in natural language, that is, "please summarize what this image is intended to express."

[0205] S102, performing block processing on the original image to obtain a plurality of local image blocks, wherein each local image block is a corresponding local area in the original image.

[0206] In an embodiment of the present application, for an original image, the original image can be cut into blocks to obtain multiple local image blocks, wherein each local image block is a corresponding local area in the original image, which means that the corresponding local area is cut into blocks to obtain a local image block.

[0207] It should be noted that, for multiple local image blocks, their resolutions are the same, which means that the resolution of each local image block is a certain resolution, for example, the resolution of each local image block is 448×448, and the embodiment of the present application does not limit this.

[0208] For example, for the original image, the original image can be cut into blocks to obtain 6 local image blocks, namely local image block 1, local image block 2, local image block 3, local image block 4, local image block 5, and local image block 6. The resolution of these 6 local image blocks is 448×448.

[0209] S103, adjusting the resolution of the original image to obtain a target image, wherein the resolution of the target image is consistent with the resolution of each local image block.

[0210] In the embodiment of the present application, for the original image, the resolution of the original image can be adjusted to obtain the target image, wherein the resolution of the target image is consistent with the resolution of each local image block, which means that the resolution of the original image is adjusted to the resolution consistent with the local image block.

[0211] For example, for an original image, the resolution of the original image can be adjusted to obtain a target image, the resolution of the target image is 448×448, and the resolution of the target image is consistent with the resolution of each local image block, that is, for the original image, its resolution is adjusted to a resolution of 448×448 that is consistent with the local image block.

[0212] S104, processing the multiple local image blocks, the target image and the requirement statement to obtain an answer statement corresponding to the requirement statement.

[0213] In an embodiment of the present application, multiple local image blocks, target images and demand statements can be processed to obtain answer statements corresponding to the demand statements (which can be understood as answers to questions raised by users).

[0214] Thus, the method of slicing the original image can expand the ability to process high-resolution images, and can achieve good processing of high-resolution images. In addition, the method of slicing the original image can also expand the ability to process complex scene images, and can achieve good processing of complex scene images.

[0215] Through the above description of the technical solution provided in the embodiment of the present application, an original image and a demand statement for the original image are obtained, the original image is cut into blocks to obtain multiple local image blocks, wherein each local image block is a corresponding local area in the original image, the resolution of the original image is adjusted to obtain a target image, wherein the resolution of the target image is consistent with the resolution of each local image block, the multiple local image blocks, the target image and the demand statement are processed to obtain an answer statement corresponding to the demand statement.

[0216] By slicing the original image to obtain multiple local image blocks, adjusting the resolution of the original image to obtain the target image, processing the multiple local image blocks, the target image, and the demand expression for the original image to obtain the answer expression corresponding to the demand expression, the method of slicing the original image can expand the ability to process high-resolution images and achieve good processing of high-resolution images. In addition, the method of slicing the original image can also expand the ability to process complex scene images and achieve good processing of complex scene images.

[0217] In addition, in the embodiment of the present application, the method of slicing the original image is essentially to expand the ability of the pre-trained large model to process high-resolution images and complex scene images, and can achieve good processing of high-resolution images and complex scene images. Figure 2 FIG. 1 is a schematic diagram of an implementation flow of another image processing method provided in an embodiment of the present application. The method is applied to an electronic device and may specifically include the following steps:

[0218] S201, obtaining an original image and a requirement statement for the original image.

[0219] In the embodiment of the present application, this step is similar to the above-mentioned step S101, and the embodiment of the present application will not be described one by one here.

[0220] S202: Determine whether the resolution of the original image is a preset resolution.

[0221] In the embodiment of the present application, the resolution of the original image may be of various types, so the resolution of the original image needs to be adjusted to a uniform resolution in preparation for subsequent image slicing processing.

[0222] Before adjusting the resolution of the original image, it is necessary to determine whether the resolution of the original image is a preset resolution. If it is the preset resolution, the image segmentation processing requirement is met and no further resolution adjustment is required.

[0223] It should be noted that the preset resolution, for example, 896×1344, can be adjusted according to actual needs. The preset resolution determines the number of local image blocks obtained by slicing the original image. As the preset resolution increases, the number of local image blocks obtained by slicing the original image will also increase, and vice versa.

[0224] S203: When the resolution of the original image is a preset resolution, the original image is sliced ​​to obtain a plurality of local image blocks.

[0225] In the embodiment of the present application, for the original image, when the resolution of the original image is a preset resolution, it means that there is no need to adjust the resolution of the original image. At this time, the original image can be directly cut into blocks to obtain multiple local image blocks.

[0226] For example, for the original image, when the resolution of the original image is 896×1344, the original image can be directly cut into blocks to obtain 6 local image blocks, namely local image block 1, local image block 2, local image block 3, local image block 4, local image block 5, and local image block 6.

[0227] S204: When the resolution of the original image is less than the preset resolution, adjust the resolution of the original image to the preset resolution.

[0228] S205, performing block processing on the original image after the resolution adjustment to obtain a plurality of local image blocks.

[0229] In the embodiment of the present application, for the original image, when the resolution of the original image is less than the preset resolution, it indicates that the resolution of the original image needs to be adjusted, thereby adjusting the resolution of the original image to the preset resolution.

[0230] It should be noted that the resolution of the original image can be adjusted to a preset resolution by interpolation. Of course, other methods are also possible, such as filling with preset pixel values ​​(such as 0 or 255). The embodiments of the present application are not limited to this.

[0231] In this way, the resolution of the original image is adjusted to reach a preset resolution, for example, 896×1344. At this time, the original image with the adjusted resolution can be cut into blocks to obtain multiple local image blocks.

[0232] S206: When the resolution of the original image is greater than the preset resolution, adjust the resolution of the original image to the preset resolution.

[0233] S207, performing block processing on the original image after the resolution adjustment to obtain a plurality of local image blocks, wherein each local image block is a corresponding local area in the original image.

[0234] In the embodiment of the present application, for the original image, when the resolution of the original image is greater than the preset resolution, it indicates that the resolution of the original image needs to be adjusted, thereby adjusting the resolution of the original image to the preset resolution.

[0235] It should be noted that, for the resolution of the original image, the size of the original image can be adjusted through corresponding image editing software to achieve the purpose of adjusting the resolution of the original image, so that the resolution of the original image reaches the preset resolution.

[0236] In this way, the resolution of the original image is adjusted to reach a preset resolution, for example, 896×1344. At this time, the original image with the adjusted resolution can be cut into blocks to obtain multiple local image blocks.

[0237] S208, adjusting the resolution of the original image to obtain a target image, wherein the resolution of the target image is consistent with the resolution of each local image block.

[0238] In the embodiment of the present application, this step is similar to the above-mentioned step S103, and the embodiment of the present application will not be described one by one here.

[0239] S209, inputting multiple local image blocks, target images and demand statements into the pre-trained large model to obtain answer statements corresponding to the demand statements.

[0240] In an embodiment of the present application, multiple local image blocks, target images and demand statements can be processed by a pre-trained large model, so that multiple local image blocks, target images and demand statements can be input into the pre-trained large model to obtain answer statements corresponding to the demand statements.

[0241] Among them, the pre-trained large model usually includes a pre-trained feature extraction model and a pre-trained large language model. The pre-trained large language model can be any current large language model. Thus, multiple local image blocks and target images are input into the pre-trained feature extraction model to obtain multiple target local visual features and target global visual features, and the demand expression is encoded to obtain text encoding features. The text encoding features, target global visual features and multiple target local visual features are input into the pre-trained large language model to obtain the answer expression corresponding to the demand expression.

[0242] For example, Figure 3 As shown, local image block 1, local image block 2, local image block 3, local image block 4, local image block 5, local image block 6, and the target image are input into a pre-trained feature extraction model to obtain multiple target local visual features and target global visual features, and the demand statement ("Please summarize what this image wants to express") is encoded (which can be understood as vectorization processing) to obtain text encoding features, and the text encoding features, target global visual features, and multiple target local visual features are input into a pre-trained large language model to obtain the answer statement corresponding to the demand statement ("A man is using a camera to shoot the distance").

[0243] Among them, the pre-trained feature extraction model usually includes multiple pre-trained visual fusion models, pre-trained visual encoders and pre-trained visual resampling models, and the number of pre-trained visual fusion models is consistent with the number of local image blocks, which means that each local image block has a corresponding pre-trained visual fusion model.

[0244] Therefore, for any local image block, the local image block is input into the corresponding pre-trained visual fusion model to obtain the original local visual features corresponding to the local image block; the original local visual features are input into the pre-trained visual resampling model to obtain the target local visual features corresponding to the local image block; the target image is input into the pre-trained visual encoder to obtain the original global visual features corresponding to the target image; the original global visual features are input into the pre-trained visual resampling model to obtain the target global visual features.

[0245] For example, Figure 4As shown, for local image block 1, local image block 2, local image block 3, local image block 4, local image block 5, and local image block 6, for local image block 1, the local image block 1 is input into the corresponding pre-trained visual fusion model 1 to obtain the original local visual features corresponding to the local image block 1, and the original local visual features are input into the pre-trained visual resampling model to obtain the target local visual features 1 corresponding to the local image blocks, and the processing of the remaining local image blocks is similar; for the target image, the target image is input into the pre-trained visual encoder to obtain the original global visual features corresponding to the target image, and the original global visual features are input into the pre-trained visual resampling model to obtain the target global visual features.

[0246] It should be noted that the number of pre-trained visual resampling models can be 1, which means that the original local visual features extracted by the pre-trained visual fusion model and the original global visual features extracted by the pre-trained visual encoder will be input into a common pre-trained visual resampling model for visual resampling. This can extract special and important target local visual features and target global visual features, and better grasp the various details in the original image. In addition, using the same pre-trained visual resampling model can effectively improve the overall reasoning speed of the model, so that the pre-trained large model can have a better user experience.

[0247] In addition, the pre-trained visual fusion model usually includes a pre-trained visual encoder (consistent with the above-mentioned pre-trained visual encoder, that is, multiple pre-trained visual encoders are reused, and the pre-trained visual encoders have a pre-trained visual bypass model added on their basis) and a pre-trained visual bypass model. The pre-trained visual bypass model can be implemented through LoRA (Low-Rank Adaptation of Large Language Models), which means that on the basis of the original pre-trained visual encoder, a LoRA bypass is added to realize the fusion of local visual features.

[0248] To this end, the local image block is input into the corresponding pre-trained visual encoder to obtain the first local visual feature corresponding to the local image block; the local image block is input into the corresponding pre-trained visual bypass model to obtain the second local visual feature corresponding to the local image block; the first local visual feature and the second local visual feature are feature fused (here feature fusion refers to the addition of the first local visual feature and the second local visual feature) to obtain the original local visual feature corresponding to the local image block. In this way, each local image block is input into the pre-trained visual encoder with LoRA bypass, so as to effectively identify and fuse the detail-sensitive features of the local area corresponding to each image block, enhance the understanding of spatial and contextual relationships, and obtain the original local visual features.

[0249] It should be noted that the pre-trained visual encoder, pre-trained visual resampling model, and pre-trained large language model mentioned above reuse the visual encoder, visual resampling model, and large language model in the current large model, and the visual encoder is reused multiple times. In addition, a visual bypass model is added on the basis of the visual encoder to extract local visual features, and the original visual encoder is retained to extract global visual features.

[0250] By slicing the original image to obtain multiple local image blocks, adjusting the resolution of the original image to obtain the target image, inputting the multiple local image blocks, the target image, and the demand statement into the pre-trained large model, the answer statement corresponding to the demand statement is obtained. In this way, the method of slicing the original image can expand the ability of the pre-trained large model to process high-resolution images, and can achieve good processing of high-resolution images. In addition, the method of slicing the original image can also expand the ability of the pre-trained large model to process complex images, and can achieve good processing of complex images.

[0251] In addition, in the embodiments of the present application, the demand expressions can generally be roughly divided into two categories, one is a general demand expression, such as "please describe this image as a whole", and the other is a specific demand expression, such as "what brand is the refrigerator in this image". For general demand expressions, it is necessary to grasp the overall image, so it is necessary to let the large model pay attention to the overall visual information of the image, while for specific demand expressions, the large model can pay more attention to the visual information related to the specific demand expression, so as to improve the accuracy of the answer expression generated by the large model.

[0252] Based on this, for the demand statement, the demand statement is input into the pre-trained semantic classification model so as to determine the category to which the demand statement belongs (i.e., the above two categories of demand statements) according to the output results of the pre-trained semantic classification model. When the category is the first category (i.e., a general demand statement), it is necessary to let the large model focus on the overall visual information of the image, and the above steps S201 to S209 can be executed, that is, the above-mentioned image processing method.

[0253] When the category is the second category (i.e., a specific demand statement), the pre-trained entity recognition model can be used to identify the entity in the demand statement (e.g., a refrigerator), and the pre-trained target detection model can be used to detect the target in the original image (e.g., a refrigerator). The entity and the target are semantically associated with each other to obtain the semantic association between the entity and the target. When the semantic association between the entity and the target exceeds a preset association threshold, it is possible to choose to capture a local image block of the area where the target is located in the original image (e.g., a local image block of the area where the refrigerator is located).

[0254] The resolution of the original image is adjusted to obtain the target image, and the resolution of the local image block of the target area in the original image is consistent with the resolution of the target image, for example, both are 448×448. In this way, the local image block of the target area, the target image and the demand statement are processed to obtain the answer statement corresponding to the demand statement.

[0255] Specifically, the local image block of the target area, the target image and the demand statement are input into the pre-trained large model to obtain the answer statement corresponding to the demand statement. If the pre-trained large model mentioned above includes a pre-trained feature extraction model and a pre-trained large language model, the local image block of the target area and the target image can be input into the pre-trained feature extraction model to obtain the target local visual features and the target global visual features, and the demand statement is encoded to obtain the text encoding features, and the text encoding features, the target global visual features and the target local visual features are input into the pre-trained large language model to obtain the answer statement corresponding to the demand statement.

[0256] If the pre-trained feature extraction model mentioned above includes multiple pre-trained visual fusion models, pre-trained visual encoders and pre-trained visual resampling models, a pre-trained visual fusion model can be started at this time. To this end, the local image block of the target area is input into the pre-trained visual fusion model to obtain the original local visual features corresponding to the local image block, the original local visual features are input into the pre-trained visual resampling model to obtain the target local visual features corresponding to the local image block, the target image is input into the pre-trained visual encoder to obtain the original global visual features corresponding to the target image, and the original global visual features are input into the pre-trained visual resampling model to obtain the target global visual features.

[0257] As mentioned above, the pre-trained visual fusion model includes a pre-trained visual encoder and a pre-trained visual bypass model. For this purpose, the local image block of the target area is input into the corresponding pre-trained visual encoder to obtain the first local visual feature corresponding to the local image block. The local image block of the target area is input into the corresponding pre-trained visual bypass model to obtain the second local visual feature corresponding to the local image block. The first local visual feature is feature fused with the second local visual feature to obtain the original local visual feature corresponding to the local image block of the target area.

[0258] It should be noted that, for the local image block of the target area, in addition to determining it through the above-mentioned clipping operation, the original image can also be cut into blocks to obtain multiple local image blocks, where each local image block is a corresponding local area in the original image, and then the similarity between the target and each local image block is determined, and the local image block with the highest similarity is selected as the local image block of the target area. The embodiments of the present application are not limited to this.

[0259] In addition, in the embodiment of the present application, the large model needs to be trained in advance to obtain the pre-trained large model. Figure 5 As shown, it is a schematic diagram of an implementation flow of a model training method provided in an embodiment of the present application. The method is applied to an electronic device and may specifically include the following steps:

[0260] S501, obtaining a sample image and a sample requirement statement for the sample image, and determining a sample answer statement corresponding to the sample requirement statement.

[0261] In the embodiment of the present application, a sample image and a sample requirement statement for the sample image (which can be understood as a question raised by a user) can be obtained. In addition, for the sample requirement statement, a sample answer statement corresponding to the sample requirement statement (which can be understood as an answer to the question raised by the user) can be determined.

[0262] It should be noted that the sample requirement statement can correspond to different tasks, such as image description tasks, general visual question answering tasks, text-centric visual question answering tasks, and document-related visual question answering tasks. For different tasks, the format of the sample requirement statement is different. For example, for the image description task, the format of the sample requirement statement is generally "Generate Chinese image description:".

[0263] In addition, for the task corresponding to the sample requirement statement, when the task is not the target task (such as an image description task), a preset data set corresponding to the task can be determined, and the preset data set includes sample images and annotated statements corresponding to the sample images. Thus, the annotated statements of the sample images can be found in the preset data set, and the annotated statements can be determined as the sample answer statements corresponding to the sample requirement statement.

[0264] It should be noted that for the task corresponding to the sample requirement expression, when the task is a general visual question answering task, the corresponding preset data set may be VQAV2, OKVQA, GQA, ScienceQA, VizWiz and other data sets; when the task is a text-centric visual question answering task, the corresponding preset data set may be TextVQA, OCRVQA, AI2D and other data sets; when the task is a document-related visual question answering task, the corresponding preset data set may be DocVQA, ChartQA, InfoVQA, DeepForm, Kleister Charity, WikiTableQuestions, TableFact and VisualMRC and other data sets. The above data sets contain sample images and their corresponding annotation expressions, which are not limited in the embodiments of the present application.

[0265] In addition, for the task corresponding to the sample requirement statement, when the task is a target task (such as an image description task), the following processing can be performed: determine a preset data set corresponding to the task, the preset data set includes a sample image; from the preset data set, search for the annotated expression of the sample image, and determine the annotated expression as the sample answer expression corresponding to the sample requirement statement; or, generate a detailed description expression of the sample image, and determine the detailed description expression as the sample answer expression corresponding to the sample requirement statement.

[0266] It should be noted that for the task corresponding to the sample requirement statement, when the task is an image description task, the corresponding preset data set can be a data set such as COCO Caption and TextCaps, wherein the data set contains sample images and their corresponding annotation statements. The annotation statement corresponding to the sample image (which can be understood as the original brief annotation for the sample image) can be regarded as the sample answer statement corresponding to the sample requirement statement, or a high-quality detailed description statement can be directly generated for the sample image. The embodiments of the present application are not limited to this.

[0267] The generation of the detailed description of the sample image can be done with the help of a multi-level image description generation pipeline. Figure 6 FIG. 1 is a schematic diagram of an implementation flow of a detailed description statement generation method provided in an embodiment of the present application. The method is applied to an electronic device and may specifically include the following steps:

[0268] S601, input the sample image into a pre-trained global description generation model to obtain a global description expression corresponding to the sample image.

[0269] In an embodiment of the present application, for a sample image, the sample image can be input into a pre-trained global description generation model to obtain a global description representation corresponding to the sample image. The pre-trained global description generation model can be, for example, BLIP2.

[0270] It should be noted that the pre-trained global description generation model provides an in-depth understanding of the relationship between images and original short annotations, so that the sample image can be input into the pre-trained global description generation model to obtain the global description representation corresponding to the sample image.

[0271] For example, for a sample image, the sample image may be input into BLIP2 to obtain a global description statement corresponding to the sample image, and the global description statement may be, for example, “a formal dining room with stone walls”.

[0272] S602, input the sample image into a pre-trained region description generation model to obtain a specific region in the sample image and a region description representation of the specific region.

[0273] In an embodiment of the present application, for a sample image, the sample image can be input into a pre-trained region description generation model to obtain a specific region in the sample image and a region description representation of the specific region. The pre-trained region description generation model can be, for example, GRIT.

[0274] It should be noted that the specific area can be expressed in the form of coordinates, specifically the coordinate values ​​corresponding to the four borders of the specific area. For example, the specific area can be expressed in the form of (0.37, 0.07, 0.63, 0.4).

[0275] For example, for a sample image, the sample image can be input into GRIT to obtain a specific area in the sample image and a region description of the specific area. The specific area is (0.37, 0.07, 0.63, 0.4), and the region description is "a chandelier hanging on the ceiling."

[0276] S603: Input the sample image into a pre-trained text detection model to obtain a text region in the sample image and text in the text region.

[0277] In an embodiment of the present application, for a sample image, the sample image can be input into a pre-trained text detection model to obtain a text region in the sample image and text in the text region. The pre-trained text detection model can be, for example, PPOCR.

[0278] It should be noted that the text area can be represented in the form of coordinates, specifically the coordinate values ​​corresponding to the four borders of the text area. For example, the text area can be represented by (0.37, 0.07, 0.63, 0.4).

[0279] S604: Input the sample image into a pre-trained object recognition model to obtain the object region in the sample image.

[0280] S605: Input the object region in the sample image into a pre-trained object description generation model to obtain an object description expression corresponding to the object region in the sample image.

[0281] In an embodiment of the present application, for a sample image, the sample image can be input into a pre-trained object recognition model to obtain an object region in the sample image, and the object region in the sample image can be input into a pre-trained object description generation model to obtain an object description representation corresponding to the object region in the sample image. The pre-trained object recognition model can be, for example, SAM, and the pre-trained object description generation model can be, for example, BLIP2.

[0282] It should be noted that the object area includes an object and can be represented in the form of coordinates, specifically the coordinate values ​​corresponding to the four borders of the object area. For example, the object area can be represented by (0.88, 0.33, 1.0, 0.74).

[0283] For example, for a sample image, the sample image is input into SAM to obtain the object area (0.88, 0.33, 1.0, 0.74) in the sample image, and the object area (0.88, 0.33, 1.0, 0.74) in the sample image is input into BLIP2 to obtain the object description corresponding to the object area in the sample image, which is “a large wooden sideboard with glass doors”.

[0284] S606, generating a detailed description of the sample image according to the global description, the specific area, the area description, the text area, the text, the object area and the object description.

[0285] In an embodiment of the present application, for a global description statement of a sample image, a specific area in the sample image, a regional description statement of the specific area, a text area in the sample image, text in the text area, an object area in the sample image, and an object description statement corresponding to the object area in the sample image, a detailed description statement of the sample image can be generated based on the global description statement, the specific area, the regional description statement, the text area, the text, the object area and the object description statement.

[0286] Among them, for the regional description expressions between specific areas in the above-mentioned sample images, the text areas in the sample images and the texts in the text areas, and the object description expressions corresponding to the object areas in the sample images, the consistency between the two can be checked and the ones with lower matching scores can be filtered out.

[0287] To this end, the specific area and the regional description are input into the pre-trained matching model to obtain the regional matching score between the specific area and the regional description. According to the regional matching score, the specific area and the regional description are filtered to obtain the target specific area and the target area description, which means filtering out the specific area and the regional description with a low regional matching score.

[0288] The text area and text are input into the pre-trained matching model to obtain the text matching score between the text area and the text. According to the text matching score, the text area and text are filtered to obtain the target text area and target text, which means filtering out the text areas and texts with lower text matching scores.

[0289] The object region and the object description are input into the pre-trained matching model to obtain the object matching score between the object region and the object description. According to the object matching score, the object region and the object description are filtered to obtain the target object region and the target object description, which means filtering out the object region and object description with a low object matching score.

[0290] Thus, a detailed description of the sample image can be generated according to the global description, the target area description, the target text, and the target object description. The final detailed description can be generated by means of the excellent ability of the large language model in natural language generation. To this end, the global description, the target area description, the target text, and the target object description are input into the pre-trained large language model to obtain a detailed description of the sample image.

[0291] It should be noted that the sample images and detailed descriptions of the sample images can form a picture-text pair, and the number of the picture-text pairs can be multiple. To this end, for any sample image, the similarity between the detailed description of the sample image and the detailed description of any other sample image is determined. If the similarity is higher than the preset similarity threshold, it means that the detailed descriptions of the two sample images are highly similar, and one of the picture-text pairs can be deleted. After the sample images and detailed descriptions of the sample images are screened, they can be used for subsequent large model training.

[0292] S502, performing block processing on the sample image to obtain a plurality of sample local image blocks, wherein each sample local image block is a corresponding sample local area in the sample image.

[0293] In the embodiment of the present application, for a sample image, the sample image may be cut into blocks to obtain a plurality of sample local image blocks, wherein each sample local image block is a corresponding sample local area in the sample image.

[0294] It should be noted that the resolution of the original image may be of various types, so before the image slicing process, the resolution of the sample image needs to be adjusted to a uniform resolution, which can facilitate the image slicing process.

[0295] Before adjusting the resolution of the sample image, it is necessary to determine whether the resolution of the sample image is a preset resolution. If it is the preset resolution, the image segmentation processing requirement is met and no further resolution adjustment is required.

[0296] For a sample image, when the resolution of the sample image is a preset resolution, it means that there is no need to adjust the resolution of the sample image. In this case, the sample image can be cut into blocks to obtain a plurality of sample local image blocks.

[0297] For the sample image, when the resolution of the sample image is less than the preset resolution, it indicates that the resolution of the sample image needs to be adjusted, so that the resolution of the sample image can be adjusted to the preset resolution.

[0298] It should be noted that the resolution of the sample image can be adjusted to a preset resolution by interpolation. Of course, other methods are also possible, such as filling with preset pixel values ​​(such as 0 or 255). The embodiments of the present application are not limited to this.

[0299] In this way, the resolution of the sample image reaches a preset resolution, for example, 896×1344, after resolution adjustment. At this time, the sample image after resolution adjustment can be cut into blocks to obtain multiple sample local image blocks.

[0300] For the sample image, when the resolution of the sample image is greater than the preset resolution, it indicates that the resolution of the sample image needs to be adjusted, so that the resolution of the sample image can be adjusted to the preset resolution.

[0301] It should be noted that, for the resolution of the sample image, the size of the sample image can be adjusted through corresponding image editing software to achieve the purpose of adjusting the resolution of the sample image, so that the resolution of the sample image reaches the preset resolution.

[0302] In this way, the resolution of the sample image reaches a preset resolution, for example, 896×1344, after resolution adjustment. At this time, the sample image after resolution adjustment can be cut into blocks to obtain multiple sample local image blocks.

[0303] S503, adjusting the resolution of the sample image to obtain a sample target image, wherein the resolution of the sample target image is consistent with the resolution of each sample local image block.

[0304] In the embodiment of the present application, for a sample image, the resolution of the sample image can be adjusted to obtain a sample target image, wherein the resolution of the sample target image is consistent with the resolution of each sample local image block, which means that the resolution of the sample image is adjusted to a resolution consistent with the sample local image block.

[0305] For example, for a sample image, the resolution of the sample image can be adjusted to obtain a sample target image, the resolution of the sample target image is 448×448, and the resolution of the sample target image is consistent with the resolution of each sample local image block, that is, for the sample image, its resolution is adjusted to a resolution of 448×448 that is consistent with the sample local image block.

[0306] S504, input multiple sample local image blocks, sample target images and sample demand statements into the large model to obtain a predicted answer statement corresponding to the sample demand statement.

[0307] In an embodiment of the present application, multiple sample local image blocks, sample target images and sample demand statements can be processed by a large model, so that multiple sample local image blocks, sample target images and sample demand statements can be input into the large model to obtain the predicted answer statement corresponding to the sample demand statement.

[0308] Among them, the large model usually includes a feature extraction model and a large language model, and the large language model can be any current large language model. Thus, multiple sample local image blocks and sample target images are input into the feature extraction model to obtain multiple predicted target local visual features and predicted target global visual features, and the sample demand expression is encoded to obtain sample text encoding features, and the sample text encoding features, predicted target global visual features and multiple predicted target local visual features are input into the large language model to obtain the predicted answer expression corresponding to the sample demand expression.

[0309] The feature extraction model usually includes multiple visual fusion models, visual encoders and visual resampling models, and the number of visual fusion models is consistent with the number of sample local image blocks, which means that each sample local image block has a corresponding visual fusion model. Therefore, for any sample local image block, the sample local image block is input into the corresponding visual fusion model to obtain the predicted original local visual features corresponding to the sample local image block, the predicted original local visual features are input into the visual resampling model to obtain the predicted target local visual features corresponding to the sample local image block, the sample target image is input into the visual encoder to obtain the predicted original global visual features corresponding to the sample target image, the predicted original global visual features are input into the visual resampling model to obtain the predicted target global visual features corresponding to the sample target image.

[0310] It should be noted that for the visual resampling model, the number can be 1, which means that the predicted original local visual features extracted by the visual fusion model and the predicted original global visual features extracted by the visual encoder will be input into a common visual resampling model for visual resampling. Special and important predicted target local visual features and predicted target global visual features can be extracted to better grasp the various details in the sample image. In addition, using the same visual resampling model can effectively improve the overall reasoning speed of the model, so that large models can have a better user experience.

[0311] In addition, the visual fusion model usually includes a visual encoder (consistent with the above-mentioned visual encoder, that is, multiple visual encoders are reused) and a visual bypass model. The visual bypass model can be implemented through LoRA (Low-Rank Adaptation of Large Language Models), which means that on the basis of the original visual encoder, a LoRA bypass is added to realize the fusion of local visual features.

[0312] To this end, the sample local image block is input into the corresponding visual encoder to obtain the predicted first local visual feature corresponding to the sample local image block, the sample local image block is input into the corresponding visual bypass model to obtain the predicted second local visual feature corresponding to the sample local image block, the predicted first local visual feature and the predicted second local visual feature are feature fused (here feature fusion refers to the addition of the predicted first local visual feature and the predicted second local visual feature) to obtain the predicted original local visual feature corresponding to the sample local image block.

[0313] It should be noted that the visual encoder, visual resampling model, and large language model mentioned above reuse the visual encoder, visual resampling model, and large language model in the current large model, and the visual encoder is reused multiple times. In addition, a visual bypass model is added on the basis of the visual encoder to extract local visual features, and the original visual encoder is retained to extract global visual features.

[0314] S505, determining the expression loss between the sample answer expression and the predicted answer expression, and training the large model based on the expression loss.

[0315] S506, when the representation loss converges, stop training and obtain a pre-trained large model.

[0316] In an embodiment of the present application, the expression loss between the sample answer expression and the predicted answer expression can be determined, and the large model can be trained based on the expression loss. When the expression loss converges (for example, the expression loss is less than a certain threshold), the training is stopped to obtain a pre-trained large model.

[0317] It should be noted that for a large model, which includes the above-mentioned multiple visual bypass models, multiple visual encoders, visual resampling models, and large language models, during the training of the large model, the parameters in the multiple visual encoders can be frozen, while the parameters in the multiple visual bypass models, visual resampling models, and large language models can be trained.

[0318] After the above model training, a pre-trained large model can be obtained. By cutting the original image into pieces, the ability of the pre-trained large model to process high-resolution images can be expanded, and high-resolution images can be processed well. In addition, by cutting the original image into pieces, the ability of the pre-trained large model to process complex images can be expanded, and complex images can be processed well.

[0319] Corresponding to the above method embodiment, the present application embodiment also provides an image processing device, such as Figure 7As shown, the device may include: an information acquisition module 710 , an image slicing module 720 , a resolution adjustment module 730 , and a description generation module 740 .

[0320] An information acquisition module 710 is used to acquire an original image and a requirement statement for the original image;

[0321] An image cutting module 720 is used to cut the original image into blocks to obtain a plurality of local image blocks, wherein each local image block is a corresponding local area in the original image;

[0322] A resolution adjustment module 730, configured to adjust the resolution of the original image to obtain a target image, wherein the resolution of the target image is consistent with the resolution of each of the local image blocks;

[0323] The expression generation module 740 is used to process the multiple local image blocks, the target image and the requirement expression to obtain an answer expression corresponding to the requirement expression.

[0324] In an optional implementation, the image slicing module is specifically used to:

[0325] Determining whether the resolution of the original image is a preset resolution;

[0326] When the resolution of the original image is the preset resolution, the original image is sliced ​​to obtain a plurality of local image blocks.

[0327] In an optional implementation, the image slicing module is further configured to:

[0328] When the resolution of the original image is less than the preset resolution, adjusting the resolution of the original image to the preset resolution;

[0329] The original image after the resolution adjustment is cut into blocks to obtain a plurality of local image blocks.

[0330] In an optional implementation, the image slicing module is further configured to:

[0331] When the resolution of the original image is greater than the preset resolution, adjusting the resolution of the original image to the preset resolution;

[0332] The original image after the resolution adjustment is cut into blocks to obtain a plurality of local image blocks.

[0333] In an optional implementation, the expression generation module is specifically used to:

[0334] The plurality of local image blocks, the target image and the requirement statement are input into a pre-trained large model to obtain an answer statement corresponding to the requirement statement.

[0335] In an optional embodiment, the pre-trained large model includes a pre-trained feature extraction model and a pre-trained large language model; the expression generation module specifically includes:

[0336] An image processing submodule, used for inputting the plurality of local image blocks and the target image into the pre-trained feature extraction model to obtain a plurality of target local visual features and a target global visual feature;

[0337] A statement encoding submodule is used to encode the requirement statement to obtain text encoding features;

[0338] The expression generation submodule is used to input the text encoding features, the target global visual features and multiple target local visual features into the pre-trained large language model to obtain an answer expression corresponding to the demand expression.

[0339] In an optional embodiment, the pre-trained feature extraction model includes a plurality of pre-trained visual fusion models, a pre-trained visual encoder and a pre-trained visual resampling model; the image processing submodule specifically includes:

[0340] An image block input unit, used for inputting any of the local image blocks into the corresponding pre-trained visual fusion model to obtain an original local visual feature corresponding to the local image block;

[0341] An original local visual feature input unit, used for inputting the original local visual feature into the pre-trained visual resampling model to obtain a target local visual feature corresponding to the local image block;

[0342] A target image input unit, used to input the target image into the pre-trained visual encoder to obtain the original global visual features corresponding to the target image;

[0343] The original global visual feature input unit is used to input the original global visual feature into the pre-trained visual resampling model to obtain the target global visual feature.

[0344] In an optional embodiment, the pre-trained visual fusion model includes a pre-trained visual encoder and a pre-trained visual bypass model; and the image block input unit is specifically used for:

[0345] Inputting the local image block into the corresponding pre-trained visual encoder to obtain a first local visual feature corresponding to the local image block;

[0346] Inputting the local image block into the corresponding pre-trained visual bypass model to obtain a second local visual feature corresponding to the local image block;

[0347] The first local visual feature and the second local visual feature are fused to obtain an original local visual feature corresponding to the local image block.

[0348] In an optional embodiment, the device further comprises:

[0349] A representation and image acquisition module, used to acquire sample images and sample requirement representations for the sample images;

[0350] A statement determination module, used to determine a sample answer statement corresponding to the sample requirement statement;

[0351] A sample image slicing module, used for slicing the sample image to obtain a plurality of sample local image blocks, wherein each of the sample local image blocks is a corresponding sample local area in the sample image;

[0352] A sample image adjustment module, used for adjusting the resolution of the sample image to obtain a sample target image, wherein the resolution of the sample target image is consistent with the resolution of each of the sample local image blocks;

[0353] An image and statement input module, used to input a plurality of the sample local image blocks, the sample target image and the sample demand statement into the large model to obtain a predicted answer statement corresponding to the sample demand statement;

[0354] A loss determination module, configured to determine a statement loss between the sample answer statement and the predicted answer statement;

[0355] The model training module is used to train the large model according to the representation loss, and stop the training when the representation loss converges to obtain the pre-trained large model.

[0356] In an optional implementation, the sample image slicing module is specifically used to:

[0357] Determining whether the resolution of the sample image is a preset resolution;

[0358] When the resolution of the sample image is the preset resolution, the sample image is cut into blocks to obtain a plurality of sample local image blocks.

[0359] In an optional implementation, the sample image slicing module is further configured to:

[0360] When the resolution of the sample image is less than the preset resolution, adjusting the resolution of the sample image to the preset resolution;

[0361] The sample image after the resolution adjustment is cut into blocks to obtain a plurality of sample local image blocks.

[0362] In an optional implementation, the sample image slicing module is further configured to:

[0363] When the resolution of the sample image is greater than the preset resolution, adjusting the resolution of the sample image to the preset resolution;

[0364] The sample image after the resolution adjustment is cut into blocks to obtain a plurality of sample local image blocks.

[0365] In an optional implementation, the large model includes a feature extraction model and a large language model; the image and expression input module specifically includes:

[0366] An image input submodule, used for inputting the plurality of sample local image blocks and the sample target image into the feature extraction model to obtain a plurality of predicted target local visual features and predicted target global visual features;

[0367] A sample requirement expression encoding submodule is used to encode the sample requirement expression to obtain sample text encoding features;

[0368] The feature input submodule is used to input the sample text encoding feature, the prediction target global visual feature and multiple prediction target local visual features into the large language model to obtain the predicted answer statement corresponding to the sample demand statement.

[0369] In an optional embodiment, the feature extraction model includes multiple visual fusion models, a visual encoder and a visual resampling model; the image input submodule specifically includes:

[0370] A sample local image block input unit, used for inputting any of the sample local image blocks into the corresponding visual fusion model to obtain a predicted original local visual feature corresponding to the sample local image block;

[0371] A predicted original local visual feature input unit, used for inputting the predicted original local visual feature into the visual resampling model to obtain a predicted target local visual feature corresponding to the sample local image block;

[0372] A sample target image input unit, used to input the sample target image into the visual encoder to obtain the predicted original global visual features corresponding to the sample target image;

[0373] The predicted original global visual feature input unit is used to input the predicted original global visual feature into the visual resampling model to obtain the predicted target global visual feature corresponding to the sample target image.

[0374] In an optional implementation, the visual fusion model includes a visual encoder and a visual bypass model; the sample local image block input unit is specifically used for:

[0375] Inputting the sample local image block into the corresponding visual encoder to obtain a predicted first local visual feature corresponding to the sample local image block;

[0376] Inputting the sample local image block into the corresponding visual bypass model to obtain a predicted second local visual feature corresponding to the sample local image block;

[0377] The predicted first local visual feature and the predicted second local visual feature are feature fused to obtain the predicted original local visual feature corresponding to the sample local image block.

[0378] In an optional implementation, the expression determination module is specifically used to:

[0379] Determine a task corresponding to the sample requirement statement, and if the task is not a target task, determine a preset data set corresponding to the task, wherein the preset data set includes the sample image;

[0380] From the preset data set, the annotated expression of the sample image is searched, and the annotated expression is determined as the sample answer expression corresponding to the sample requirement expression.

[0381] In an optional implementation, the expression determination module specifically includes:

[0382] The task determination submodule is used to determine the task corresponding to the sample requirement statement, and when the task is a target task, perform the following processing:

[0383] A data set determination submodule, used to determine a preset data set corresponding to the task, wherein the preset data set includes the sample image;

[0384] A first submodule for determining a statement is used to search for a labeled statement of the sample image from the preset data set, and determine the labeled statement as a sample answer statement corresponding to the sample requirement statement;

[0385] or,

[0386] A detailed description generation submodule, used to generate a detailed description of the sample image;

[0387] The second submodule for determining the expression is used to determine the detailed description expression as the sample answer expression corresponding to the sample requirement expression.

[0388] In an optional implementation, the detailed description statement generation submodule specifically includes:

[0389] A global description generation unit, used for inputting the sample image into a pre-trained global description generation model to obtain a global description corresponding to the sample image;

[0390] A region description generation unit, used for inputting the sample image into a pre-trained region description generation model to obtain a specific region in the sample image and a region description of the specific region;

[0391] A text generation unit, used for inputting the sample image into a pre-trained text detection model to obtain a text region in the sample image and text in the text region;

[0392] An object region generating unit, used for inputting the sample image into a pre-trained object recognition model to obtain an object region in the sample image;

[0393] An object description generation unit, configured to input the object region in the sample image into a pre-trained object description generation model to obtain an object description corresponding to the object region in the sample image;

[0394] A detailed description statement generating unit is used to generate a detailed description statement of the sample image according to the global description statement, the specific area, the area description statement, the text area, the text, the object area and the object description statement.

[0395] In an optional implementation, the detailed description statement generating unit specifically includes:

[0396] A region matching score generating subunit, configured to input the specific region and the region description into a pre-trained matching model to obtain a region matching score between the specific region and the region description;

[0397] A region description filtering subunit, configured to filter the specific region and the region description according to the region matching score to obtain a target specific region and a target region description;

[0398] A text matching score generating subunit, used for inputting the text region and the text into a pre-trained matching model to obtain a text matching score between the text region and the text;

[0399] A text filtering subunit, used for filtering the text area and the text according to the text matching score to obtain a target text area and a target text;

[0400] an object matching score generating subunit, configured to input the object region and the object description into a pre-trained matching model to obtain an object matching score between the object region and the object description;

[0401] An object description filtering unit, configured to filter the object region and the object description according to the object matching score to obtain a target object region and a target object description;

[0402] The detailed description statement generating subunit is used to generate a detailed description statement of the sample image according to the global description statement, the target area description, the target text and the target object description statement.

[0403] In an optional implementation, the detailed description statement generating subunit is specifically used for:

[0404] The global description, the target area description, the target text and the target object description are input into a pre-trained large language model to obtain a detailed description of the sample image.

[0405] The present application also provides an electronic device, such as Figure 8 As shown, it includes a processor 81, a communication interface 82, a memory 83 and a communication bus 84, wherein the processor 81, the communication interface 82, and the memory 83 communicate with each other through the communication bus 84.

[0406] A memory 83, for storing computer programs;

[0407] The processor 81 is used to execute the program stored in the memory 83 to implement the following steps:

[0408] Acquire an original image and a requirement statement for the original image; cut the original image into blocks to obtain a plurality of local image blocks, wherein each of the local image blocks is a corresponding local area in the original image; adjust the resolution of the original image to obtain a target image, wherein the resolution of the target image is consistent with the resolution of each of the local image blocks; process the plurality of local image blocks, the target image and the requirement statement to obtain an answer statement corresponding to the requirement statement.

[0409] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0410] The communication interface is used for communication between the above electronic device and other devices.

[0411] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0412] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0413] In another embodiment provided in the present application, a storage medium is provided. The storage medium stores instructions. When the instructions are executed on a computer, the computer executes the image processing method described in any one of the above embodiments.

[0414] In another embodiment provided in the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute the image processing method described in any one of the above embodiments.

[0415] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a storage medium, or transmitted from one storage medium to another storage medium, for example, the computer instructions may be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The storage medium may be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive Solid State Disk (SSD)), etc.

[0416] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0417] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0418] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire an original image and a requirement statement for the original image; Performing block processing on the original image to obtain a plurality of local image blocks, wherein each of the local image blocks is a corresponding local area in the original image; Adjusting the resolution of the original image to obtain a target image, wherein the resolution of the target image is consistent with the resolution of each of the local image blocks; The plurality of local image blocks, the target image and the requirement statement are processed to obtain an answer statement corresponding to the requirement statement.

2. The method according to claim 1, characterized in that The block processing of the original image to obtain a plurality of local image blocks includes: Determining whether the resolution of the original image is a preset resolution; When the resolution of the original image is the preset resolution, the original image is sliced ​​to obtain a plurality of local image blocks.

3. The method according to claim 1, characterized in that The processing of the plurality of local image blocks, the target image, and the requirement statement to obtain an answer statement corresponding to the requirement statement includes: The plurality of local image blocks, the target image and the requirement statement are input into a pre-trained large model to obtain an answer statement corresponding to the requirement statement.

4. The method according to claim 3, characterized in that The pre-trained large model includes a pre-trained feature extraction model and a pre-trained large language model; The step of inputting the plurality of local image blocks, the target image, and the requirement statement into a pre-trained large model to obtain an answer statement corresponding to the requirement statement includes: Inputting the plurality of local image blocks and the target image into the pre-trained feature extraction model to obtain a plurality of target local visual features and a target global visual feature; Encoding the requirement statement to obtain text encoding features; The text encoding feature, the target global visual feature and a plurality of the target local visual features are input into the pre-trained large language model to obtain an answer statement corresponding to the demand statement.

5. The method according to claim 4, characterized in that The pre-trained feature extraction model includes a plurality of pre-trained visual fusion models, a pre-trained visual encoder and a pre-trained visual resampling model; The step of inputting the plurality of local image blocks and the target image into the pre-trained feature extraction model to obtain a plurality of target local visual features and target global visual features includes: For any of the local image blocks, input the local image block into the corresponding pre-trained visual fusion model to obtain the original local visual features corresponding to the local image block; Inputting the original local visual features into the pre-trained visual resampling model to obtain the target local visual features corresponding to the local image block; Inputting the target image into the pre-trained visual encoder to obtain original global visual features corresponding to the target image; The original global visual features are input into the pre-trained visual resampling model to obtain target global visual features.

6. The method according to claim 5, characterized in that The pre-trained visual fusion model includes a pre-trained visual encoder and a pre-trained visual bypass model; Inputting the local image block into the corresponding pre-trained visual fusion model to obtain the original local visual features corresponding to the local image block includes: Inputting the local image block into the corresponding pre-trained visual encoder to obtain a first local visual feature corresponding to the local image block; Inputting the local image block into the corresponding pre-trained visual bypass model to obtain a second local visual feature corresponding to the local image block; The first local visual feature and the second local visual feature are fused to obtain an original local visual feature corresponding to the local image block.

7. The method according to claim 3, characterized in that Before executing the method, the method further includes: Obtaining a sample image and a sample requirement statement for the sample image, and determining a sample answer statement corresponding to the sample requirement statement; Performing block processing on the sample image to obtain a plurality of sample local image blocks, wherein each of the sample local image blocks is a corresponding sample local area in the sample image; Adjusting the resolution of the sample image to obtain a sample target image, wherein the resolution of the sample target image is consistent with the resolution of each of the sample local image blocks; Inputting the plurality of sample local image blocks, the sample target image and the sample requirement statement into the large model to obtain a predicted answer statement corresponding to the sample requirement statement; Determining a representation loss between the sample answer representation and the predicted answer representation, and training the large model based on the representation loss; When the representation loss converges, the training is stopped to obtain a pre-trained large model.

8. The method according to claim 7, characterized in that The determining of a sample answer statement corresponding to the sample requirement statement includes: Determine a task corresponding to the sample requirement statement, and if the task is not a target task, determine a preset data set corresponding to the task, wherein the preset data set includes the sample image; From the preset data set, the annotated expression of the sample image is searched, and the annotated expression is determined as the sample answer expression corresponding to the sample requirement expression.

9. The method according to claim 7, characterized in that: The determining of a sample answer statement corresponding to the sample requirement statement includes: Determine the task corresponding to the sample requirement statement, and if the task is a target task, perform the following processing: Determine a preset data set corresponding to the task, the preset data set including the sample image; Searching for the annotated expression of the sample image from the preset data set, and determining the annotated expression as the sample answer expression corresponding to the sample requirement expression; or, A detailed description statement of the sample image is generated, and the detailed description statement is determined as a sample answer statement corresponding to the sample requirement statement.

10. The method according to claim 9, characterized in that The detailed description of generating the sample image includes: Inputting the sample image into a pre-trained global description generation model to obtain a global description expression corresponding to the sample image; Inputting the sample image into a pre-trained region description generation model to obtain a specific region in the sample image and a region description expression of the specific region; Inputting the sample image into a pre-trained text detection model to obtain a text region in the sample image and text in the text region; Inputting the sample image into a pre-trained object recognition model to obtain an object region in the sample image; Inputting the object region in the sample image into a pre-trained object description generation model to obtain an object description expression corresponding to the object region in the sample image; A detailed description statement of the sample image is generated according to the global description statement, the specific area, the area description statement, the text area, the text, the object area and the object description statement.

11. The method according to claim 10, characterized in that The generating a detailed description of the sample image according to the global description, the specific area, the area description, the text area, the text, the object area and the object description includes: Inputting the specific area and the area description into a pre-trained matching model to obtain an area matching score between the specific area and the area description; According to the area matching score, the specific area and the area description are filtered to obtain a target specific area and a target area description; Inputting the text region and the text into a pre-trained matching model to obtain a text matching score between the text region and the text; Filtering the text area and the text according to the text matching score to obtain a target text area and a target text; Inputting the object region and the object description into a pre-trained matching model to obtain an object matching score between the object region and the object description; According to the object matching score, the object region and the object description are filtered to obtain a target object region and a target object description; A detailed description statement of the sample image is generated according to the global description statement, the target area description, the target text and the target object description statement.

12. The method according to claim 11, characterized in that The generating a detailed description of the sample image according to the global description, the target area description, the target text and the target object description includes: The global description, the target area description, the target text and the target object description are input into a pre-trained large language model to obtain a detailed description of the sample image.

13. An image processing device, characterized in that: The device comprises: An information acquisition module, used to acquire an original image and a requirement statement for the original image; An image slicing module, used for slicing the original image to obtain a plurality of local image blocks, wherein each local image block is a corresponding local area in the original image; A resolution adjustment module, used to adjust the resolution of the original image to obtain a target image, wherein the resolution of the target image is consistent with the resolution of each of the local image blocks; The expression generation module is used to process the multiple local image blocks, the target image and the demand expression to obtain an answer expression corresponding to the demand expression.

Images