Image processing method and device based on large language model, equipment and medium

Abstract

The invention belongs to the technical field of natural language processing, is suitable for the medical field and the financial field, and discloses an image processing method, device, equipment and medium based on a large language model.The method comprises the steps that image information to be processed is input into an image pyramid complementary module of a pre-constructed large language model for image feature extraction; at least three groups of complementary image sets of different visual fields are generated, and the complementary image sets at least comprise a detail image group, a global image group and a boundary supplementary image group; inputting the complementary image set into a multi-scale visual compression module of a pre-constructed large language model for compression processing to obtain a compressed complementary image set; and performing fusion processing on the multiple groups of visual tokens and the text tokens, inputting a fused token sequence into a pre-constructed large language model for reasoning processing, and generating a response result corresponding to the text question information. According to the method, the accuracy of image recognition of the large language model is effectively improved.

Description

Technical Field

[0001] This invention belongs to the field of natural language processing technology and is applicable to the medical and financial fields. In particular, it relates to an image processing method, device, equipment and medium based on a large language model. Background Technology

[0002] In recent years, large language models have made significant progress in joint visual and language understanding, becoming an important research direction in the field of artificial intelligence. These models can process both image and text information simultaneously, enabling complex tasks such as visual question answering, image description generation, and cross-modal retrieval. They have also demonstrated broad potential in various application scenarios. For example, in the financial sector, large language models can be used for automated customer service, enabling rapid information extraction and risk assessment by analyzing user-submitted documents and images. In the medical field, large language models can assist doctors in interpreting medical images and, combined with patient medical records, provide more accurate diagnostic recommendations.

[0003] However, large language models face significant technical challenges when processing high-resolution images. The sheer volume of pixel information in high-resolution images dramatically increases computational complexity. Directly processing the original high-resolution image globally requires extremely high computational and storage resources. To address this issue, current mainstream methods typically employ image segmentation strategies, cropping the input image into multiple local regions, processing each region separately, and then fusing the information from each region. While this method reduces the computational burden to some extent, it also introduces a serious semantic fragmentation problem. When the target object is segmented by cropping boundaries, the model can only acquire local fragments, lacking global context, leading to recognition errors.

[0004] Therefore, improving the accuracy of large language models in image recognition is a technical problem that urgently needs to be solved. Summary of the Invention

[0005] This invention provides an image processing method, apparatus, device, and medium based on a large language model, to solve the technical problem of how to improve the accuracy of image recognition using a large language model.

[0006] In a first aspect, the present invention provides an image processing method based on a large language model, comprising:

[0007] Obtain the image information to be processed, as well as the text query information associated with the image information to be processed;

[0008] The image information to be processed is input into the image pyramid complementation module of the pre-constructed large language model for image feature extraction, generating at least three sets of complementary image sets with different views, wherein the complementary image sets include at least a detail image set, a global image set, and a boundary supplement image set;

[0009] The complementary image set is input into the multi-scale visual compression module of the pre-built large language model for compression processing to obtain the compressed complementary image set.

[0010] The compressed complementary image set is input into the visual encoder of the pre-built large language model for visual encoding, generating multiple sets of visual tokens for the image information to be processed.

[0011] The text query information is input into the text encoder of a pre-built large language model for text encoding to generate a text token for the text query information;

[0012] The multiple sets of visual tokens and text tokens are fused together, and the fused token sequence is input into a pre-built large language model for inference processing to generate a response result corresponding to the text question information.

[0013] Secondly, the present invention provides an image processing apparatus based on a large language model, comprising:

[0014] The acquisition module is used to acquire the image information to be processed, as well as the text query information associated with the image information to be processed;

[0015] The extraction module is used to input the image information to be processed into the image pyramid complementation module of the pre-constructed large language model for image feature extraction, and generate at least three sets of complementary image sets with different views, wherein the complementary image sets include at least a detail image set, a global image set, and a boundary supplement image set;

[0016] The compression module is used to input the complementary image set into the multi-scale visual compression module of the pre-built large language model for compression processing to obtain the compressed complementary image set.

[0017] The first encoding module is used to input the compressed complementary image set into the visual encoder of the pre-built large language model for visual encoding, and generate multiple sets of visual tokens for the image information to be processed.

[0018] The second encoding module is used to input the text query information into the text encoder of the pre-built large language model for text encoding, and generate a text token for the text query information;

[0019] The generation module is used to fuse the multiple sets of visual tokens and text tokens, and input the fused token sequence into a pre-built large language model for inference processing to generate a response result corresponding to the text question information.

[0020] Thirdly, the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described image processing method based on a large language model.

[0021] Fourthly, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described image processing method based on a large language model.

[0022] The aforementioned image processing method, apparatus, device, and medium based on a large language model can acquire image information to be processed and associated textual question information through a client. The image information to be processed is input into a pre-constructed large language model's image pyramid complementation module for image feature extraction, generating at least three sets of complementary image sets with different viewpoints. Each complementary image set includes at least a detail image set, a global image set, and a boundary supplement image set. The complementary image sets are then input into a pre-constructed large language model's multi-scale visual compression module for compression processing, resulting in a compressed complementary image set. The compressed complementary image set is then input into a pre-constructed large language model's visual encoder for visual encoding, generating multiple sets of visual tokens for the image information to be processed. The textual question information is input into a pre-constructed large language model's text encoder for text encoding, generating textual tokens for the textual question information. The multiple sets of visual tokens and textual tokens are fused, and the fused token sequence is input into a pre-constructed large language model for inference processing, generating a response result corresponding to the textual question information. In this invention, by inputting the image information to be processed into the image pyramid complementation module of a pre-constructed large language model, detail image sets, global image sets, and boundary supplementation image sets are generated. This comprehensively captures important information in the image to be processed, significantly improving the ability to understand image content. Furthermore, after inputting the generated complementary image set into the multi-scale visual compression module of the large language model for compression processing, the resulting compressed complementary image set significantly reduces the computational cost of the image to be processed, effectively improving the accuracy of the large language model in image recognition. Attached Figure Description

[0023] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a schematic diagram of an application environment for an image processing method based on a large language model according to an embodiment of the present invention;

[0025] Figure 2 This is a flowchart illustrating an image processing method based on a large language model according to an embodiment of the present invention;

[0026] Figure 3 yes Figure 2 A schematic diagram of a specific implementation method for step S20;

[0027] Figure 4 yes Figure 2 A schematic diagram of a specific implementation method for step S30;

[0028] Figure 5 yes Figure 2 A schematic diagram of a specific implementation method for step S60;

[0029] Figure 6 This is a schematic diagram of an image processing device based on a large language model according to an embodiment of the present invention;

[0030] Figure 7 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention;

[0031] Figure 8 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0032] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0033] The image processing method based on a large language model provided in this invention can be applied to, for example... Figure 1 In the application environment, Figure 1This is a schematic diagram of an application environment for an image processing method based on a large language model according to an embodiment of the present invention; wherein, the client communicates with the server via a network. The server can obtain the image information to be processed and the text query information associated with the image information through the client; the image information to be processed is input into the image pyramid complementarity module of the pre-built large language model for image feature extraction, generating at least three sets of complementary image sets with different views, wherein the complementary image sets include at least a detail image set, a global image set, and a boundary supplement image set; the complementary image sets are input into the multi-scale visual compression module of the pre-built large language model for compression processing, obtaining a compressed complementary image set; the compressed complementary image set is input into the visual encoder of the pre-built large language model for visual encoding, generating multiple sets of visual tokens for the image information to be processed; the text query information is input into the text encoder of the pre-built large language model for text encoding, generating text tokens for the text query information; the multiple sets of visual tokens and text tokens are fused, and the fused token sequence is input into the pre-built large language model for inference processing, generating a response result corresponding to the text query information. In this invention, by inputting the image information to be processed into the image pyramid complementation module of a pre-constructed large language model, detail image sets, global image sets, and boundary supplementation image sets are generated. This comprehensively captures important information in the image to be processed, significantly improving the ability to understand image content. Furthermore, after inputting the generated complementary image set into the multi-scale visual compression module of the large language model for compression processing, the resulting compressed complementary image set significantly reduces the computational cost of the image to be processed, effectively improving the accuracy of the large language model in image recognition. The invention will be described in detail below through specific embodiments.

[0034] Please see Figure 2 As shown, Figure 2 This is a flowchart illustrating an image processing method based on a large language model provided in an embodiment of the present invention. The image processing method based on a large language model specifically includes the following steps:

[0035] S10: Obtain the image information to be processed, and the text question information associated with the image information to be processed. In this embodiment of the invention, firstly, a high-resolution image and related text questions need to be obtained. For example, in financial services, a user may upload an image of a financial statement and ask, "What types of investment returns does this statement show?" In this case, both the image and the question need to be obtained simultaneously. In medical image recognition, a doctor may upload an image examination result and ask, "Does this image show a tumor?" In this case, the image and the corresponding question need to be processed.

[0036] S20: The image information to be processed is input into the image pyramid complementation module of the pre-constructed large language model for image feature extraction, generating at least three sets of complementary image sets with different views. Each complementary image set includes at least a detail image set, a global image set, and a boundary supplementation image set. In this embodiment of the invention, by generating detail image sets, global image sets, and boundary supplementation image sets, important information in the image can be comprehensively captured, improving the ability to understand the image content. Specifically, for example... Figure 3 The above, Figure 3 This is a flowchart illustrating a specific implementation of step S20, which specifically includes the following steps S21-S23:

[0037] S21: The image information to be processed is input into the convolutional layer of the pyramid complement module for uniform grid partitioning, generating a set of detailed images containing high-resolution local regions. Specifically, in this embodiment of the invention, uniform grid partitioning allows the convolutional layer to focus on the detailed features of the image, such as texture, color, and shape. This set of detailed images can provide rich information for subsequent analysis. For example, in the financial field, a user may upload a report image containing multiple financial indicators. Through the set of detailed images, key numbers and indicators, such as profit and cost information, can be extracted. In medical image analysis, a doctor may upload a CT scan image; the set of detailed images can help identify the specific morphology of the tumor, enabling the doctor to make a more accurate diagnosis.

[0038] S22: The image information to be processed is input into the pooling layer of the pyramid complement module for downsampling processing, generating a global image group containing global context information. Specifically, in this embodiment of the invention, the pooling layer can effectively capture the global context of the image through downsampling, helping the model understand the overall structure and layout. For example, in financial services, a customer may query an annual financial statement. The global image group can help large language models identify overall trends such as revenue changes and cost control, rather than just focusing on a single number. In medical scenarios, when analyzing an X-ray, the global image group can help doctors quickly identify the state of the entire organ, rather than just local lesion areas.

[0039] S23: The image information to be processed is input into the boundary compensation layer of the pyramid complementarity module for misaligned mesh division, generating a boundary compensation image group containing compensation information. Specifically, in this embodiment of the invention, through misaligned mesh division, the boundary compensation layer focuses on the edges and contours of the image, enhancing the accuracy of object recognition. For example, in financial chart analysis, the boundary compensation image group can help the model more clearly identify trend lines and boundaries in the chart, ensuring the accuracy of the analysis results.

[0040] S30: The complementary image set is input into the multi-scale visual compression module of the pre-built large language model for compression processing to obtain the compressed complementary image set. In this embodiment of the invention, compression processing can significantly reduce the amount of data, reduce computational costs, and make subsequent processing more efficient. Specifically, as shown... Figure 4 The above, Figure 4 This is a flowchart illustrating a specific implementation of step S30, which specifically includes the following steps S31-S33:

[0041] S31: Based on the text token corresponding to the text question information and the visual token corresponding to the global image group in the complementary image set, a query vector related to the text question information is generated. Specifically, in this embodiment of the invention, by converting the text question information into a query vector, the user's intent can be clearly identified and the information related to that intent can be focused. For example, in a financial scenario, when analyzing financial statements, if a user asks, "What is the net profit of this financial statement?", the generated query vector can clearly point to the items related to net profit in the statement, thus providing a clear direction for subsequent image analysis. When analyzing medical images, if a doctor asks, "Does this image show a tumor?", the generated query vector will point to features related to tumors, such as abnormal shadows or specific tissue structures, which is helpful for subsequent image analysis.

[0042] S32: Calculate the correlation weight between the query vector and each visual token in the detail image group, and filter the visual tokens in the detail image group based on the correlation weight, retaining visual tokens with a weight higher than a preset threshold. Specifically, in this embodiment of the invention, by calculating the correlation between the query vector and each visual token in the detail image group, visual information highly relevant to the user's intent can be filtered out. For example, when analyzing financial statements, visual tokens related to net profit, such as revenue and cost, can be identified first and retained, while other irrelevant data is ignored, thereby ensuring that the information provided is more accurate.

[0043] S321: Calculate the attention weights between the query vector and each visual token using an attention mechanism, and normalize the attention weights to obtain the relevance weights between the query vector and each visual token. Specifically, in this embodiment of the invention, the model can identify visual tokens with stronger relevance to the query vector through the attention mechanism, thereby selecting the most important information from a large amount of visual data. Normalizing the attention weights ensures that the weights of different visual tokens are comparable, further enhancing the model's ability to judge relevance. For example, in financial data analysis, assuming a user inputs the query: "What is the largest expenditure item in this report?", the attention mechanism can calculate the weights of visual tokens related to "expenditure," focusing on the numbers and descriptions related to expenditure items, ensuring the accuracy and relevance of the analysis results. In a medical scenario, if a doctor asks: "Is there any abnormality in this image?" the model will calculate the attention weights between the query vector and all visual tokens in the image. Through the attention mechanism, areas that may indicate abnormalities can be highlighted, thereby helping doctors make diagnoses more quickly.

[0044] In one embodiment of the present invention, the formula for calculating the relevance weight between the query vector and each visual token is as follows:

[0045]

[0046] Where Q represents the query vector, V i Let d represent the i-th visual token in the detail image group, d represent the feature dimension of the visual token, softmax represent the normalization function, and Attention(Q,V) represent the function of the normalization function. i ) represents the relevance weight between the query vector and the i-th visual token.

[0047] S322: Sort the visual tokens in the detailed image group according to the relevance weights, and select the top K visual tokens by weight for retention, where K is a preset number of visual tokens to retain. Specifically, in this embodiment of the invention, by sorting the visual tokens according to their relevance weights, the information most relevant to the user's query can be effectively filtered out. This filtering process ensures that the model retains only the most important visual features, reduces redundant information, and improves the efficiency of subsequent processing. For example, in medical imaging, if visual tokens related to "tumor" are filtered out, only K visual tokens directly related to tumor size, shape, and boundary clarity may be retained in the end. Such information focus can help doctors quickly identify lesions and improve diagnostic efficiency.

[0048] S33: The visual tokens retained in the detail image group are fused with the visual tokens of the global image group and the boundary compensation image group to obtain a compressed complementary image set. Specifically, in this embodiment of the invention, by fusing the retained detail visual tokens with the visual tokens of the global image group and the boundary compensation image group, a more comprehensive complementary image set can be formed, providing richer contextual information for subsequent inference. The fused complementary image set can better reflect the overall content and local features of the image, thereby improving the model's inference ability. For example, when finally generating a financial analysis report, the fused image set can simultaneously provide key boundary information such as detailed data on net profit, overall trends, and the relationship between expenditures and revenues, making the analysis results more comprehensive and easier for users to understand.

[0049] S40: The compressed complementary image set is input into the visual encoder of the pre-built large language model for visual encoding, generating multiple sets of visual tokens for the image information to be processed. Specifically, in this embodiment of the invention, converting image features into visual tokens can transform image information into a representation suitable for model processing, thereby improving the efficiency of model processing. For example, in medical image analysis, visual tokens can accurately represent information such as lesion areas and tissue structures, making subsequent inference by the model more targeted.

[0050] S50: The text query information is input into the text encoder of a pre-built large language model for text encoding to generate a text token for the text query information. Specifically, in this embodiment of the invention, text encoding can effectively capture the semantics of the user's question, ensuring that the model understands the user's true needs. For example, in a medical scenario, a question such as "Does this image show a lesion?", after encoding, can focus on the potential lesion area in the image.

[0051] S60: The multiple sets of visual tokens and text tokens are fused, and the fused token sequence is input into a pre-built large language model for inference processing to generate a response result corresponding to the text query information. In this embodiment of the invention, the fusion of visual tokens and text tokens enables comprehensive analysis of image and text information, improving the model's ability to handle complex tasks. The fused token sequence is input into the large language model for inference, generating more accurate and relevant response results to meet user needs. Specifically, such as... Figure 5 The above, Figure 5 This is a flowchart illustrating a specific implementation of step S60, which specifically includes the following steps S61-S64:

[0052] S61: Based on the calculated relevance weights between the query vector and each visual token, assign corresponding importance weights to the multiple sets of visual tokens. Specifically, in this embodiment of the invention, by calculating the relevance weights between the query vector and each visual token, importance weights can be assigned to different visual information. This ensures that the model can highlight the most representative and relevant features, thereby improving the accuracy of the analysis. For example, when a user asks, "How does this quarter's sales compare to last quarter?", visual tokens related to sales, such as this quarter's sales data, last quarter's sales data, and their rate of change, will be assigned higher importance weights. This allows these key data to be prioritized in subsequent analysis, generating a more accurate financial analysis report.

[0053] S62: The visual tokens, after being assigned importance weights, are dynamically concatenated with the text tokens to obtain a fused token sequence. Specifically, in this embodiment of the invention, visual tokens are concatenated with text tokens, allowing visual and linguistic information to be combined. This fusion enhances the model's ability to understand information and helps generate more accurate responses. For example, when generating a summary of financial statements, the text token for "sales revenue this quarter" can be concatenated with the corresponding visual token to generate a comprehensive description that includes sales revenue and explanations, making it easier for users to understand and use.

[0054] S63: The fused token sequence is input into the cross-attention decoder of the pre-built large language model for multi-round interactive decoding processing, and the visual evidence region related to the text token is located based on the attention weight distribution generated during the decoding process. Specifically, in this embodiment of the invention, through the cross-attention mechanism, the large language model can dynamically adjust its focus on visual and textual information during the decoding process, thereby better understanding the relationship between the two. This includes the following steps S631-S634:

[0055] S631: The fused token sequence is input to the first decoding layer of the cross-attention decoder for a first attention query, generating an initial attention weight distribution. Specifically, in this embodiment of the invention, the first decoding layer is the input network layer of the cross-attention decoder. By inputting the fused token sequence into the cross-attention decoder, information processing can be performed simultaneously from both visual and textual dimensions. This process helps the model establish a preliminary association between visual and textual information. The generated initial attention weight distribution provides an important basis for subsequent steps, enabling the identification of which visual information is highly correlated with textual information. For example, in a financial scenario, when a user asks "How are the sales this quarter?", the first attention query can generate initial attention weights related to "sales performance," potentially highlighting visual tokens related to sales data, charts, etc., providing direction for subsequent analysis.

[0056] S632: Based on the initial attention weight distribution, candidate visual evidence regions whose correlation with the text token is within a preset threshold range are selected. Specifically, in this embodiment of the invention, selecting candidate visual evidence regions whose correlation with the text token is within a preset threshold range based on the initial attention weight can effectively filter out irrelevant information and focus on the most promising visual features. For example, in a financial scenario, when generating an analysis about "sales performance," visual regions related to the main sales sources may be selected.

[0057] S633: The candidate visual evidence region and text token are input into the second decoding layer of the cross-attention decoder for a second attention query, updating the attention weight distribution. Specifically, in this embodiment of the invention, the second decoding layer is a network layer other than the input layer. Inputting the candidate visual evidence region and text token into the second decoding layer can further update the attention weight distribution and deepen the understanding of the relationship between vision and text. When analyzing the sales of a specific product, the sales data and market trends related to that product can be re-analyzed to update the relevant visual weights, thereby providing more accurate data support for the final sales forecast.

[0058] S634: Based on the updated attention weight distribution, calculate the sum of weights of the image regions corresponding to each visual token, and determine the continuous image regions whose sum of weights exceeds a preset weight threshold as visual evidence regions related to the text token. Specifically, in this embodiment of the invention, by setting continuous image regions whose sum of weights exceeds a preset weight threshold as relevant visual evidence, it is possible to ensure that the generated natural language response has sufficient visual basis. For example, when generating a diagnosis of medical images, if the sum of weights of certain image regions found exceeds a set threshold, these regions will be regarded as important visual evidence related to the diagnosis, helping doctors make more accurate judgments.

[0059] S64: Based on the visual evidence region and the text token, generate a natural language response result containing visual positioning information. Specifically, in this embodiment of the invention, by generating a natural language response containing visual positioning information, the large language model can not only provide an answer but also attach specific visual evidence, enabling users to intuitively understand the analysis results. For example, in a medical imaging report, it can be stated that "a tumor measuring 2cm was found in the right side of the image," along with corresponding image fragments, to help doctors better understand and make judgments.

[0060] As can be seen, in the above scheme, by inputting the image information to be processed into the image pyramid complementation module of the pre-constructed large language model, detail image sets, global image sets, and boundary supplementation image sets are generated. This comprehensively captures important information in the image to be processed, significantly improving the ability to understand the image content. Furthermore, after inputting the generated complementary image set into the multi-scale visual compression module of the large language model for compression processing, the resulting compressed complementary image set significantly reduces the computational cost of the image to be processed, effectively improving the accuracy of the large language model in image recognition.

[0061] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0062] In one embodiment, an image processing apparatus based on a large language model is provided, which corresponds one-to-one with the image processing method based on a large language model described in the above embodiments. For example... Figure 6 As shown, Figure 6 This is a schematic diagram of an image processing device based on a large language model according to an embodiment of the present invention. The image processing device includes an acquisition module 61, an extraction module 62, a compression module 63, a first encoding module 64, a second encoding module 65, and a generation module 66. Detailed descriptions of each functional module are as follows:

[0063] The acquisition module 61 is used to acquire the image information to be processed, as well as the text query information associated with the image information to be processed;

[0064] The extraction module 62 is used to input the image information to be processed into the image pyramid complementation module of the pre-constructed large language model for image feature extraction, and generate at least three sets of complementary image sets with different views, wherein the complementary image sets include at least a detail image set, a global image set, and a boundary supplement image set;

[0065] Compression module 63 is used to input the complementary image set into the multi-scale visual compression module of the pre-constructed large language model for compression processing to obtain the compressed complementary image set;

[0066] The first encoding module 64 is used to input the compressed complementary image set into the visual encoder of the pre-built large language model for visual encoding, and generate multiple sets of visual tokens for the image information to be processed.

[0067] The second encoding module 65 is used to input the text question information into the text encoder of the pre-built large language model for text encoding to generate a text token of the text question information.

[0068] The generation module 66 is used to fuse the multiple sets of visual tokens and text tokens, and input the fused token sequence into a pre-built large language model for reasoning processing to generate a response result corresponding to the text question information.

[0069] In one embodiment, the extraction module 62 is specifically used for:

[0070] The image information to be processed is input into the convolutional layer of the pyramid complement module for uniform grid division processing, generating a group of detailed images containing high-resolution local regions.

[0071] The image information to be processed is input into the pooling layer of the pyramid complement module for downsampling processing to generate a global image group containing global context information;

[0072] The image information to be processed is input into the boundary compensation layer of the pyramid complement module for misaligned mesh division, generating a boundary compensation image group containing compensation information.

[0073] In one embodiment, the compression module 63 is specifically used for:

[0074] Based on the text token corresponding to the text query information and the visual token corresponding to the global image group in the complementary image set, a query vector related to the text query information is generated.

[0075] Calculate the correlation weight between the query vector and each visual token in the detail image group, and filter the visual tokens in the detail image group based on the correlation weight, retaining visual tokens with a weight higher than a preset threshold.

[0076] The visual tokens retained in the detail image group are fused with the visual tokens of the global image group and the boundary compensation image group to obtain a compressed complementary image set.

[0077] In one embodiment, the compression module 63 is further configured to:

[0078] The attention weights between the query vector and each visual token are calculated using an attention mechanism, and the attention weights are normalized to obtain the correlation weights between the query vector and each visual token.

[0079] The visual tokens in the detail image group are sorted according to the relevance weights, and the top K visual tokens by weight are selected for retention, where K is a preset number of visual tokens to retain.

[0080] In one embodiment, the compression module 63 is further configured to:

[0081] The formula for calculating the relevance weights between the query vector and each visual token is as follows:

[0082]

[0083] Where Q represents the query vector, V i Let d represent the i-th visual token in the detail image group, d represent the feature dimension of the visual token, softmax represent the normalization function, and Attention(Q,V) represent the function of the normalization function. i ) represents the relevance weight between the query vector and the i-th visual token.

[0084] In one embodiment, the generation module 66 is specifically used for:

[0085] Based on the calculated correlation weights between the query vectors and each visual token, corresponding importance weights are assigned to the multiple sets of visual tokens.

[0086] The visual tokens, after being assigned importance weights, are dynamically concatenated with the text tokens to obtain the fused token sequence;

[0087] The fused token sequence is input into the cross-attention decoder of a pre-built large language model for multi-round interactive decoding processing, and the visual evidence region related to the text token is located based on the attention weight distribution generated during the decoding process.

[0088] Based on the visual evidence region and the text token, a natural language response result containing visual positioning information is generated.

[0089] In one embodiment, the generation module 66 is further configured to:

[0090] The fused token sequence is input into the first decoding layer of the cross-attention decoder for the first attention query, generating the initial attention weight distribution;

[0091] Based on the initial attention weight distribution, candidate visual evidence regions whose correlation with the text token is within a preset threshold range are selected.

[0092] The candidate visual evidence region and the text token are input into the second decoding layer of the cross-attention decoder for a second attention query, and the attention weight distribution is updated.

[0093] Based on the updated attention weight distribution, the sum of weights of the image regions corresponding to each visual token is calculated, and continuous image regions whose sum of weights exceeds a preset weight threshold are identified as visual evidence regions related to the text token.

[0094] This invention provides an image processing device based on a large language model. By inputting the image information to be processed into the image pyramid complementation module of a pre-constructed large language model, it generates detail image sets, global image sets, and boundary supplementation image sets. This comprehensively captures important information in the image to be processed, significantly improving the ability to understand image content. Furthermore, after inputting the generated complementary image set into the multi-scale visual compression module of the large language model for compression processing, the resulting compressed complementary image set significantly reduces the computational cost of the image to be processed, effectively improving the accuracy of the large language model in image recognition.

[0095] Specific limitations regarding the image processing device based on large language models can be found in the limitations of the image processing method based on large language models mentioned above, and will not be repeated here. Each module in the aforementioned image processing device based on large language models can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0096] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 7 As shown, Figure 7 This is a schematic diagram of a computer device according to an embodiment of the present invention. The computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a server-side image processing method based on a large language model.

[0097] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 8 As shown, Figure 8This is another schematic diagram of a computer device according to an embodiment of the present invention. The computer device includes a processor, memory, network interface, display screen, and input device connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements client-side functions or steps of an image processing method based on a large language model.

[0098] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:

[0099] Obtain the image information to be processed, as well as the text query information associated with the image information to be processed;

[0100] The image information to be processed is input into the image pyramid complementation module of the pre-constructed large language model for image feature extraction, generating at least three sets of complementary image sets with different views, wherein the complementary image sets include at least a detail image set, a global image set, and a boundary supplement image set;

[0101] The complementary image set is input into the multi-scale visual compression module of the pre-built large language model for compression processing to obtain the compressed complementary image set.

[0102] The compressed complementary image set is input into the visual encoder of the pre-built large language model for visual encoding, generating multiple sets of visual tokens for the image information to be processed.

[0103] The text query information is input into the text encoder of a pre-built large language model for text encoding to generate a text token for the text query information;

[0104] The multiple sets of visual tokens and text tokens are fused together, and the fused token sequence is input into a pre-built large language model for inference processing to generate a response result corresponding to the text question information.

[0105] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0106] Obtain the image information to be processed, as well as the text query information associated with the image information to be processed;

[0107] The image information to be processed is input into the image pyramid complementation module of the pre-constructed large language model for image feature extraction, generating at least three sets of complementary image sets with different views, wherein the complementary image sets include at least a detail image set, a global image set, and a boundary supplement image set;

[0108] The complementary image set is input into the multi-scale visual compression module of the pre-built large language model for compression processing to obtain the compressed complementary image set.

[0109] The compressed complementary image set is input into the visual encoder of the pre-built large language model for visual encoding, generating multiple sets of visual tokens for the image information to be processed.

[0110] The text query information is input into the text encoder of a pre-built large language model for text encoding to generate a text token for the text query information;

[0111] The multiple sets of visual tokens and text tokens are fused together, and the fused token sequence is input into a pre-built large language model for inference processing to generate a response result corresponding to the text question information.

[0112] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0113] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0114] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0115] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A large language model-based image processing method, characterized in that, The method comprises the following steps: acquiring image information to be processed and text question information associated with the image information to be processed; inputting the image information to be processed into an image pyramid complementary module of a pre-constructed large language model to extract image features, and generating at least three groups of complementary image sets with different fields of view, wherein the complementary image sets at least include a detail image group, a global image group and a boundary supplementary image group; inputting the complementary image sets into a multi-scale visual compression module of the pre-constructed large language model for compression processing to obtain compressed complementary image sets; inputting the compressed complementary image sets into a visual encoder of the pre-constructed large language model for visual coding to generate multiple groups of visual tokens of the image information to be processed; inputting the text question information into a text encoder of the pre-constructed large language model for text coding to generate text tokens of the text question information; fusing the multiple groups of visual tokens and the text tokens, and inputting the fused token sequence into the pre-constructed large language model for reasoning processing to generate a response result corresponding to the text question information.

2. The large language model based image processing method according to claim 1, characterized in that, The step of inputting the image information to be processed into the image pyramid complementary module of the pre-constructed large language model to extract image features, and generating at least three groups of complementary image sets with different fields of view, wherein the complementary image sets at least include a detail image group, a global image group and a boundary supplementary image group, comprises the following steps: inputting the image information to be processed into a convolutional layer of the pyramid complementary module for uniform grid division processing to generate the detail image group containing high-resolution local regions; inputting the image information to be processed into a pooling layer of the pyramid complementary module for down-sampling processing to generate the global image group containing global context information; inputting the image information to be processed into a boundary compensation layer of the pyramid complementary module for staggered grid division to generate the boundary compensation image group containing compensation information.

3. The large language model based image processing method according to claim 1, characterized in that, The step of inputting the complementary image sets into the multi-scale visual compression module of the pre-constructed large language model for compression processing to obtain compressed complementary image sets comprises the following steps: generating a query vector related to the text question information based on the text token corresponding to the text question information and the visual token corresponding to the global image group in the complementary image sets; calculating the correlation weight between the query vector and each visual token in the detail image group, and screening the visual tokens in the detail image group based on the correlation weight to retain the visual tokens with a weight higher than a preset threshold; fusing the retained visual tokens in the detail image group with the visual tokens of the global image group and the boundary compensation image group to obtain the compressed complementary image sets.

4. The large language model based image processing method according to claim 3, characterized in that, The step of calculating the correlation weight between the query vector and each visual token in the detail image group, and screening the visual tokens in the detail image group based on the correlation weight to retain the visual tokens with a weight higher than a preset threshold comprises the following steps: calculating the attention weight between the query vector and each visual token through an attention mechanism, and performing normalization processing on the attention weight to obtain the correlation weight between the query vector and each visual token; According to the correlation weight, the visual tokens in the detail image group are sorted, and the top K visual tokens in the weight are selected for reservation, where K is a preset number of visual token reservation.

5. The large language model based image processing method according to claim 4, characterized in that, The attention mechanism is used to calculate the attention weight between the query vector and each visual token, and the attention weight is normalized to obtain the correlation weight between the query vector and each visual token, including: The formula for calculating the correlation weight between the query vector and each visual token is: wherein Q represents the query vector, V i represents the i-th visual token in the detail image group, d represents the feature dimension of the visual token, softmax represents a normalization function, and Attention(Q, V i ) represents the relevance weight between the query vector and the i-th visual token.

6. The large language model based image processing method according to claim 1, characterized in that, The multi-group visual tokens and the text tokens are fused, and the fused token sequence is input into a pre-built large language model for inference processing to generate a response result corresponding to the text question information, including: Based on the calculated correlation weight between the query vector and each visual token, the multi-group visual tokens are assigned corresponding importance weights; The visual tokens with importance weights are dynamically spliced with the text tokens to obtain a fused token sequence; The fused token sequence is input into a cross-attention decoder of a pre-built large language model for multi-round interactive decoding processing, and based on the attention weight distribution generated during decoding, a visual evidence area related to the text token is located; Based on the visual evidence area and the text token, a natural language response result containing visual positioning information is generated.

7. The large language model based image processing method according to claim 6, characterized in that, The fused token sequence is input into a cross-attention decoder of a pre-built large language model for multi-round interactive decoding processing, and based on the attention weight distribution generated during decoding, a visual evidence area related to the text token is located, including: The fused token sequence is input into a first decoding layer of the cross-attention decoder for first attention query to generate an initial attention weight distribution; Based on the initial attention weight distribution, candidate visual evidence areas with a correlation degree within a preset threshold range with the text token are screened out; The candidate visual evidence areas and the text token are input into a second decoding layer of the cross-attention decoder for second attention query to update the attention weight distribution; Based on the updated attention weight distribution, the weight sum of the image area corresponding to each visual token is calculated, and the continuous image area with a weight sum exceeding a preset weight threshold is determined as the visual evidence area related to the text token.

8. A large language model-based image processing apparatus, characterized by comprising: including: An acquisition module is configured to acquire image information to be processed and text question information associated with the image information to be processed; An extraction module is configured to input the image information to be processed into an image pyramid complementary module of a pre-built large language model to perform image feature extraction and generate at least three groups of complementary image sets with different fields of view, wherein the complementary image sets at least include a detail image group, a global image group, and a boundary supplementary image group; A compression module is configured to input the complementary image sets into a multi-scale visual compression module of a pre-built large language model for compression processing to obtain compressed complementary image sets; The first encoding module is configured to input the compressed complementary image set into a visual encoder of a pre-built large language model for visual encoding to generate a plurality of groups of visual tokens of the image information to be processed. The second encoding module is configured to input the text question information into a text encoder of the pre-built large language model for text encoding to generate text tokens of the text question information. The generation module is configured to perform fusion processing on the plurality of groups of visual tokens and the text tokens, and input a fused token sequence into the pre-built large language model for inference processing to generate a response result corresponding to the text question information.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the steps of the image processing method based on the large language model according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 9. The computer program is executed by the processor to implement the steps of the image processing method based on the large language model according to any one of claims 1 to 7.

Images