Report generation method and device based on thinking chain and large model

Abstract

The invention discloses a report generation method based on a thinking chain and a large model, and belongs to the technical field of image processing. The method comprises the following steps: inputting image information and text information into a multi-modal hybrid encoder-decoder model trained by adopting a multi-task training method to obtain a target text report; the multi-modal hybrid encoder-decoder model comprises an encoding module, an alignment module, a thinking chain reasoning module and a decoding module, and the encoding module is used for performing feature extraction on image information and text information to obtain image features and text features; the alignment module is used for performing image-text alignment on the image features and the text features to obtain an image-text alignment result; the thinking chain reasoning module is used for reasoning an image-text alignment result to generate a thinking chain; and the decoding module is used for decoding the image-text alignment result and the thinking chain to obtain a target text report. According to the method, the continuity and accuracy of text report generation are improved.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and in particular relates to a report generation method and device based on thought chains and large models. Background Art

[0002] With the rapid development of digital pathology technology, deep learning-based pathology image analysis has made significant progress in medical tasks such as disease screening, tumor grading, and prognosis assessment. However, the extraction of complex pathological features and the accurate expression of diagnostic logic are still technical bottlenecks that need to be overcome. In the field of disease diagnosis, tissue heterogeneity and cell morphological diversity can interfere with the deep learning model's capture of key pathological features, resulting in a reduction in the significance of the diagnostic basis. In terms of report generation, the lack of semantic relevance of multidimensional information in pathological images, such as the spatial connection between cell distribution patterns and tissue structure, can seriously affect the logical rigor of diagnostic conclusions. In remote consultation scenarios, alignment deviations and semantic gaps in cross-modal data can reduce the interpretability of visual diagnostic evidence, making it difficult for generated reports to support clinical decision-making. Therefore, improving the model's ability to deconstruct pathological features and the level of deduction of diagnostic logic is a core challenge to ensure the accuracy of intelligent diagnosis and the credibility of reports.

[0003] Existing pathology report generation methods mainly use a hybrid architecture of visual encoders and sequence decoders. Although they can achieve basic image-text conversion, they still have defects such as single feature representation and weak semantic association when faced with complex pathological features and multi-dimensional diagnostic logic. Traditional cross-modal alignment methods based on attention mechanisms are limited by shallow feature interactions and find it difficult to establish pathological-level semantic mappings between histological features and diagnostic terms. In addition, in view of the multi-scale feature dependencies and logical chain derivation requirements in the diagnostic reasoning process, existing generation models are easily interfered by noise features, resulting in the omission of key diagnostic elements or frequent causal inversion. In recent years, although Transformer-based pre-training models can improve the fluency of text generation, their general architecture has not been specially designed for the feature decoupling and logical chain modeling unique to pathological diagnosis. Existing methods have problems with insufficient feature extraction, poor logic and coherence, and low accuracy when generating diagnostic reports. Summary of the Invention

[0004] The present application aims to solve at least one of the technical problems existing in the prior art. To this end, the present application proposes a report generation method and device based on thought chain and large model, which improves the coherence and accuracy of text report generation.

[0005] In a first aspect, the present application provides a report generation method based on thought chain and big model, the method comprising:

[0006] Obtain image information and text information;

[0007] Inputting the image information and text information into a multimodal hybrid encoder-decoder model trained using a multi-task training method to obtain a target text report;

[0008] The multimodal hybrid encoder-decoder model includes an encoding module, an alignment module, a thought chain reasoning module and a decoding module, and the encoding module includes a visual encoder and a text encoder;

[0009] The encoding module is used to extract features from the image information and text information to obtain image features and text features;

[0010] The alignment module is used to perform image-text alignment on the image features and text features to obtain an image-text alignment result;

[0011] The thought chain reasoning module is used to reason about the image-text alignment result and generate a thought chain;

[0012] The decoding module is used to decode the image-text alignment result and the thought chain to obtain a target text report.

[0013] According to one embodiment of the present application, extracting features from the image information and text information to obtain image features and text features includes:

[0014] After resizing and normalizing the image information, the image information is input into a visual encoder to obtain image features;

[0015] After word segmentation and vectorization operations are performed on the text information, it is input into a text encoder to obtain text features.

[0016] According to one embodiment of the present application, performing image-text alignment on the image features and the text features to obtain an image-text alignment result includes:

[0017] Calculating the similarity between the image features and the text features based on the attention mechanism;

[0018] The image features and text features are matched based on the similarity to obtain an image-text alignment result.

[0019] According to one embodiment of the present application, reasoning on the image-text alignment result to generate a thought chain includes:

[0020] Inputting the image-text alignment result into the thought chain reasoning module to obtain the association information between the image and the text;

[0021] Based on the association information, a thought chain is generated.

[0022] According to one embodiment of the present application, the training process of the multimodal hybrid encoder-decoder model includes:

[0023] Build a pre-defined multimodal hybrid encoder-decoder model;

[0024] Constructing a data set based on multiple image data and text reports corresponding to the image data;

[0025] Based on the data set, a preset multimodal hybrid encoder-decoder model is trained using a multi-task training method to obtain a trained multimodal hybrid encoder-decoder model;

[0026] The multiple tasks include contrastive learning tasks, image-text matching tasks, and text generation tasks.

[0027] According to one embodiment of the present application, the training process of the text generation task includes:

[0028] Input the dataset into a preset multimodal hybrid encoder-decoder model to obtain the probability distribution of the next word at each time step;

[0029] Adjusting the probability distribution according to a smoothing coefficient and a vocabulary size, and calculating a cross entropy loss value at each time step based on the adjusted probability distribution and the actual distribution;

[0030] Based on the cross entropy loss value at each time step, the language modeling loss is obtained;

[0031] Based on the language modeling loss, the preset multimodal hybrid encoder-decoder model is trained.

[0032] According to one embodiment of the present application, the multimodal hybrid encoder-decoder model includes one of a BLIP model, a Q-Former-Transformer model and a Vision Transformer-Transformer model.

[0033] In a second aspect, the present application provides a report generation device based on thought chain and large model, the device comprising:

[0034] An acquisition module, used to acquire image information and text information;

[0035] a processing module, configured to input the image information and text information into a multimodal hybrid encoder-decoder model trained using a multi-task training method to obtain a target text report;

[0036] The multimodal hybrid encoder-decoder model includes an encoding module, an alignment module, a thought chain reasoning module and a decoding module, and the encoding module includes a visual encoder and a text encoder;

[0037] The encoding module is used to extract features from the image information and text information to obtain image features and text features;

[0038] The alignment module is used to perform image-text alignment on the image features and text features to obtain an image-text alignment result;

[0039] The thought chain reasoning module is used to reason about the image-text alignment result and generate a thought chain;

[0040] The decoding module is used to decode the image-text alignment result and the thought chain to obtain a target text report.

[0041] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the report generation method based on the thinking chain and the large model as described in the first aspect above is implemented.

[0042] In a fourth aspect, the present application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the report generation method based on thought chains and large models as described in the first aspect above.

[0043] In a fifth aspect, the present application provides a chip comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the report generation method based on thought chains and large models as described in the first aspect.

[0044] In a sixth aspect, the present application provides a computer program product, comprising a computer program, which, when executed by a processor, implements the report generation method based on thought chains and large models as described in the first aspect above.

[0045] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become obvious from the description below, or will be learned through practice of the present application.

[0046] The report generation method based on thought chain and large model provided by the present invention has the following beneficial effects compared with the prior art:

[0047] (1) The present invention obtains image information and text information and inputs them into a trained multimodal hybrid encoder-decoder model, combines the encoding module, alignment module, thought chain reasoning module and decoding module to generate a target text report, thereby effectively improving the accuracy of image-text information processing and reasoning. By extracting image features and text features, performing image-text alignment and thought chain reasoning on them, the target text report related to the image and text information can be generated more accurately. By adding a thought chain reasoning module, the model gradually generates a logical chain of "observation → analysis → diagnosis", thereby improving the model's understanding and reasoning ability of semantics, and improving the coherence, logic and accuracy of text report generation.

[0048] (2) The present invention constructs a preset multimodal hybrid encoder-decoder model, builds a data set based on multiple image data and text reports corresponding to the image data, and trains the model using a multi-task training method, thereby effectively improving the correlation and reasoning ability between images and texts. Through multi-task training of comparative learning tasks, image-text matching tasks, and text generation tasks, it helps to enhance the deep fusion between images and texts, improve the accuracy of understanding image information and text information, and improve the logic and accuracy of target text report generation.

[0049] (3) The present invention effectively improves the training efficiency of the text generation task by inputting the dataset into a preset multimodal hybrid encoder-decoder model and calculating the cross-entropy loss value of each time step according to the adjusted probability distribution. By comparing the probability distribution after adjusting the smoothing coefficient and vocabulary size with the actual distribution, the model's language modeling ability when generating text is optimized, enabling it to better understand and process the correlation between images and text, and generate more accurate and fluent text reports. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0051] Figure 1 This is one of the flow charts of the report generation method based on the thought chain and the large model provided in the embodiment of the present application;

[0052] Figure 2 This is the second flow chart of the report generation method based on the thought chain and the large model provided in the embodiment of the present application;

[0053] Figure 3 This is a flow chart of feature extraction provided by an embodiment of the present application;

[0054] Figure 4 This is a schematic diagram of the structure of a report generation device based on thought chain and large model provided in an embodiment of the present application;

[0055] Figure 5 It is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0056] The following will be combined with the accompanying drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the embodiments described are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of this application.

[0057] The terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of this application can be implemented in an order other than that illustrated or described herein, and that the objects distinguished by "first," "second," and the like are generally of the same type, and do not limit the number of objects; for example, the first object can be one or more. In addition, the term "and / or" in the specification and claims refers to at least one of the connected objects, and the character " / " generally indicates that the objects connected are in an "or" relationship.

[0058] Below, in combination with the accompanying drawings, the report generation method based on thought chain and big model, the report generation device based on thought chain and big model, the electronic device and the readable storage medium provided in the embodiments of the present application are described in detail through specific embodiments and their application scenarios.

[0059] Among them, the report generation method based on the thought chain and the large model can be applied to the terminal, and can be specifically executed by the hardware or software in the terminal.

[0060] The terminal includes, but is not limited to, a portable communication device such as a mobile phone or tablet computer having a touch-sensitive surface (e.g., a touch screen display and / or a touch pad). It should also be understood that, in some embodiments, the terminal may not be a portable communication device, but a desktop computer having a touch-sensitive surface (e.g., a touch screen display and / or a touch pad).

[0061] In the following embodiments, a terminal including a display and a touch-sensitive surface is described. However, it should be understood that the terminal may include one or more other physical user interface devices such as a physical keyboard, a mouse, and a joystick.

[0062] The embodiment of the present application provides a report generation method based on thought chains and large models. The execution subject of the report generation method based on thought chains and large models can be an electronic device or a functional module or functional entity in the electronic device that can implement the report generation method based on thought chains and large models. The electronic devices mentioned in the embodiment of the present application include but are not limited to mobile phones, tablet computers, computers, cameras and wearable devices, etc. The report generation method based on thought chains and large models provided in the embodiment of the present application is explained below using electronic devices as the execution subject as an example.

[0063] With the continuous advancement of medical imaging technology, digital pathology has become an essential component of modern medical diagnosis. Pathology images acquired through high-resolution scanners and other devices provide physicians with unprecedented levels of detailed information to support disease diagnosis and treatment planning. However, efficiently and accurately extracting useful information from these massive amounts of digital pathology images and converting it into structured, easily understandable text descriptions remains a key challenge in current research and technological development.

[0064] Digital pathology involves the use of digital means to record, manage, analyze, and share pathology information. It not only significantly improves diagnostic efficiency but also promotes the development of telemedicine. However, traditional pathology image analysis methods, which rely on manual interpretation, are subject to significant subjectivity and time-consuming. Therefore, the development of technologies that can automatically analyze pathology images and generate reports is of great practical significance.

[0065] In medicine, particularly pathology, image-text alignment refers to the process of converting visual information (such as cell morphology and tissue structure on pathology slides) into precise textual descriptions. This step is crucial for ensuring the accuracy of medical records. Traditional rule-based methods struggle to cope with complex image content, while the recent rise of deep learning technology has demonstrated tremendous potential.

[0066] Although existing AI technologies have improved the efficiency and accuracy of pathology image analysis to a certain extent, they still face many challenges when dealing with complex and changing pathological conditions. Traditional pathology image analysis relies primarily on manual evaluation by experienced pathologists. In recent years, with the development of deep learning technology, AI-based automatic pathology report generation methods have gradually become a research hotspot. However, existing methods have the following problems:

[0067] 1. Insufficient feature extraction: Existing methods usually use pre-trained convolutional neural networks to extract pathological image features, which makes it difficult to capture key information related to pathological diagnosis in the image.

[0068] 2. Poor logic and coherence: The pathology report texts generated by existing methods often lack logic and coherence, making it difficult to accurately describe the lesion characteristics in pathology images.

[0069] Figure 1 This is one of the flow charts of the report generation method based on thought chain and large model provided in the embodiment of the present application, such as Figure 1 As shown, the report generation method based on thought chain and big model includes: step 110 and step 120.

[0070] Step 110: Acquire image information and text information;

[0071] Step 120: input the image information and text information into a multimodal hybrid encoder-decoder model trained using a multi-task training method to obtain a target text report;

[0072] The multimodal hybrid encoder-decoder model includes an encoding module, an alignment module, a thought chain reasoning module and a decoding module, and the encoding module includes a visual encoder and a text encoder;

[0073] The encoding module is used to extract features from the image information and text information to obtain image features and text features;

[0074] The alignment module is used to perform image-text alignment on the image features and text features to obtain an image-text alignment result;

[0075] The thought chain reasoning module is used to reason about the image-text alignment result and generate a thought chain;

[0076] The decoding module is used to decode the image-text alignment result and the thought chain to obtain a target text report.

[0077] Figure 2 This is the second flow chart of the report generation method based on thought chain and large model provided in the embodiment of the present application, such as Figure 2 As shown in the figure, we first obtain image information and text information, input the image information and text information into the multimodal hybrid encoder-decoder model trained using the multi-task training method, and obtain the target text report through the following process:

[0078] (1) Preprocessing and feature extraction of image information and text information to obtain image features and text features;

[0079] (2) Image-text alignment;

[0080] The image features and text features are aligned so that they are in the same semantic space.

[0081] (3) Generation of thought chains;

[0082] Furthermore, based on the image-text alignment results, a logically coherent thought chain targeting the image analysis process is generated through the thought chain reasoning module.

[0083] (4) Report generation

[0084] Finally, the image-text alignment results and thought chains are decoded by the decoding module to obtain the target text report. For example, the target text report is a pathology report, which not only contains a detailed description of the lesion and the diagnosis conclusion, but also provides suggestions for further examination or treatment.

[0085] According to the image and text information generation method based on the multimodal hybrid encoder-decoder model provided in the embodiment of the present application, by acquiring image information and text information and inputting them into the trained multimodal hybrid encoder-decoder model, the target text report is generated by combining the encoding module, alignment module, thought chain reasoning module and decoding module, thereby effectively improving the accuracy of image-text information processing and reasoning. By performing feature extraction, image-text alignment and thought chain reasoning on image features and text features, the target text report related to the image and text information can be generated more accurately. By adding the thought chain reasoning module, the model gradually generates a logical chain of "observation → analysis → diagnosis", thereby improving the model's understanding and reasoning ability of semantics, and improving the coherence, logic and accuracy of text report generation.

[0086] In some embodiments, extracting features from the image information and text information to obtain image features and text features includes:

[0087] After resizing and normalizing the image information, the image information is input into a visual encoder to obtain image features;

[0088] After word segmentation and vectorization operations are performed on the text information, it is input into a text encoder to obtain text features.

[0089] Figure 3 This is a flow chart of feature extraction provided by the embodiment of the present application, such as Figure 3 As shown, image information and text information are input and preprocessed respectively, wherein the image preprocessing includes adjusting the image size and resolution, normalizing the image pixel value, removing noise, and enhancing the image contrast or brightness. The preprocessed image is then input into the visual encoder to obtain image features.

[0090] Text preprocessing includes word segmentation (segmenting text into words or phrases), stop word removal (removing common but meaningless words), stemming or lemmatization (converting words to their basic form), and vectorization (converting text into numerical vectors, such as using word embedding). The preprocessed text is then input into a text encoder to obtain text features.

[0091] In this embodiment, the image information is resized and normalized and then input into the visual encoder for feature extraction, and the text information is segmented and vectorized and then input into the text encoder for feature extraction. This effectively improves the representation capability of image and text information, makes the extraction of image features and text features more accurate, provides a more reliable foundation for subsequent image-text alignment and reasoning, and can better support the generation of target text reports.

[0092] In some embodiments, performing image-text alignment on the image features and the text features to obtain an image-text alignment result includes:

[0093] Calculating the similarity between the image features and the text features based on the attention mechanism;

[0094] The image features and text features are matched based on the similarity to obtain an image-text alignment result.

[0095] It is easy to understand that the image-text alignment result can be obtained through the attention mechanism. First, the attention mechanism is used to calculate the similarity between image features and text features, and establish an association between image features and text features, so that each description of text information can find the most relevant image information. Then, the image features and text features are aligned based on the similarity to obtain the image-text alignment result.

[0096] In this embodiment, the image-text alignment result is obtained by calculating the similarity between image features and text features based on the attention mechanism, and matching the image features and text features based on the similarity, which effectively improves the correlation between image and text information, can more accurately align the semantic relationship between image and text, and improves the accuracy of multimodal information fusion.

[0097] In some embodiments, reasoning about the image-text alignment result to generate a thought chain includes:

[0098] Inputting the image-text alignment result into the thought chain reasoning module to obtain the association information between the image and the text;

[0099] Based on the association information, a thought chain is generated.

[0100] For example, the image-text alignment result is input into the thought chain reasoning module, which begins to simulate the diagnostic thinking process of human experts, gradually identifying and describing the lesion area, analyzing its morphological characteristics, inferring the type and severity of the lesion, obtaining the association information between the image and the text, and generating a series of structured text descriptions based on the association information to obtain a thought chain.

[0101] In this embodiment, by inputting the image-text alignment result into the thought chain reasoning module, the association information between the image and the text is obtained, and a thought chain is generated based on the association information, which effectively improves the understanding and association reasoning ability between the image and the text, can more accurately capture the deep connection between the image and text content, enhance the fusion effect of multimodal data, and improve the accuracy and quality of subsequent target text report generation.

[0102] In some embodiments, the training process of the multimodal hybrid encoder-decoder model includes:

[0103] Build a pre-defined multimodal hybrid encoder-decoder model;

[0104] Constructing a data set based on multiple image data and text reports corresponding to the image data;

[0105] Based on the data set, a preset multimodal hybrid encoder-decoder model is trained using a multi-task training method to obtain a trained multimodal hybrid encoder-decoder model;

[0106] The multiple tasks include contrastive learning tasks, image-text matching tasks, and text generation tasks.

[0107] For example, the dataset is pathological images and their diagnostic reports, which include various types of pathological images (such as tissue section images) and their corresponding diagnostic reports. It not only provides rich visual information, but also includes detailed text descriptions, such as lesion characteristics, diagnostic conclusions, etc. In addition to basic image and text descriptions, the correspondence between different areas in the image and the corresponding text descriptions should also be collected or generated, which helps to improve the accuracy of the model in understanding the image content.

[0108] During the training process, contrastive learning loss and cross-entropy loss can be used to optimize the model. Contrastive learning loss is used for image feature extraction and image-text alignment, so that image and text features can be effectively aligned in a shared space, enhancing the model's ability to understand the complex relationship between images and text. Cross-entropy loss is used for thought chain generation and report generation. By minimizing the difference between the predicted text and the real text, the quality of the generated text is improved, making it as close as possible to the real pathology report.

[0109] Optionally, weight decay can be used with the AdamW optimizer to reduce overfitting. Gradually reducing the learning rate as training progresses can accelerate convergence and reduce oscillations that occur in the later stages of training, thereby improving the overall performance of the model.

[0110] It is worth noting that the multimodal hybrid encoder-decoder model is trained using multi-task training, which includes contrastive learning tasks, image-text matching tasks, and text generation tasks. In addition, tasks specific to pathological analysis can be added, such as identifying specific types of cell abnormalities and assessing tumor grades, so that the model can better understand and generate reports for specific pathological conditions.

[0111] In some embodiments, the contrastive learning task is matching medical images with case texts. Through the contrastive learning task, the model can identify which images and texts match each other. The loss function used in the contrastive learning task is image-text contrast loss. The training process includes the following steps:

[0112] (1) Use the encoder to extract feature representations of all images and texts in the batch separately;

[0113] (2) Update the weight of the momentum encoder according to the current encoder state;

[0114] (3) The momentum encoder is used to process the same batch of data to produce more stable feature representations.

[0115] Suppose we have a batch of N image-text pairs, for each image i and its corresponding text t, we can get their feature representation v i and u t ,. The momentum encoder is used to generate smoother versions of these features, denoted as and

[0116] (4) Based on the similarity formula, calculate the similarity score between the positive sample and the negative sample pair.

[0117] Positive sample pairs: For each pair of matching images and text, calculate the similarity score, usually achieved by dot product, that is, and

[0118] Negative sample pairs: For non-matching image-text pairs, similarity scores are also calculated to form a similarity matrix.

[0119] (5) Loss calculation and optimization: The loss is calculated based on the image-text contrast loss, and the model parameters are adjusted through the back-propagation algorithm to minimize the loss.

[0120] For image i and its positive sample text i, the image-text contrast loss is calculated as follows:

[0121]

[0122] Among them, τ is the temperature parameter, L i is the image-text contrast loss.

[0123] In some embodiments, the training process of the image-text matching task is as follows:

[0124] (1) Feature extraction: Use image encoder and text encoder to extract feature representations of images and text respectively;

[0125] (2) Feature fusion and score calculation: The extracted features are fused and the score of each image-text pair is calculated through a linear layer;

[0126] First, we obtain the feature representations of the image and text from the image encoder and the text encoder, and fuse them through some methods (such as dot product, splicing, etc.) to form a joint representation. Then, we use a linear layer to map the joint representation to a scalar value, which represents the probability that the image-text pair belongs to a positive sample. Let s i,t is the score after pairing image i and text t, then: i,t =W T [v i ;u t ]+b

[0127] Among them, W and b are the weight matrix and bias term of the linear layer respectively, [v i ;u t ] represents the image feature v i and text features u t Some form of combination (such as direct splicing).

[0128] (3) Loss calculation: Calculate the image-text matching loss for each sample based on the actual matching situation (label y) and the model prediction score;

[0129] For each image-text pair, if they are a match, the label y = 1; if they are not a match, y = 0. Therefore, the image-text matching loss is calculated as follows:

[0130] L ITM =-ylog(σ(s i,t ))-(1-y)log(1-σ(s i,t ))

[0131] Among them, σ represents the sigmoid function, L ITM is the image-text matching loss.

[0132] (4) Hard negative sample selection: Select the most difficult negative samples within a batch based on the current prediction results of the model.

[0133] In order to enhance the learning effect of the model, the negative samples that are currently the most difficult for the model to distinguish (that is, samples that the model gives high scores but are actually mismatched) are selected to force the model to learn more complex patterns. Finally, the model parameters are updated based on the sum of the losses of all samples (including hard negative samples).

[0134] In this embodiment, by constructing a preset multimodal hybrid encoder-decoder model, and constructing a data set based on multiple image data and text reports corresponding to the image data, the model is trained using a multi-task training method, which effectively improves the correlation and reasoning ability between images and texts. Through multi-task training of comparative learning tasks, image-text matching tasks, and text generation tasks, it helps to enhance the deep fusion between images and texts, improve the accuracy of understanding image information and text information, and improve the logic and accuracy of target text report generation.

[0135] In some embodiments, the training process of the text generation task includes:

[0136] Input the dataset into a preset multimodal hybrid encoder-decoder model to obtain the probability distribution of the next word at each time step;

[0137] Adjusting the probability distribution according to a smoothing coefficient and a vocabulary size, and calculating a cross entropy loss value at each time step based on the adjusted probability distribution and the actual distribution;

[0138] Based on the cross entropy loss value at each time step, the language modeling loss is obtained;

[0139] Based on the language modeling loss, the preset multimodal hybrid encoder-decoder model is trained.

[0140] It is easy to understand that the training process of the text generation task includes the following steps:

[0141] (1) Input the text sequence in the dataset into the preset multimodal hybrid encoder-decoder model, obtain the probability distribution of the model for the next predicted word at each time step t, and calculate the original cross entropy loss for each time step based on the probability distribution and the next real target word at each time step t. For example, for a given text sequence {w1,w2,...,w T}, T is the sequence length, predict the next word w at each position t t+1 The probability distribution P(w t+1 |w1,w2,...,w t), the original cross entropy loss at each time step is calculated as follows:

[0142]

[0143] in, is the next real target word, L CE is the original cross entropy loss.

[0144] (2) According to the preset smoothing coefficient and vocabulary size, the probability estimation of the target word and the predicted word is adjusted. The probability of the target word is no longer absolutely 1, but is adjusted to Where ε is the label smoothing coefficient (e.g. 0.1), V is the vocabulary size, and the probability of the predicted word is adjusted from 0 to

[0145] (3) Use the adjusted probability to recalculate the loss value of each time step to obtain the cross entropy loss. The calculation formula is as follows:

[0146]

[0147] Among them, L LS is the cross entropy loss.

[0148] (4) Based on the cross entropy loss value of each time step, the losses of all time steps are added and averaged to obtain the language modeling loss. The calculation formula is as follows:

[0149]

[0150] Where L is the language modeling loss.

[0151] In this embodiment, by inputting the dataset into a preset multimodal hybrid encoder-decoder model and calculating the cross-entropy loss value of each time step according to the adjusted probability distribution, the training efficiency of the text generation task is effectively improved. By combining the smoothing coefficient and the probability distribution after the vocabulary size adjustment and comparing it with the actual distribution, the language modeling ability of the model when generating text is optimized, enabling it to better understand and process the correlation between images and text, and generate more accurate and fluent text reports.

[0152] In some embodiments, the multimodal hybrid encoder-decoder model comprises one of a BLIP model, a Q-Former-Transformer model, and a Vision Transformer-Transformer model.

[0153] Optionally, the multimodal hybrid encoder-decoder model can be a BLIP (Bootstrapping Language-Image Pretraining) model. The BLIP model can be enhanced with captioning and filtering techniques. A caption generator is used to create synthetic captions (i.e., preliminary descriptions) for images, and then filters are used to remove inaccurate or noisy descriptions, which helps improve the quality of the final report.

[0154] Optionally, the multimodal hybrid encoder-decoder model can be a Q-Former-Transformer model, where Q-Former is the encoder and Transformer is the decoder.

[0155] Optionally, the multimodal hybrid encoder-decoder model can be a Vision Transformer-Transformer model, where Vision Transformer is the encoder and Transforme is the decoder.

[0156] It should be noted that different models can be selected as the multimodal hybrid encoder-decoder model according to the actual application scenario, and the embodiments of this application are not limited thereto.

[0157] In this embodiment, by adopting one of the BLIP model, Q-Former-Transformer model or VisionTransformer-Transformer model, the most suitable model architecture can be flexibly selected according to the actual application scenario, thereby improving the flexibility and accuracy of the text generation task and enhancing the adaptability of the model in different application scenarios.

[0158] The report generation method based on thought chain and large model provided in the embodiment of the present application can be executed by a report generation device based on thought chain and large model. In the embodiment of the present application, the report generation method based on thought chain and large model is executed by a report generation device based on thought chain and large model as an example to illustrate the report generation device based on thought chain and large model provided in the embodiment of the present application.

[0159] The embodiment of the present application also provides a report generation device based on thought chain and large model, such as Figure 4 As shown, the report generation device based on thought chain and large model includes: an acquisition module 410 and a processing module 420.

[0160] An acquisition module 410 is used to acquire image information and text information;

[0161] A processing module 420 is configured to input the image information and text information into a multimodal hybrid encoder-decoder model trained using a multi-task training method to obtain a target text report;

[0162] The multimodal hybrid encoder-decoder model includes an encoding module, an alignment module, a thought chain reasoning module and a decoding module, and the encoding module includes a visual encoder and a text encoder;

[0163] The encoding module is used to extract features from the image information and text information to obtain image features and text features;

[0164] The alignment module is used to perform image-text alignment on the image features and text features to obtain an image-text alignment result;

[0165] The thought chain reasoning module is used to reason about the image-text alignment result and generate a thought chain;

[0166] The decoding module is used to decode the image-text alignment result and the thought chain to obtain a target text report.

[0167] According to the report generation method based on thought chain and large model provided in the embodiment of the present application, by acquiring image information and text information and inputting them into the trained multimodal hybrid encoder-decoder model, the encoding module, alignment module, thought chain reasoning module and decoding module are combined to generate the target text report, thereby effectively improving the accuracy of image-text information processing and reasoning. By performing feature extraction, image-text alignment and thought chain reasoning on image features and text features, the target text report related to the image and text information can be generated more accurately. By adding the thought chain reasoning module, the model gradually generates a logical chain of "observation → analysis → diagnosis", which improves the model's understanding and reasoning ability of semantics, and improves the coherence, logic and accuracy of text report generation.

[0168] The report generation device based on thought chain and large model provided in the embodiment of the present application can achieve Figures 1 to 3 To avoid repetition, the various processes implemented in the embodiment of the report generation method based on the thinking chain and the large model will not be described here.

[0169] In some embodiments, as Figure 5 As shown, an embodiment of the present application also provides an electronic device 500, including a processor 501, a memory 502, and a computer program stored in the memory 502 and executable on the processor 501. When the program is executed by the processor 501, the various processes of the above-mentioned report generation method embodiment based on the thinking chain and the large model are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0170] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0171] An embodiment of the present application also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the above-mentioned report generation method embodiment based on the thinking chain and the large model are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0172] The processor is the processor in the electronic device in the above embodiment. The readable storage medium includes a computer-readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0173] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the above-mentioned report generation method based on thought chain and large model.

[0174] The processor is the processor in the electronic device in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0175] An embodiment of the present application further provides a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned report generation method embodiment based on thinking chains and large models, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0176] It should be understood that the chip mentioned in the embodiments of the present application can also be called a device-level chip, a device chip, a chip device, or an on-chip device chip, etc.

[0177] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, it should be noted that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0178] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, or network device, etc.) to execute the report generation method based on the thinking chain and large model of each embodiment of the present application.

[0179] In the description of this application, "first feature" and "second feature" may include one or more such features.

[0180] In the description of this application, “plurality” means two or more.

[0181] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

[0182] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "illustrative embodiments," "examples," "specific examples," or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0183] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and intent of the present application, and that the scope of the present application is defined by the claims and their equivalents.

Claims

1. A report generation method based on thought chain and large model, characterized in that: The method comprises: Obtain image information and text information; Inputting the image information and text information into a multimodal hybrid encoder-decoder model trained using a multi-task training method to obtain a target text report; The multimodal hybrid encoder-decoder model includes an encoding module, an alignment module, a thought chain reasoning module and a decoding module, and the encoding module includes a visual encoder and a text encoder; The encoding module is used to extract features from the image information and text information to obtain image features and text features; The alignment module is used to perform image-text alignment on the image features and text features to obtain an image-text alignment result; The thought chain reasoning module is used to reason about the image-text alignment result and generate a thought chain; The decoding module is used to decode the image-text alignment result and the thought chain to obtain a target text report.

2. The report generation method based on thought chain and large model according to claim 1 is characterized in that: The extracting features of the image information and text information to obtain image features and text features includes: After resizing and normalizing the image information, the image information is input into a visual encoder to obtain image features; After word segmentation and vectorization operations are performed on the text information, it is input into a text encoder to obtain text features.

3. The report generation method based on thought chain and large model according to claim 1 is characterized in that: The performing image-text alignment on the image features and the text features to obtain an image-text alignment result includes: Calculating the similarity between the image features and the text features based on the attention mechanism; The image features and text features are matched based on the similarity to obtain an image-text alignment result.

4. The report generation method based on thought chain and large model according to claim 1 is characterized in that: The reasoning on the image-text alignment result to generate a thought chain includes: Inputting the image-text alignment result into the thought chain reasoning module to obtain the association information between the image and the text; Based on the association information, a thought chain is generated.

5. The report generation method based on thought chain and large model according to claim 1 is characterized in that: The training process of the multimodal hybrid encoder-decoder model includes: Build a pre-defined multimodal hybrid encoder-decoder model; Constructing a data set based on multiple image data and text reports corresponding to the image data; Based on the data set, a preset multimodal hybrid encoder-decoder model is trained using a multi-task training method to obtain a trained multimodal hybrid encoder-decoder model; The multiple tasks include contrastive learning tasks, image-text matching tasks, and text generation tasks.

6. The report generation method based on thought chain and large model according to claim 5 is characterized in that: The training process of the text generation task includes: Input the dataset into a preset multimodal hybrid encoder-decoder model to obtain the probability distribution of the next word at each time step; Adjusting the probability distribution according to a smoothing coefficient and a vocabulary size, and calculating a cross entropy loss value at each time step based on the adjusted probability distribution and the actual distribution; Based on the cross entropy loss value at each time step, the language modeling loss is obtained; Based on the language modeling loss, the preset multimodal hybrid encoder-decoder model is trained.

7. The report generation method based on thought chain and large model according to claim 5 is characterized in that: The multimodal hybrid encoder-decoder model includes one of a BLIP model, a Q-Former-Transformer model and a VisionTransformer-Transformer model.

8. A report generation device based on thought chain and large model, implemented by the report generation method based on thought chain and large model according to any one of claims 1 to 7, characterized in that: The device comprises: An acquisition module, used to acquire image information and text information; a processing module, configured to input the image information and text information into a multimodal hybrid encoder-decoder model trained using a multi-task training method to obtain a target text report; The multimodal hybrid encoder-decoder model includes an encoding module, an alignment module, a thought chain reasoning module and a decoding module, and the encoding module includes a visual encoder and a text encoder; The encoding module is used to extract features from the image information and text information to obtain image features and text features; The alignment module is used to perform image-text alignment on the image features and text features to obtain an image-text alignment result; The thought chain reasoning module is used to reason about the image-text alignment result and generate a thought chain; The decoding module is used to decode the image-text alignment result and the thought chain to obtain a target text report.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the report generation method based on thought chain and large model as described in any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the report generation method based on thought chain and big model according to any one of claims 1 to 7 is implemented.

Images