Report generation method and device based on thought chain and large model

Abstract

The application discloses a report generation method based on a thinking chain and a large model, and belongs to the technical field of image processing. The method comprises the following steps: inputting image information and text information into a multi-modal hybrid encoder-decoder model trained by using a multi-task training method to obtain a target text report; the multi-modal hybrid encoder-decoder model comprises an encoding module, an alignment module, a thinking chain reasoning module and a decoding module; the encoding module is used for extracting features of the image information and the text information to obtain image features and text features; the alignment module is used for aligning the image features and the text features to obtain an image-text alignment result; the thinking chain reasoning module is used for reasoning the image-text alignment result to generate a thinking chain; and the decoding module is used for decoding the image-text alignment result and the thinking chain to obtain the target text report. The method improves the coherence and accuracy of text report generation.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and particularly relates to a report generation method and device based on a thought chain and a large model. BACKGROUND

[0002] With the rapid development of digital pathology technology, deep learning-based pathological image analysis has made significant progress in medical tasks such as disease screening, tumor grading, and prognosis evaluation, but the extraction of complex pathological features and the accurate expression of diagnostic logic are still technical bottlenecks that need to be broken through. In the field of disease diagnosis, tissue heterogeneity and cell morphology diversity can interfere with the capture of key pathological features by deep learning models, leading to a significant decrease in the diagnostic basis. In terms of report generation, the lack of semantic relevance of multi-dimensional information in pathological images, such as the spatial relationship between cell distribution patterns and tissue structure, can severely affect the logical rigor of the diagnostic conclusion. In the remote consultation scenario, the alignment deviation and semantic gap of cross-modal data can reduce the explainability of the visual diagnostic evidence, making it difficult for the generated report to support clinical decision-making. Therefore, improving the model's ability to deconstruct pathological features and the level of diagnostic logic is a core challenge to ensure the accuracy of intelligent diagnosis and the credibility of the report.

[0003] Existing pathological report generation methods mainly use a hybrid architecture of visual encoders and sequence decoders, which can achieve basic-level image-text conversion, but still have defects such as single feature representation and weak semantic association when faced with complex pathological features and multi-dimensional diagnostic logic. Traditional cross-modal alignment methods based on attention mechanisms are limited by shallow feature interaction and cannot establish a pathological-level semantic mapping between histological features and diagnostic terminology. In addition, existing generation models are easily disturbed by noise features due to the multi-scale feature dependence and logic chain derivation requirements in the diagnostic reasoning process, leading to the omission of key diagnostic elements or frequent cause-and-effect inversion phenomena. In recent years, pre-training models based on Transformers have improved the fluency of text generation, but their general-purpose architecture has not been specifically designed for feature decoupling and logic chain modeling unique to pathological diagnosis. Existing methods have problems such as insufficient feature extraction, poor logicality and coherence, and low accuracy when generating diagnostic reports. SUMMARY

[0004] The present application aims to at least solve one of the technical problems existing in the prior art. To this end, the present application proposes a report generation method and device based on a thought chain and a large model, which improves the coherence and accuracy of text report generation.

[0005] In a first aspect, the present application provides a report generation method based on a thought chain and a large model, the method comprising:

[0006] obtaining image information and text information;

[0007] The image and text information are input into a multimodal hybrid encoder-decoder model trained using a multi-task training method to obtain the target text report;

[0008] The multimodal hybrid encoder-decoder model includes an encoding module, an alignment module, a thought chain reasoning module, and a decoding module. The encoding module includes a visual encoder and a text encoder.

[0009] The encoding module is used to extract features from the image information and text information to obtain image features and text features;

[0010] The alignment module is used to perform image-text alignment on the image features and text features to obtain the image-text alignment result;

[0011] The thought chain reasoning module is used to reason about the image-text alignment results and generate a thought chain;

[0012] The decoding module is used to decode the image-text alignment result and the thought chain to obtain the target text report.

[0013] According to one embodiment of this application, the step of extracting features from the image information and text information to obtain image features and text features includes:

[0014] After the image information is resized and normalized, it is input into the visual encoder to obtain image features;

[0015] After the text information is segmented and vectorized, it is input into the text encoder to obtain text features.

[0016] According to one embodiment of this application, the step of performing image-text alignment on the image features and text features to obtain the image-text alignment result includes:

[0017] The similarity between the image features and text features is calculated based on an attention mechanism;

[0018] Based on the similarity, the image features and text features are matched to obtain the image-text alignment result.

[0019] According to one embodiment of this application, the step of reasoning about the image-text alignment result to generate a thought chain includes:

[0020] The image-text alignment result is input into the thought chain reasoning module to obtain the association information between the image and the text;

[0021] Based on the aforementioned related information, a thought chain is generated.

[0022] According to one embodiment of this application, the training process of the multimodal hybrid encoder-decoder model includes:

[0023] Construct a pre-defined multimodal hybrid encoder-decoder model;

[0024] A dataset is constructed based on multiple image datasets and corresponding text reports.

[0025] Based on the dataset, a multi-task training method is used to train the preset multimodal hybrid encoder-decoder model to obtain the trained multimodal hybrid encoder-decoder model.

[0026] The multi-tasks include a contrastive learning task, an image-text matching task, and a text generation task.

[0027] According to one embodiment of this application, the training process for the text generation task includes:

[0028] The dataset is input into a preset multimodal hybrid encoder-decoder model to obtain the probability distribution of the next word at each time step;

[0029] The probability distribution is adjusted according to the smoothing coefficient and vocabulary size, and the cross-entropy loss value at each time step is calculated based on the adjusted probability distribution and the actual distribution.

[0030] Based on the cross-entropy loss value at each time step, the language modeling loss is obtained;

[0031] The preset multimodal hybrid encoder-decoder model is trained based on the language modeling loss.

[0032] According to one embodiment of this application, the multimodal hybrid encoder-decoder model includes one of the BLIP model, the Q-Former-Transformer model, and the Vision Transformer-Transformer model.

[0033] Secondly, this application provides a report generation device based on thought chains and large models, the device comprising:

[0034] The acquisition module is used to acquire image and text information;

[0035] The processing module is used to input the image information and text information into a multimodal hybrid encoder-decoder model trained using a multi-task training method to obtain the target text report;

[0036] The multimodal hybrid encoder-decoder model includes an encoding module, an alignment module, a thought chain reasoning module, and a decoding module. The encoding module includes a visual encoder and a text encoder.

[0037] The encoding module is used to extract features from the image information and text information to obtain image features and text features;

[0038] The alignment module is used to perform image-text alignment on the image features and text features to obtain the image-text alignment result;

[0039] The thought chain reasoning module is used to reason about the image-text alignment results and generate a thought chain;

[0040] The decoding module is used to decode the image-text alignment result and the thought chain to obtain the target text report.

[0041] Thirdly, this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the report generation method based on thought chain and large model as described in the first aspect above.

[0042] Fourthly, this application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the report generation method based on thought chain and large model as described in the first aspect above.

[0043] Fifthly, this application provides a chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the report generation method based on thought chains and large models as described in the first aspect.

[0044] Sixthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the report generation method based on thought chain and large model as described in the first aspect above.

[0045] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application.

[0046] The report generation method based on thought chain and large model provided by this invention has the following advantages over the prior art:

[0047] (1) This invention acquires image and text information and inputs it into a trained multimodal hybrid encoder-decoder model. It combines the encoding module, alignment module, thought chain reasoning module and decoding module to generate target text reports, which effectively improves the accuracy of image-text information processing and reasoning. By performing feature extraction, image-text alignment and thought chain reasoning on image and text features, it can generate target text reports related to image and text information more accurately. By adding the thought chain reasoning module, the model can gradually generate a logical chain of "observation → analysis → diagnosis", which improves the model's understanding and reasoning ability of semantics and enhances the coherence, logic and accuracy of text report generation.

[0048] (2) This invention constructs a preset multimodal hybrid encoder-decoder model and builds a dataset based on multiple image data and corresponding text reports. It trains the model using a multi-task training method, which effectively improves the correlation and reasoning ability between images and text. Through multi-task training of comparative learning tasks, image-text matching tasks and text generation tasks, it helps to enhance the deep fusion between images and text, improve the accuracy of understanding image information and text information, and improve the logic and accuracy of target text report generation.

[0049] (3) This invention improves the training efficiency of text generation tasks by inputting the dataset into a preset multimodal hybrid encoder-decoder model and calculating the cross-entropy loss value at each time step according to the adjusted probability distribution. By comparing the adjusted probability distribution with the smoothing coefficient and vocabulary size, the model's language modeling ability when generating text is optimized, enabling it to better understand and process the correlation between images and text, and generate more accurate and fluent text reports. Attached Figure Description

[0050] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0051] Figure 1 This is one of the flowcharts illustrating the report generation method based on thought chain and large model provided in the embodiments of this application;

[0052] Figure 2 This is the second flowchart of the report generation method based on thought chain and large model provided in the embodiments of this application;

[0053] Figure 3 This is a schematic diagram of the feature extraction process provided in an embodiment of this application;

[0054] Figure 4 This is a schematic diagram of the structure of the report generation device based on thought chain and large model provided in the embodiments of this application;

[0055] Figure 5 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0056] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0057] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0058] The following description, in conjunction with the accompanying drawings, details the report generation method, apparatus, electronic device, and readable storage medium based on thought chains and large models provided in this application, through specific embodiments and application scenarios.

[0059] Among them, the report generation method based on thought chain and large model can be applied to the terminal, specifically executed by the hardware or software in the terminal.

[0060] The terminal includes, but is not limited to, portable communication devices such as mobile phones or tablets with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads). It should also be understood that, in some embodiments, the terminal may not be a portable communication device, but rather a desktop computer with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads).

[0061] The following embodiments describe a terminal including a display and a touch-sensitive surface. However, it should be understood that the terminal may include one or more other physical user interface devices such as a physical keyboard, mouse, and joystick.

[0062] The report generation method based on thought chain and large model provided in this application embodiment can be executed by an electronic device or a functional module or functional entity in an electronic device that can implement the report generation method based on thought chain and large model. The electronic devices mentioned in this application embodiment include, but are not limited to, mobile phones, tablets, computers, cameras and wearable devices. The report generation method based on thought chain and large model provided in this application embodiment will be described below using an electronic device as the execution subject.

[0063] With the continuous development of medical imaging technology, digital pathology has gradually become an important component of modern medical diagnosis. Pathological images acquired through devices such as high-resolution scanners provide doctors with unprecedented detailed information to support disease diagnosis and treatment planning. However, how to efficiently and accurately extract useful information from these large volumes of digital pathological images and transform it into structured and easily understandable text descriptions has become one of the key challenges in current research and technological development.

[0064] Digital pathology involves recording, managing, analyzing, and sharing pathological information using digital means. It has not only greatly improved diagnostic efficiency but also promoted the development of telemedicine. However, traditional pathological image analysis methods, which rely on manual interpretation, suffer from problems such as high subjectivity and time-consuming processes. Therefore, developing technologies capable of automatically analyzing pathological images and generating reports has significant practical implications.

[0065] In the medical field, particularly pathology, image-text alignment refers to the process of converting visual information (such as cell morphology and tissue structure on pathological slides) into precise textual descriptions. This step is crucial for ensuring the accuracy of medical records. Traditional rule-based methods struggle to handle complex image content, while the emerging deep learning technology has demonstrated enormous potential.

[0066] While existing AI technologies have improved the efficiency and accuracy of pathological image analysis to some extent, they still face many challenges in handling complex and variable pathological conditions. Traditional pathological image analysis mainly relies on manual evaluation by experienced pathologists. In recent years, with the development of deep learning technology, AI-based automatic pathology report generation methods have gradually become a research hotspot. However, existing methods have the following problems:

[0067] 1. Insufficient feature extraction: Existing methods typically use pre-trained convolutional neural networks to extract features from pathological images, which makes it difficult to capture key information related to pathological diagnosis in the images.

[0068] 2. Poor logic and coherence: Pathology report texts generated by existing methods often lack logic and coherence, making it difficult to accurately describe the lesion features in pathological images.

[0069] Figure 1 This is one of the flowcharts illustrating the report generation method based on thought chain and large model provided in the embodiments of this application, such as... Figure 1 As shown, the report generation method based on thought chain and large model includes steps 110 and 120.

[0070] Step 110: Obtain image and text information;

[0071] Step 120: Input the image information and text information into the multimodal hybrid encoder-decoder model trained using a multi-task training method to obtain the target text report;

[0072] The multimodal hybrid encoder-decoder model includes an encoding module, an alignment module, a thought chain reasoning module, and a decoding module. The encoding module includes a visual encoder and a text encoder.

[0073] The encoding module is used to extract features from the image information and text information to obtain image features and text features;

[0074] The alignment module is used to perform image-text alignment on the image features and text features to obtain the image-text alignment result;

[0075] The thought chain reasoning module is used to reason about the image-text alignment results and generate a thought chain;

[0076] The decoding module is used to decode the image-text alignment result and the thought chain to obtain the target text report.

[0077] Figure 2 This is the second flowchart illustrating the report generation method based on thought chain and large model provided in the embodiments of this application, as shown below. Figure 2 As shown, image and text information are first acquired, and then input into a multimodal hybrid encoder-decoder model trained using a multi-task training method. The target text report is obtained through the following process:

[0078] (1) Preprocess and extract features from image and text information to obtain image features and text features;

[0079] (2) Image-text alignment;

[0080] Image features and text features are aligned so that they are in the same semantic space.

[0081] (3) Generation of thought chains;

[0082] Furthermore, based on the image-text alignment results, a logically coherent thought chain is generated through the thought chain reasoning module, which is tailored to the image analysis process.

[0083] (4) Report generation

[0084] Finally, the image-text alignment results and thought chain are decoded by the decoding module to obtain the target text report. For example, the target text report is a pathology report, which not only contains a detailed description of the lesion and the diagnosis conclusion, but also provides suggestions for further examination or treatment.

[0085] The image and text information generation method based on a multimodal hybrid encoder-decoder model provided in this application acquires image and text information and inputs it into a trained multimodal hybrid encoder-decoder model. The model combines an encoding module, an alignment module, a thought chain reasoning module, and a decoding module to generate a target text report, effectively improving the accuracy of image-text information processing and reasoning. By performing feature extraction, image-text alignment, and thought chain reasoning on image and text features, it can more accurately generate target text reports related to image and text information. By adding a thought chain reasoning module, the model gradually generates a logical chain of "observation → analysis → diagnosis," enhancing the model's semantic understanding and reasoning capabilities, and improving the coherence, logic, and accuracy of the generated text report.

[0086] In some embodiments, the step of extracting features from the image information and text information to obtain image features and text features includes:

[0087] After the image information is resized and normalized, it is input into the visual encoder to obtain image features;

[0088] After the text information is segmented and vectorized, it is input into the text encoder to obtain text features.

[0089] Figure 3 This is a schematic diagram of the feature extraction process provided in the embodiments of this application, such as... Figure 3 As shown, input image information and text information are processed separately. Image preprocessing includes adjusting image size and resolution, normalizing image pixel values, removing noise, and enhancing image contrast or brightness. The preprocessed image is then input into a visual encoder to obtain image features.

[0090] Text preprocessing includes word segmentation (segmenting text into words or phrases), stop word removal (removing common but meaningless words), stemming or lemmatization (converting words into their basic forms), and vectorization (converting text into numerical vectors, such as using word embeddings). The preprocessed text is then input into a text encoder to obtain text features.

[0091] In this embodiment, image information is resized and normalized before being input into a visual encoder for feature extraction. Text information is segmented and vectorized before being input into a text encoder for feature extraction. This effectively improves the representation capabilities of image and text information, making the extraction of image and text features more accurate. This provides a more reliable foundation for subsequent image-text alignment and inference, and better supports the generation of target text reports.

[0092] In some embodiments, the step of performing image-text alignment on the image features and text features to obtain the image-text alignment result includes:

[0093] The similarity between the image features and text features is calculated based on an attention mechanism;

[0094] Based on the similarity, the image features and text features are matched to obtain the image-text alignment result.

[0095] It is easy to understand that the image-text alignment result can be obtained through an attention mechanism. First, the similarity between image features and text features is calculated using the attention mechanism to establish the association between image features and text features, so that each description of text information can find the most relevant image information. Then, the image features and text features are aligned based on the similarity to obtain the image-text alignment result.

[0096] In this embodiment, the similarity between image features and text features is calculated based on an attention mechanism, and the image features and text features are matched based on this similarity to obtain image-text alignment results. This effectively improves the correlation between image and text information, enables more accurate alignment of semantic relationships between images and text, and improves the accuracy of multimodal information fusion.

[0097] In some embodiments, the reasoning process based on the image-text alignment result to generate a thought chain includes:

[0098] The image-text alignment result is input into the thought chain reasoning module to obtain the association information between the image and the text;

[0099] Based on the aforementioned related information, a thought chain is generated.

[0100] For example, the image-text alignment result is input into the thought chain reasoning module. The thought chain reasoning module begins to simulate the diagnostic thinking process of human experts, gradually identifying and describing the lesion area, analyzing its morphological characteristics, inferring the lesion type and severity, obtaining the association information between the image and the text, and generating a series of structured text descriptions based on the association information to obtain the thought chain.

[0101] In this embodiment, by inputting the image-text alignment result into the thought chain reasoning module, the association information between the image and the text is obtained, and a thought chain is generated based on the association information. This effectively improves the understanding and association reasoning ability between the image and the text, can more accurately capture the deep connection between the image and the text content, enhances the fusion effect of multimodal data, and improves the accuracy and quality of subsequent target text report generation.

[0102] In some embodiments, the training process of the multimodal hybrid encoder-decoder model includes:

[0103] Construct a pre-defined multimodal hybrid encoder-decoder model;

[0104] A dataset is constructed based on multiple image datasets and corresponding text reports.

[0105] Based on the dataset, a multi-task training method is used to train the preset multimodal hybrid encoder-decoder model to obtain the trained multimodal hybrid encoder-decoder model.

[0106] The multi-tasks include a contrastive learning task, an image-text matching task, and a text generation task.

[0107] For example, the dataset consists of pathological images and their diagnostic reports, including various types of pathological images (such as tissue slide images) and their corresponding diagnostic reports. It not only provides rich visual information but also includes detailed text descriptions, such as lesion features and diagnostic conclusions. In addition to basic image and text descriptions, the correspondence between different regions in the image and the corresponding text descriptions should also be collected or generated, which helps to improve the accuracy of the model in understanding image content.

[0108] During training, contrastive learning loss and cross-entropy loss can be used to optimize the model. Contrastive learning loss is used for image feature extraction and image-text alignment, enabling image and text features to be effectively aligned in a shared space, thus enhancing the model's ability to understand the complex relationships between images and text. Cross-entropy loss is used for thought chain generation and report generation, improving the quality of generated text by minimizing the difference between predicted and real text, making it as close as possible to the real pathology report.

[0109] Optionally, the AdamW optimizer can be used to reduce overfitting by using weight decay. As the training process progresses, gradually reducing the learning rate can accelerate convergence and reduce oscillations that occur in the later stages of training, thereby improving the overall performance of the model.

[0110] It is worth noting that the multimodal hybrid encoder-decoder model is trained using a multi-task training method. The multi-task includes contrastive learning tasks, image-text matching tasks, and text generation tasks. In addition, pathology-specific tasks can be added, such as identifying specific types of cell abnormalities and assessing tumor grades, so that the model can better understand and generate reports for specific pathological conditions.

[0111] In some embodiments, the contrastive learning task is matching medical images with case text. Through the contrastive learning task, the model can identify which images and texts match each other. The loss function used in the contrastive learning task is image-text contrast loss. The training process includes the following steps:

[0112] (1) Use an encoder to extract feature representations of all images and texts in the batch respectively;

[0113] (2) Update the weights of the momentum encoder according to the current encoder state;

[0114] (3) Use momentum encoders to process the same batch of data to generate more stable feature representations.

[0115] Suppose we have a batch of N image-text pairs. For each image i and its corresponding text t, we can obtain their feature representation v. i and u t A momentum encoder is used to generate a smoother version of these features, denoted as... and

[0116] (4) Based on the similarity formula, calculate the similarity score between positive and negative sample pairs.

[0117] Positive sample pairs: For each pair of matching images and text, a similarity score is calculated, usually through a dot product, i.e. and

[0118] Negative sample pairs: For non-matching image-text pairs, similarity scores are also calculated to form a similarity matrix.

[0119] (5) Loss calculation and optimization: The loss is calculated based on the image-text contrast loss, and the model parameters are adjusted through the backpropagation algorithm to minimize the loss.

[0120] For image i and its positive sample text i, the formula for calculating the image-text contrast loss is as follows:

[0121]

[0122] Where τ is the temperature parameter, L i Image-text contrast loss.

[0123] In some embodiments, the training process for the image-text matching task is as follows:

[0124] (1) Feature extraction: The image encoder and text encoder are used to extract feature representations of the image and text respectively;

[0125] (2) Feature fusion and score calculation: The extracted features are fused and the score of each image-text pair is calculated through a linear layer;

[0126] First, feature representations of the image and text are obtained from the image encoder and text encoder, and then fused using methods such as dot product and concatenation to form a joint representation. A linear layer is then used to map the joint representation to a scalar value, representing the probability that the image-text pair belongs to a positive sample. Let s be the probability that the image-text pair belongs to a positive sample. i,t Let s be the score after pairing image i and text t. i,t =W T [v i ;u t ]+b

[0127] Where W and b are the weight matrix and bias term of the linear layer, respectively, [v i ;u t ] represents image feature v i and text features u t A combination of some form (e.g., direct splicing).

[0128] (3) Loss calculation: Calculate the image-text matching loss for each sample based on the actual matching situation (label y) and the score predicted by the model;

[0129] For each image-text pair, if they match, the label y = 1; otherwise, y = 0. Therefore, the formula for calculating the image-text matching loss is as follows:

[0130] L ITM =-ylog(σ(s) i,t ))-(1-y)log(1-σ(s i,t ))

[0131] Where σ represents the sigmoid function, L ITM This represents the loss for image-text matching.

[0132] (4) Hard negative sample selection: Select the most difficult negative sample to distinguish within a batch based on the current prediction results of the model.

[0133] To enhance the model's learning performance, the negative samples that the model currently finds most difficult to distinguish (i.e., samples that the model scores highly but are actually mismatched) are selected to force the model to learn more complex patterns. Finally, the model parameters are updated based on the sum of the losses of all samples (including hard negative samples).

[0134] In this embodiment, a preset multimodal hybrid encoder-decoder model is constructed, and a dataset is built based on multiple image data and corresponding text reports. The model is trained using a multi-task training method, which effectively improves the correlation and reasoning ability between images and text. Through multi-task training of comparative learning tasks, image-text matching tasks, and text generation tasks, it helps to enhance the deep fusion between images and text, improve the accuracy of understanding image and text information, and enhance the logic and accuracy of target text report generation.

[0135] In some embodiments, the training process for the text generation task includes:

[0136] The dataset is input into a preset multimodal hybrid encoder-decoder model to obtain the probability distribution of the next word at each time step;

[0137] The probability distribution is adjusted according to the smoothing coefficient and vocabulary size, and the cross-entropy loss value at each time step is calculated based on the adjusted probability distribution and the actual distribution.

[0138] Based on the cross-entropy loss value at each time step, the language modeling loss is obtained;

[0139] The preset multimodal hybrid encoder-decoder model is trained based on the language modeling loss.

[0140] It's easy to understand that the training process for a text generation task includes the following steps:

[0141] (1) Input the text sequence in the dataset into the preset multimodal hybrid encoder-decoder model, obtain the probability distribution of the next predicted word of the model at each time step t, and calculate the original cross-entropy loss for each time step based on the probability distribution and the next true target word at each time step t. For example, for a given text sequence {w1, w2, ..., w...} T}, where T is the sequence length, predict the next word w at each position t. t+1 The probability distribution P(w) t+1 |w1,w2,...,w tThe formula for calculating the original cross-entropy loss at each time step is shown below:

[0142]

[0143] in, For the next real target word, L CE This represents the original cross-entropy loss.

[0144] (2) Based on the preset smoothing coefficient and vocabulary size, the probability estimates of the target word and the predicted word are adjusted. The probability of the target word is no longer an absolute 1, but is adjusted to... Where ε is the label smoothing coefficient (e.g., 0.1), V is the vocabulary size, and the probability of the predicted word is adjusted from 0 to...

[0145] (3) Recalculate the loss value at each time step using the adjusted probability to obtain the cross-entropy loss, as shown in the following formula:

[0146]

[0147] Among them, L LS This represents the cross-entropy loss.

[0148] (4) Based on the cross-entropy loss value at each time step, the losses of all time steps are summed and averaged to obtain the language modeling loss. The calculation formula is shown below:

[0149]

[0150] Where L represents the language modeling loss.

[0151] In this embodiment, by inputting the dataset into a preset multimodal hybrid encoder-decoder model, the cross-entropy loss value at each time step is calculated based on the adjusted probability distribution, which effectively improves the training efficiency of the text generation task. By comparing the probability distribution adjusted by combining the smoothing coefficient and vocabulary size with the actual distribution, the model's language modeling ability when generating text is optimized, enabling it to better understand and process the correlation between images and text, and generate more accurate and fluent text reports.

[0152] In some embodiments, the multimodal hybrid encoder-decoder model includes one of the BLIP model, the Q-Former-Transformer model, and the Vision Transformer-Transformer model.

[0153] Optionally, the multimodal hybrid encoder-decoder model can be a BLIP (Bootstrapping Language-Image Pretraining) model. The BLIP model can perform data augmentation through captioning and filtering techniques, using a caption generator to create synthetic captions (i.e., preliminary descriptions) for the image, and then using filters to remove inaccurate or noisy descriptions, which helps to improve the quality of the final report.

[0154] Optionally, the multimodal hybrid encoder-decoder model can be a Q-Former-Transformer model, where Q-Former is the encoder and Transformer is the decoder.

[0155] Optionally, the multimodal hybrid encoder-decoder model can be a Vision Transformer-Transformer model, where the Vision Transformer is the encoder and the Transformer is the decoder.

[0156] It should be noted that different models can be selected as multimodal hybrid encoder-decoder models according to the actual application scenario, and this application embodiment does not impose any restrictions.

[0157] In this embodiment, by employing one of the BLIP model, Q-Former-Transformer model, or VisionTransformer-Transformer model, the most suitable model architecture can be flexibly selected according to the actual application scenario, which improves the flexibility and accuracy of text generation tasks and enhances the adaptability of the model in different application scenarios.

[0158] The report generation method based on thought chains and large models provided in this application can be executed by a report generation device based on thought chains and large models. This application uses the example of a report generation device based on thought chains and large models executing the report generation method to illustrate the report generation device based on thought chains and large models provided in this application.

[0159] This application also provides a report generation device based on thought chains and large models, such as Figure 4 As shown, the report generation device based on thought chain and large model includes: acquisition module 410 and processing module 420.

[0160] The acquisition module 410 is used to acquire image information and text information;

[0161] Processing module 420 is used to input the image information and text information into a multimodal hybrid encoder-decoder model trained using a multi-task training method to obtain a target text report;

[0162] The multimodal hybrid encoder-decoder model includes an encoding module, an alignment module, a thought chain reasoning module, and a decoding module. The encoding module includes a visual encoder and a text encoder.

[0163] The encoding module is used to extract features from the image information and text information to obtain image features and text features;

[0164] The alignment module is used to perform image-text alignment on the image features and text features to obtain the image-text alignment result;

[0165] The thought chain reasoning module is used to reason about the image-text alignment results and generate a thought chain;

[0166] The decoding module is used to decode the image-text alignment result and the thought chain to obtain the target text report.

[0167] According to the report generation method based on thought chain and large model provided in the embodiments of this application, image information and text information are acquired and input into a trained multimodal hybrid encoder-decoder model. The target text report is generated by combining the encoding module, alignment module, thought chain reasoning module and decoding module. This effectively improves the accuracy of image-text information processing and reasoning. By performing feature extraction, image-text alignment and thought chain reasoning on image features and text features, the target text report related to image and text information can be generated more accurately. By adding the thought chain reasoning module, the model can gradually generate a logical chain of "observation → analysis → diagnosis", which improves the model's understanding and reasoning ability of semantics and enhances the coherence, logic and accuracy of text report generation.

[0168] The report generation device based on thought chain and large model provided in this application embodiment can achieve... Figures 1 to 3 The various processes implemented in the report generation method based on thought chain and large model will not be described in detail here to avoid repetition.

[0169] In some embodiments, such as Figure 5 As shown, this application embodiment also provides an electronic device 500, including a processor 501, a memory 502, and a computer program stored in the memory 502 and executable on the processor 501. When the program is executed by the processor 501, it implements the various processes of the above-described report generation method embodiment based on thought chain and large model, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0170] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0171] This application also provides a non-transitory computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the above-described report generation method embodiments based on thought chains and large models, and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0172] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0173] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described report generation method based on thought chains and large models.

[0174] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0175] This application also provides a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled. The processor is used to run programs or instructions to implement the various processes of the above-described report generation method embodiments based on thought chains and large models, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0176] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a device-level chip, device chip, chip device, or on-chip device chip, etc.

[0177] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0178] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the report generation method based on thought chain and large model of the various embodiments of this application.

[0179] In the description of this application, "first feature" and "second feature" may include one or more of the features.

[0180] In the description of this application, "multiple" means two or more.

[0181] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

[0182] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0183] Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

Claims

1. A report generation method based on thought chains and large models, characterized in that, The method includes: Acquire image and text information; The image and text information are input into a multimodal hybrid encoder-decoder model trained using a multi-task training method to obtain the target text report; The multimodal hybrid encoder-decoder model includes an encoding module, an alignment module, a thought chain reasoning module, and a decoding module. The encoding module includes a visual encoder and a text encoder. The training process of the multimodal hybrid encoder-decoder model includes: constructing a preset multimodal hybrid encoder-decoder model; constructing a dataset based on multiple image data and corresponding text reports; and training the preset multimodal hybrid encoder-decoder model using a multi-task training method based on the dataset to obtain a trained multimodal hybrid encoder-decoder model. The multi-task includes a contrastive learning task, an image-text matching task, and a text generation task. The training process for the text generation task includes: inputting the dataset into a preset multimodal hybrid encoder-decoder model to obtain the probability distribution of the next word at each time step; adjusting the probability distribution according to the smoothing coefficient and vocabulary size; and calculating the cross-entropy loss value at each time step based on the adjusted probability distribution and the actual distribution; the cross-entropy loss is calculated as follows: ； in, For cross-entropy loss, - The formula for calculating the raw cross-entropy loss at each time step is: Given a text sequence, where T is the sequence length, For the next real target word, It is the label smoothing coefficient; based on the cross-entropy loss value at each time step, the language modeling loss is obtained; based on the language modeling loss, the preset multimodal hybrid encoder-decoder model is trained; It refers to the size of the vocabulary list; This represents the probability estimate corresponding to target words other than the actual target words; The loss function used in the contrastive learning task is the image-text contrastive loss. The training process of the contrastive learning task includes: extracting feature representations of images and text within a batch using an encoder; updating the weights of the momentum encoder according to the current encoder state; processing the same batch of data using the momentum encoder to generate smoothed version feature representations; calculating the similarity score between positive and negative sample pairs based on the similarity formula, where the similarity score of positive sample pairs is achieved through dot product and the similarity score of negative sample pairs is achieved through a similarity matrix; calculating the loss according to the image-text contrastive loss, and adjusting the model parameters through the backpropagation algorithm to minimize the loss; The loss function used in the image-text matching task is the image-text matching loss. The training process of the image-text matching task includes: extracting feature representations of images and text using an image encoder and a text encoder, respectively; fusing the extracted features and calculating the score of each image-text pair through a linear layer; and calculating the image-text matching loss for each sample based on the actual matching situation and the score predicted by the model. The calculation formula for the image-text matching loss is as follows: ； in, This represents the sigmoid function. For image-text matching loss, The score is the result of pairing image i and text t; hard negative samples are selected in a batch based on the current prediction results of the model, and the model parameters are updated based on the total loss, which is the sum of the losses of all samples, including the hard negative samples; This represents the label, i.e., the actual match for each image and text pair; The encoding module is used to extract features from the image information and text information to obtain image features and text features; The alignment module is used to perform image-text alignment on the image features and text features to obtain the image-text alignment result; The thought chain reasoning module is used to reason about the image-text alignment results and generate a thought chain; The decoding module is used to decode the image-text alignment result and the thought chain to obtain the target text report.

2. The report generation method based on thought chain and large model according to claim 1, characterized in that, The step of extracting features from the image information and text information to obtain image features and text features includes: After the image information is resized and normalized, it is input into the visual encoder to obtain image features; After the text information is segmented and vectorized, it is input into the text encoder to obtain text features.

3. The report generation method based on thought chain and large model according to claim 1, characterized in that, The step of performing image-text alignment on the image features and text features to obtain the image-text alignment result includes: The similarity between the image features and text features is calculated based on an attention mechanism; Based on the similarity, the image features and text features are matched to obtain the image-text alignment result.

4. The report generation method based on thought chain and large model according to claim 1, characterized in that, The reasoning process based on the image-text alignment result to generate a thought chain includes: The image-text alignment result is input into the thought chain reasoning module to obtain the association information between the image and the text; Based on the aforementioned related information, a thought chain is generated.

5. The report generation method based on thought chain and large model according to claim 1, characterized in that, The multimodal hybrid encoder-decoder model includes one of the BLIP model, Q-Former-Transformer model, and VisionTransformer-Transformer model.

6. A report generation device based on thought chain and large model, implemented using the report generation method based on thought chain and large model as described in any one of claims 1 to 5, characterized in that, The device includes: The acquisition module is used to acquire image and text information; The processing module is used to input the image information and text information into a multimodal hybrid encoder-decoder model trained using a multi-task training method to obtain the target text report; The multimodal hybrid encoder-decoder model includes an encoding module, an alignment module, a thought chain reasoning module, and a decoding module. The encoding module includes a visual encoder and a text encoder. The training process of the multimodal hybrid encoder-decoder model includes: constructing a preset multimodal hybrid encoder-decoder model; constructing a dataset based on multiple image data and corresponding text reports; and training the preset multimodal hybrid encoder-decoder model using a multi-task training method based on the dataset to obtain a trained multimodal hybrid encoder-decoder model. The multi-task includes a contrastive learning task, an image-text matching task, and a text generation task. The training process for the text generation task includes: inputting the dataset into a preset multimodal hybrid encoder-decoder model to obtain the probability distribution of the next word at each time step; adjusting the probability distribution according to the smoothing coefficient and vocabulary size; and calculating the cross-entropy loss value at each time step based on the adjusted probability distribution and the actual distribution; the cross-entropy loss is calculated as follows: ； in, For cross-entropy loss, - The formula for calculating the raw cross-entropy loss at each time step is: Given a text sequence, where T is the sequence length, For the next real target word, It is the label smoothing coefficient; based on the cross-entropy loss value at each time step, the language modeling loss is obtained; based on the language modeling loss, the preset multimodal hybrid encoder-decoder model is trained; It refers to the size of the vocabulary list; This represents the probability estimate corresponding to target words other than the actual target words; The loss function used in the contrastive learning task is the image-text contrastive loss. The training process of the contrastive learning task includes: extracting feature representations of images and text within a batch using an encoder; updating the weights of the momentum encoder according to the current encoder state; processing the same batch of data using the momentum encoder to generate smoothed version feature representations; calculating the similarity score between positive and negative sample pairs based on the similarity formula, where the similarity score of positive sample pairs is achieved through dot product and the similarity score of negative sample pairs is achieved through a similarity matrix; calculating the loss according to the image-text contrastive loss, and adjusting the model parameters through the backpropagation algorithm to minimize the loss; The loss function used in the image-text matching task is the image-text matching loss. The training process of the image-text matching task includes: extracting feature representations of images and text using an image encoder and a text encoder, respectively; fusing the extracted features and calculating the score of each image-text pair through a linear layer; and calculating the image-text matching loss for each sample based on the actual matching situation and the score predicted by the model. The calculation formula for the image-text matching loss is as follows: ； in, This represents the sigmoid function. For image-text matching loss, The score is the result of pairing image i and text t; hard negative samples are selected in a batch based on the current prediction results of the model, and the model parameters are updated based on the total loss, which is the sum of the losses of all samples, including the hard negative samples; This represents the label, i.e., the actual match for each image and text pair; The encoding module is used to extract features from the image information and text information to obtain image features and text features; The alignment module is used to perform image-text alignment on the image features and text features to obtain the image-text alignment result; The thought chain reasoning module is used to reason about the image-text alignment results and generate a thought chain; The decoding module is used to decode the image-text alignment result and the thought chain to obtain the target text report.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the report generation method based on thought chain and large model as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the report generation method based on thought chain and large model as described in any one of claims 1 to 5.

Images