End-to-end image file processing method, model building method and apparatus

Abstract

This disclosure relates to an end-to-end image file processing method, model construction method, and apparatus, comprising: acquiring an image file to be processed; inputting the image file into a pre-constructed image report generation model for image semantic encoding processing, generating a context vector based on a spatiotemporal attention mechanism, and performing text sequence prediction and decoding processing, outputting an image report text; wherein, generating the context vector based on the spatiotemporal attention mechanism includes: based on the results of image semantic encoding processing and text sequence prediction and decoding processing, dynamically mapping and learning the correspondence between the importance of different image regions and the text prediction at each time step based on an attention mechanism, obtaining a context vector for the global image at each time step; the context vector is obtained by weighting the results of image semantic encoding processing with attention weights. The output report has high accuracy, high processing efficiency, and good model generalization performance.

Description

Technical Field

[0001] This disclosure relates to the fields of image processing and artificial intelligence technology, and in particular to an end-to-end image file processing method, model building method and apparatus. Background Technology

[0002] With the advancement of imaging technology, a large number of image files exist in various fields, such as image files obtained from image or video capture, image files obtained from remote sensing mapping of geographic information, image files obtained from the detection of tissues or microstructures of materials or functional devices, and various image files obtained from medical examinations (such as X-ray images, computed tomography (CT) images, magnetic resonance imaging (MRI) images, etc.). These image files are usually analyzed and interpreted manually to obtain image reports that reflect the main content expressed in the images.

[0003] In realizing the concept disclosed herein, the inventors discovered at least the following technical problems in the related technologies: Manual processing of large numbers of image files is inefficient; manual processing introduces human cognitive biases and differences in expression, resulting in poor standardization of the final image reports, which may even contain errors; some related technologies use a combination of visual models and artificial intelligence agents for information extraction and report generation, but the generalization performance of these AI models needs improvement. Furthermore, in processing scenarios involving complex graphics or requiring specialized identification, reports may sometimes deviate from professional knowledge or even contain errors, thus the processing accuracy also needs improvement. Summary of the Invention

[0004] To solve the above-mentioned technical problems, or at least partially solve them, embodiments of this disclosure provide an end-to-end image file processing method, model building method, and apparatus.

[0005] In a first aspect, embodiments of this disclosure provide an end-to-end image file processing method. The method includes: acquiring an image file to be processed; inputting the image file into a pre-constructed image report generation model for image semantic encoding processing, generating a context vector based on a spatiotemporal attention mechanism, and performing text sequence prediction and decoding processing, outputting an image report text; wherein, generating the context vector based on the spatiotemporal attention mechanism includes: based on the results of image semantic encoding processing and text sequence prediction and decoding processing, dynamically mapping and learning the correspondence between the importance of different image regions and the text prediction at each time step, based on an attention mechanism, to obtain a context vector for the global image at each time step; the context vector is obtained by weighting the results of image semantic encoding processing with attention weights.

[0006] In some embodiments, the image report generation model includes: an image semantic encoding model, an attention module, a word embedding module, and a text sequence prediction and decoding model. The image semantic encoding model performs image semantic encoding processing on the image file to obtain image semantic feature embedding vectors corresponding to each image region. The attention module performs the step of generating context vectors based on a spatiotemporal attention mechanism; wherein, during the dynamic mapping learning process for the context vector at each time step, the result of the image semantic encoding processing is: image semantic feature embedding vectors corresponding to each image region; the result of the text sequence prediction and decoding processing is: the hidden state output at the previous time step. The word embedding module maps the text predicted at the previous time step to word vectors. The text sequence prediction and decoding model predicts the output text at the current time step based on the word vectors, the context vectors, and the hidden state output at the previous time step, and performs multiple time steps to obtain an output text sequence; wherein, the image report text is obtained based on the output text sequence.

[0007] In some embodiments, based on an attention mechanism, dynamic mapping learning is performed to learn the correspondence between the importance of different image regions and the text prediction at each time step, resulting in a context vector for each time step. This includes: for the current time step, based on the attention mechanism, performing importance analysis on different image regions according to the image semantic feature embedding vectors corresponding to each image region and the hidden state output from the previous time step, to obtain the attention score for each image region at the current time step; normalizing the attention weights for each image region at the current time step to obtain the attention weights for each image region at the current time step; and weighting and summing the image semantic feature embedding vectors corresponding to each image region and the corresponding attention weights to obtain the context vector for the global image at the current time step.

[0008] In some embodiments, the aforementioned image file includes a medical image file; the aforementioned image report generation model further includes a verification module. The verification module is used to match the lesion location and lesion type in the image described in the aforementioned output text sequence against a manually annotated clinical case file library, perform medical consistency verification between the aforementioned output text sequence and the matched target clinical case file, and output the aforementioned image report text if the verification passes.

[0009] In some embodiments, the image report generation model further includes an image region labeling module. This image region labeling module is used to associate corresponding image regions in the image file based on key information in the image report text and to perform visual labeling on the corresponding image regions. Associating corresponding image regions in the image file based on key information in the image report text includes: associating the image region corresponding to the maximum attention score from the attention module when generating the context vector at each time step with the image region corresponding to the maximum attention score.

[0010] In some embodiments, the network structure of the above-described image semantic coding model includes: a Region Convolutional Neural Network (R-CNN) with a deep convolutional neural network (VGG16) as its backbone. The network structure of the above-described text sequence prediction and decoding model includes: a Long Short-Term Memory (LSTM) network.

[0011] In some embodiments, the above-described image report generation model is constructed in the following manner:

[0012] Obtain multiple training image files and multiple corresponding training image reports from the training database;

[0013] The text sequences corresponding to each training image report are length aligned, and the length-aligned text sequences are then segmented, deduplicated, and mapped using word encoding to generate training word vector sequences corresponding to the text sequences of each training image report. The word vectors obtained through word encoding mapping have the same encoding dimension as those encoded for the aforementioned training image files.

[0014] The above-mentioned training image report is input into the target neural network model to be trained, and the output image report prediction text sequence is generated. The above-mentioned training word vector sequence is used as training labels to train the above-mentioned target neural network model. Based on the trained target neural network model, the image report generation model is constructed. The above-mentioned target neural network model has the same structure as the above-mentioned image report generation model but different parameters.

[0015] In some embodiments, the training image files in the training database include at least some of the following types: video image files, picture image files, remote sensing image files, material testing image files, and medical image files. When the training image files include medical image files, these medical image files cover a variety of disease types and multiple image modalities.

[0016] In some embodiments, before performing length alignment on the text sequences, the method further includes: performing expression standardization on the text sequences corresponding to each training image report, wherein the expression standardization includes at least one of the following operations: unifying terminology style, standardizing syntactic structure, and removing redundant modifiers. The target neural network model is trained using an adaptive gradient descent algorithm; the target neural network model is considered to be trained successfully when the loss function is less than a set threshold or the number of training iterations exceeds a set value; the loss function represents the result of accumulating the cross-entropy loss between the predicted text sequence of the image report and the training labels at each time step over time steps.

[0017] Secondly, embodiments of this disclosure provide a method for constructing an image report generation model. The method includes: acquiring multiple training image files and multiple corresponding training image reports from a training database; performing length alignment processing on the text sequences corresponding to each training image report, and performing word segmentation, lexical deduplication, and lexical encoding mapping processing on the length-aligned text sequences to generate training word vector sequences corresponding to the text sequences of each training image report; wherein the word vectors obtained through lexical encoding mapping have the same encoding dimension as the encoding of the aforementioned training image files; inputting the aforementioned training image reports into a target neural network model to be trained for image semantic encoding processing, generating context vectors based on a spatiotemporal attention mechanism, and performing text sequence prediction decoding processing to output a predicted text sequence of the image report; using the aforementioned training word vector sequences as training labels to train the aforementioned target neural network model; and constructing an image report generation model based on the trained target neural network model. The generation of context vectors based on the spatiotemporal attention mechanism includes: based on the results of image semantic encoding and text sequence prediction decoding, dynamically mapping and learning the correspondence between the importance of different image regions and the text prediction at each time step, based on the attention mechanism, to obtain the context vector for the global image at each time step; the above context vector is obtained by weighting the results of image semantic encoding with attention weights.

[0018] Thirdly, embodiments of this disclosure provide an end-to-end image file processing apparatus. The apparatus includes a file acquisition module and an automatic image report generation module. The file acquisition module is used to acquire an image file to be processed. The automatic image report generation module is used to input the image file into a pre-built image report generation model for image semantic encoding processing, generating a context vector based on a spatiotemporal attention mechanism, and performing text sequence prediction and decoding processing, outputting an image report text. Specifically, generating the context vector based on the spatiotemporal attention mechanism includes: based on the results of image semantic encoding processing and text sequence prediction and decoding processing, dynamically mapping and learning the correspondence between the importance of different image regions and the text prediction at each time step, based on an attention mechanism, to obtain a context vector for the global image at each time step; the context vector is obtained by weighting the results of image semantic encoding processing with attention weights.

[0019] Fourthly, embodiments of this disclosure provide an apparatus for constructing an image report generation model. The apparatus includes: a training data acquisition module, a label processing module, and a model construction module. The training data acquisition module is used to acquire multiple training image files and multiple corresponding training image reports from a training database. The label processing module is used to perform length alignment processing on the text sequences corresponding to each training image report, and to perform word segmentation, lexical deduplication, and lexical encoding mapping processing on the length-aligned text sequences to generate training word vector sequences corresponding to the text sequences of each training image report; wherein the word vectors obtained through lexical encoding mapping have the same encoding dimension as those encoded for the training image files. The model construction module is used to input the training image reports into a target neural network model to be trained for image semantic encoding processing, generation of context vectors based on a spatiotemporal attention mechanism, and text sequence prediction decoding processing, outputting a predicted text sequence for the image report. The training word vector sequences are used as training labels to train the target neural network model, and an image report generation model is constructed based on the trained target neural network model. The generation of context vectors based on the spatiotemporal attention mechanism includes: based on the results of image semantic encoding and text sequence prediction decoding, dynamically mapping and learning the correspondence between the importance of different image regions and the text prediction at each time step, based on the attention mechanism, to obtain the context vector for the global image at each time step; the above context vector is obtained by weighting the results of image semantic encoding with attention weights.

[0020] Fifthly, embodiments of this disclosure provide an electronic device. The electronic device includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, communication interface, and memory communicate with each other via the communication bus; the memory stores computer programs; and the processor, when executing the program stored in the memory, implements the end-to-end image file processing method or the image report generation model construction method described above.

[0021] Sixthly, embodiments of this disclosure provide a computer-readable storage medium. The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the end-to-end image file processing method or the image report generation model construction method as described above.

[0022] The technical solutions provided in the embodiments of this disclosure have at least some or all of the following advantages:

[0023] By constructing an image report generation model, this model can perform image semantic encoding processing, generate context vectors based on a spatiotemporal attention mechanism, and perform text sequence prediction and decoding processing on the input image file, outputting image report text. In the process of generating context vectors based on the spatiotemporal attention mechanism, the model dynamically maps and learns the correspondence between the importance of different image regions and the text prediction at each time step based on the results of image semantic encoding and text sequence prediction and decoding, obtaining the context vector for the global image at each time step. This context vector is obtained by weighting the results of image semantic encoding with attention weights. Therefore, in the word-by-word prediction process, each time step... The text predicted in each step takes into account the relative importance of each region in the global image. This dynamic mapping learning process combines the decoding results of the previous time step with the results of image semantic encoding processing for each region of the image. It integrates differentiated attention in the temporal dimension and global attention weights in the spatial dimension, demonstrating the effect of spatiotemporal cross-learning and precise correspondence with image regions. This not only improves the generalization performance of the image report generation model for various types of image files, but also outputs more accurate image report text for image files in relatively specialized fields (such as remote sensing image files, material testing image files, medical image files, etc.). Moreover, the end-to-end processing time is very short, improving the processing efficiency for a large number of image files. Attached Figure Description

[0024] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0025] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0026] Figure 1 A flowchart illustrating an end-to-end image file processing method according to an embodiment of the present disclosure is shown schematically.

[0027] Figure 2 A schematic diagram illustrating the structure and processing of an image report generation model according to an embodiment of the present disclosure is shown.

[0028] Figure 3 The diagram illustrates a simplified process for processing medical image files using the end-to-end image file processing method provided in this disclosure.

[0029] Figure 4 The diagram illustrates the structure and processing of a text sequence prediction decoding model according to an embodiment of the present disclosure.

[0030] Figure 5 A flowchart illustrating a method for constructing an image report generation model according to an embodiment of the present disclosure is shown schematically.

[0031] Figure 6 A schematic block diagram of an electronic device provided in an embodiment of the present disclosure is shown. Detailed Implementation

[0032] During the research and development process, it was found that manual processing of large numbers of image files is inefficient. Furthermore, manual processing introduces human cognitive biases and differences in expression, leading to poor standardization and potential errors in the final image reports. Some related technologies combine visual models with artificial intelligence agents for information extraction and report generation; however, the generalization performance of these AI models needs improvement. Moreover, in processing scenarios involving complex graphics or requiring specialized identification, reports sometimes deviate from professional knowledge or even contain errors, thus the accuracy of the processing also needs to be improved.

[0033] For example, when processing one or more types of image files, such as video files, image files, remote sensing files, material testing files, and medical files, the process largely relies on the experience of professionals. However, individual differences in cognition and expression also exist, and each image file needs to be identified manually, resulting in relatively low processing efficiency.

[0034] In view of this, embodiments of this disclosure provide an end-to-end image file processing method, model building method, and apparatus. The end-to-end image file processing method includes: acquiring an image file to be processed; inputting the image file into a pre-built image report generation model for image semantic encoding processing, generating a context vector based on a spatiotemporal attention mechanism, and performing text sequence prediction and decoding processing, outputting an image report text; wherein, generating the context vector based on the spatiotemporal attention mechanism includes: based on the results of image semantic encoding processing and text sequence prediction and decoding processing, dynamically mapping and learning the correspondence between the importance of different image regions and the text prediction at each time step, based on an attention mechanism, to obtain a context vector for the global image at each time step; the context vector is obtained by weighting the results of image semantic encoding processing with attention weights.

[0035] In the process of generating context vectors based on the spatiotemporal attention mechanism, the model dynamically maps and learns the correspondence between the importance of different image regions and the text prediction at each time step, based on the results of image semantic encoding and text sequence prediction decoding. This results in a context vector for the global image at each time step. This context vector is obtained by weighting the results of image semantic encoding with attention weights. Therefore, in the word-by-word prediction process, the text predicted at each time step takes into account the relative importance of each region in the global image. This dynamic mapping learning process can combine the decoding results of the previous time step with the results of image semantic encoding for each region of the image, and integrate differentiated attention in the temporal dimension and global attention weights in the spatial dimension. This reflects the effect of spatiotemporal cross-learning and precise correspondence with image regions. It can not only improve the generalization performance of the image report generation model for various image files, but also output more accurate image report text for image files in relatively professional fields (such as remote sensing image files, material testing image files, medical image files, etc.). Moreover, the end-to-end processing time is very short, improving the processing efficiency for a large number of image files.

[0036] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0037] A first exemplary embodiment of this disclosure provides an end-to-end image file processing method. This method can be executed by an electronic device with computing capabilities.

[0038] Figure 1 A flowchart illustrating an end-to-end image file processing method according to an embodiment of the present disclosure is shown schematically.

[0039] Reference Figure 1 As shown, the end-to-end image file processing method provided in this embodiment includes the following steps: S110 and S120.

[0040] In step S110, the image file to be processed is obtained.

[0041] In step S120, the above image file is input into a pre-built image report generation model for image semantic encoding, generation of context vectors based on spatiotemporal attention mechanism, and text sequence prediction decoding, and the output is the image report text.

[0042] The generation of context vectors based on the spatiotemporal attention mechanism includes: based on the results of image semantic encoding and text sequence prediction decoding, dynamically mapping and learning the correspondence between the importance of different image regions and the text prediction at each time step, based on the attention mechanism, to obtain the context vector for the global image at each time step; the context vector is obtained by weighting the results of image semantic encoding with attention weights.

[0043] Figure 2 A schematic diagram illustrating the structure and processing of an image report generation model according to an embodiment of the present disclosure is shown.

[0044] Reference Figure 2 As shown, the image report generation model 200 provided in this embodiment includes: an image semantic coding model 210, an attention module 220, a word embedding module 230, and a text sequence prediction decoding model 240.

[0045] The aforementioned image semantic coding model 210 is used to perform image semantic coding processing on the aforementioned image file to obtain the image semantic feature embedding vector corresponding to each image region.

[0046] In some embodiments, the network structure of the image semantic coding model 210 includes: a region convolutional neural network R-CNN with a deep convolutional neural network VGG16 as the backbone network structure.

[0047] By overlaying an R-CNN framework onto the VGG16 backbone network, a selective search approach is used to extract potential candidate regions from image files, and region features are extracted for each candidate box. All region features are then integrated into a two-dimensional feature tensor A = [a1, a2, ..., ...]. , ..., a n ], each of which ( The tensor A represents the high-dimensional representation of the i-th candidate image region in an image file (representing a 256-dimensional space). This high-dimensional representation is the image semantic feature embedding vector, where i is the region index, ranging from 1 to n, where n ≥ 2 and is a positive integer. This tensor A not only preserves the global structural information of the image but also refines the semantic description of each representative region in the image (e.g., for medical image files, it includes lesion regions; for material microstructure image files, it includes interface regions, micro-tissue regions, etc.), which forms the basis for the subsequent attention mechanism.

[0048] The image feature extraction module acts similarly to the perceptual layer in the entire system. By mapping low-level image pixels to high-level semantic embeddings, it bridges the gap between image information and linguistic expression, providing structurally complete and semantically clear visual input for the automated report generation process. Meanwhile, the combination of R-CNN and VGG16 fully integrates the advantages of localization sensitivity and feature stability, effectively supporting the subsequent attention module 220 and text sequence prediction and decoding model 240 in their dynamic attention to different spatial regions during sequence generation.

[0049] The attention module 220 is used to perform the above-mentioned step of generating context vectors based on spatiotemporal attention mechanism; wherein, in the process of performing the above-mentioned dynamic mapping learning for the context vector at each time step, the result of the above-mentioned image semantic encoding processing is: the image semantic feature embedding vector corresponding to each image region; the result of the above-mentioned text sequence prediction decoding processing is: the hidden state output at the previous time step.

[0050] Specifically, the attention module 220 is used to dynamically map and learn the correspondence between the importance of different image regions and the text prediction at each time step based on the image semantic feature embedding vector corresponding to each image region and the hidden state output at the previous time step, thereby obtaining the context vector for the global image at each time step.

[0051] In some embodiments, based on an attention mechanism, a dynamic mapping learning is performed to learn the correspondence between the importance of different image regions and the text prediction at each time step, resulting in a context vector corresponding to each time step, including:

[0052] For the current time step, based on the attention mechanism, importance analysis of different image regions is performed according to the image semantic feature embedding vectors corresponding to each image region and the hidden state output from the previous time step, so as to obtain the attention score of each image region for the current time step.

[0053] The attention weights for each image region at the current time step are normalized to obtain the attention weights for each image region at the current time step.

[0054] The image semantic feature embedding vectors corresponding to each image region are weighted and summed with the corresponding attention weights to obtain the context vector of the global image at the current time step.

[0055] Understandably, during dynamic mapping learning, in the first time step, since the text sequence prediction decoding model 240 has not yet output its hidden state, the default preset weight coefficients are used as the initial attention weights for each image region at t=1. Starting from the second time step, i.e., t≥2, importance analysis of different image regions is performed based on the image semantic feature embedding vectors corresponding to each image region and the hidden state output from the previous time step. This yields the attention score for each image region at the current time step, and after softmax normalization (as an example of normalization), the attention weights for each image region are obtained.

[0056] The aforementioned attention module 220 provides a dynamic weight allocation method based on the visual attention mechanism. It is set between the text sequence prediction and decoding model 240 and the image semantic coding model 210 to construct an information path for image-text fusion. Based on the aforementioned attention module 220, when generating each word, the model can automatically determine which region of the image should be focused on in the current context at each time step, and allocate different levels of attention weights among different regions of image features, thereby achieving more refined and goal-oriented text generation.

[0057] As an example, when t≥2, the input to the attention module 220 includes: the two-dimensional feature tensor A=[a1, a2, ..., ... corresponding to the image semantic feature embedding vectors of each image region. , ..., a n ] and the hidden state corresponding to the output at time step t-1. h t-1 .

[0058] Attention scores for each image region are calculated using a feedforward neural network, fattn. :

[0059] (1)

[0060] in, A generator function representing the attention score for an image region; express transpose; , , and b are network parameters in the feedforward neural network fattn, representing the degree of matching between image region features and language context state.

[0061] To obtain normalized attention weights, the system performs softmax normalization (as an example of normalization) on the scores of all regions, satisfying the following expression:

[0062] (2)

[0063] in, This represents the level of attention paid to the i-th image region when the t-th word is generated at the t-th time step, which is the attention weight of the i-th image region corresponding to the t-th time step.

[0064] The semantic feature embedding vectors corresponding to each image region are weighted and summed with their corresponding attention weights to obtain the context vector for the global image at time step t. As a dynamic summary of the overall semantics of the image at the current time step, the calculation process is as follows:

[0065] (3)

[0066] The context vector Subsequently, the word vector x obtained from the word embedding module 230 is compared with... t Together, they are input into the text sequence prediction and decoding model 240 to guide text prediction in the next time step. The entire interaction mechanism can be regarded as follows: the image semantic feature embedding vector obtained after image semantic encoding of the image file is weighted by spatiotemporal attention to obtain the context vector. The context vector, the word vector obtained by mapping the actual predicted text in the previous time step, and the hidden state output in the previous time step are used as inputs to the text sequence prediction and decoding model 240. This allows the text sequence prediction and decoding model 240 to flexibly adjust the focus of attention and contextual association of the image region in the text output at each time step, so as to express the connotation of the image region relatively accurately, especially for cases where there are complex microstructures or complex overall recognition semantics in the image file.

[0067] For example, it can accurately identify multiple anatomical structures and complex lesions in medical image files and automatically generate reports.

[0068] The word embedding module 230 described above is used to map the text predicted in the previous time step into word vectors.

[0069] In embodiments of this disclosure, the word embedding module 230 is used during the usage phase to embed the text result y predicted at the (t-1)th time step. t-1 The word vector is mapped to a word vector, which is then used as input to the decoding stage to guide the text generation at time step t.

[0070] The text result y predicted at time step t-1 t-1 The word vectors obtained by mapping are represented as follows , .

[0071] To establish an expressive channel between images and text, a word embedding module is introduced before the decoding stage to map the text vocabulary in the report into dense vector representations. During the training phase, all reports in the training samples undergo word segmentation, deduplication, and vocabulary construction, mapping each word to a corresponding integer number. Subsequently, an embedding layer is used to transform these numbers into fixed-dimensional vectors, typically 256-dimensional, consistent with image encoding, thus constructing the vocabulary. For sentences that are too short, sequence padding is used to unify them to the maximum length T, ensuring that each training sample has a consistent time-step structure.

[0072] That is, in order to facilitate the batch training of the model and matrix calculation in the future, the length of all report texts will be standardized in a uniform manner before the word vectors are input.

[0073] Length standardization operations include:

[0074] (1) Set the maximum length T (e.g., 128 or 256);

[0075] (2) If the report text length is less than T, then fill the end with a special placeholder (such as [PAD]);

[0076] (3) If the length is greater than T, then truncate it appropriately;

[0077] (4) Finally, the text input of each sample is converted into a vector sequence of length T to match the fixed time step input requirements of the subsequent text sequence prediction decoding model 240 (such as LSTM model).

[0078] During the training phase, at each time step, the text sequence prediction decoding model 240 receives the embedding vectors of the real label words as the current input (i.e., the teacher forcing strategy). This is combined with the image's context vector to participate in state updates, minimizing the loss function between the image report predicted text sequence and the real report text sequence output during the decoding phase. For example, this could be the cross-entropy loss function. In other words, at each time step, the input to the decoding phase is the real previous words (training labels) rather than the predicted words. This approach effectively avoids the cumulative amplification problem of error propagation during long sequence generation, thereby enhancing the stability of the training phase.

[0079] In practical use, the words generated in the previous step (the words with the highest probability selected from the Softmax output) are mapped to word vectors, which are then used as input for the next time step. Combined with the image context vector, the decoding continues until an end marker is generated.

[0080] The aforementioned text sequence prediction and decoding model 240 is used to predict the output text of the current time step based on the aforementioned word vectors, the aforementioned context vectors, and the hidden state output at the previous time step, and to execute multiple time steps to obtain the output text sequence; wherein, the aforementioned image report text is obtained based on the aforementioned output text sequence.

[0081] The aforementioned text sequence prediction decoding model 240 is mainly used for decoding processing. Its core task is to organically combine image encoding representation with language information and automatically generate image report text that conforms to professional description specifications through a serialization modeling mechanism.

[0082] The input to the text sequence prediction and decoding model 240 at each time step includes the following components:

[0083] (1) Word vectors : Represents the input word at the current time step, a dense vector obtained after transformation by the embedding layer. It serves as the training label during the training phase and as the word vector corresponding to the predicted text generated at the (t-1)th time step when the model is in use.

[0084] (2) The context vector of the global image at the t-th time step : The context representation calculated by the attention mechanism based on the image encoding features and the current decoding state.

[0085] (3) Hidden state at time step t-1 h t-1 : Represents the hidden memory state at time step t-1, which participates in state updates and text prediction.

[0086] In some embodiments, the network structure of the above-mentioned text sequence prediction and decoding model includes a Long Short-Term Memory (LSTM) network. The LSTM model is an improved gated recurrent neural network with stronger time-dependent modeling and memory retention capabilities.

[0087] For example, when processing at least one type of image file, such as remote sensing image file, material testing image file, or medical image file, the above LSTM model can be applied to the construction of multi-level semantic logic for various professional image reports.

[0088] Each image report is treated as a word sequence. During the generation process, the LSTM model predicts the next appropriate medical term or sentence unit step by step based on the vocabulary information of the previous time step, the hidden state, and the current context vector, thereby completing the automatic and natural growth of the sentence.

[0089] Figure 4 The diagram illustrates the structure and processing of a text sequence prediction decoding model according to an embodiment of the present disclosure.

[0090] Reference Figure 4 As shown, the internal mechanism of the LSTM model in the text generation process includes the input gate, forget gate, output gate, and memory state transition function. Its core objective is to determine whether the current input should be retained, whether the historical state should be forgotten, and when to output the current hidden state to predict the next word through a series of gating operations.

[0091] For time step t, the state update formula for the LSTM model is as follows:

[0092] (4-1)

[0093] (4-2)

[0094] (4-3)

[0095] (4-4)

[0096] (4-5)

[0097] (4-6)

[0098] in, The word vector is the current input (i.e., the input word vector corresponding to the t-th time step; the training labels are used during the training phase, and the word vector corresponding to the predicted text output at the (t-1)-th time step is used during the model usage phase). This represents the hidden state corresponding to the (t-1)th time step; This represents element-wise multiplication; σ and tanh are the Sigmoid and hyperbolic tangent activation functions, respectively. This represents different intermediate processing results, among which , , and For different preset parameters; , This represents the context vector corresponding to the (t-1)th and tth time steps of the long-term memory unit. The current output hidden state (i.e., the hidden state corresponding to the t-th time step) is used to predict the next word.

[0099] In the decoding process, the text sequence prediction decoding model 240 not only receives input from the word embedding module, but also integrates the context vector calculated by the attention mechanism in the previous round; the two together drive the evolution of the hidden state after merging, and then calculate the probability distribution of each word in the vocabulary through the Softmax layer.

[0100] For example, the following expression can be used:

[0101] (5)

[0102] in, This indicates the output text at time step t. The probability; Ws is the output mapping matrix; bs is the bias term.

[0103] The final model selects the word with the highest probability at each time step as the current prediction result and uses it as the input for the next time step, realizing a word-by-word progressive report building process until it encounters the end-of-time marker "endseq".

[0104] The method provided in this embodiment can be widely applied to end-to-end processing of various image files, such as video image files, picture image files, remote sensing image files, material testing image files, medical image files, etc., and can output professional image reports accordingly.

[0105] For example, an image report automatically generated after processing medical image files can include at least one of the following information: lesion description information, pathological interpretation information, and diagnostic suggestion information from the input image file.

[0106] In the embodiment including steps S110-S120, an image report generation model is constructed. This model can perform image semantic encoding processing, generate context vectors based on a spatiotemporal attention mechanism, and perform text sequence prediction and decoding processing on the input image file, outputting image report text. During the generation of the context vector based on the spatiotemporal attention mechanism, dynamic mapping learning is performed based on the correspondence between the importance of different image regions and the text prediction at each time step, according to the results of image semantic encoding processing and text sequence prediction and decoding processing, to obtain the context vector for the global image at each time step. The aforementioned context vector is obtained by weighting the results of image semantic encoding processing with attention weights. Therefore, in the process of generating the context vector, the image report is generated... During word prediction, the text predicted at each time step takes into account the relative importance of each region in the global image. This dynamic mapping learning process combines the decoding results of the previous time step with the results of image semantic encoding processing for each region of the image, and integrates differentiated attention in the temporal dimension and global attention weights in the spatial dimension. This reflects the effect of spatiotemporal cross-learning and precise correspondence with image regions. It can not only improve the generalization performance of the above image report generation model for various types of image files, but also output more accurate image report text for image files in relatively professional fields (such as remote sensing image files, material testing image files, medical image files, etc.). Moreover, the end-to-end processing time is very short, improving the processing efficiency for a large number of image files.

[0107] Figure 3 The diagram illustrates a simplified process for processing medical image files using the end-to-end image file processing method provided in this disclosure.

[0108] Using medical image files as examples of image files to be processed, the aforementioned medical image files are input into a pre-built image report generation model. Based on the image semantic coding model 210, the aforementioned medical image files undergo image semantic coding processing to obtain the image semantic feature embedding vectors [a1, a2, ... ] corresponding to each image region. , ..., a n Based on the attention module 220, the image semantic features corresponding to each image region are used to embed vectors [a1, a2, ..., ...]. , ..., a n [and the hidden state output from the previous time step] h t-1 Based on an attention mechanism, a dynamic mapping learning is performed to learn the correspondence between the importance of different image regions and the text prediction at each time step, resulting in the context vector c1 for the global image at each time step. Based on the word embedding module (in order to simplify the data flow, in...) Figure 3 (Not illustrated in the text) Map the text predicted at the previous time step to word vectors; based on the text sequence prediction and decoding model 240, according to the above word vectors and the context vectors c1~ at each time step... The hidden state output by the previous time step and the current time step. h t-1 To predict the output text at the current time step, and execute multiple time steps (e.g., the total number of time steps is T) to obtain the output text sequence [y1, y2, ..., y]. t …,y T ], corresponding to Figure 3 The text in the diagram illustrates the text output for each time step, such as crack, deformity, bone, etc.

[0109] In some embodiments, the output text sequence obtained by the above-described text sequence prediction and decoding model 240 can be directly output as image report text.

[0110] In other embodiments, for medical image files, in order to further improve the reliability and professionalism of the output results, the above-mentioned image report generation model 200 also includes a verification module.

[0111] The aforementioned verification module is used to match the lesion location and lesion type in the image described in the aforementioned output text sequence with a manually annotated clinical case file library, to perform medical consistency verification between the aforementioned output text sequence and the matched target clinical case file, and to output the aforementioned image report text if the verification is successful.

[0112] In some embodiments, the image report generation model 200 described above further includes an image region marking module.

[0113] The aforementioned image region labeling module is used to associate corresponding image regions in the aforementioned image file based on key information in the aforementioned image report text and to perform visual labeling on the corresponding image regions. Specifically, associating corresponding image regions in the aforementioned image file based on key information in the image report text includes: associating the image region corresponding to the maximum attention score for different image regions based on the maximum attention score obtained by the aforementioned attention module when generating the context vector at each time step.

[0114] Because the attention mechanism introduces explicit intermediate variables (attention weights) This mechanism can be aligned with manually annotated lesion areas or image interpretation areas, which is beneficial for improving interpretability under supervised learning. Furthermore, it possesses strong scalability and can be integrated with strategies such as location encoding, multi-head attention structures, and bidirectional interaction mechanisms in the future to further enhance the coupling and expressive ability of image content and text semantics. Therefore, the aforementioned attention mechanism not only connects the encoding and decoding stages in terms of technical structure, but also, as a dynamic interface, greatly improves the contextual relevance of the text generation process, the directionality of key image areas, and the sensitivity of visual attention, forming the foundation for supporting the high-quality automated generation of image reports.

[0115] In the above embodiments, the generated reports are supported to be annotated in a structured manner and to be medically consistent with real cases. This enables continuous optimization of the model under unsupervised and semi-supervised conditions, further improving the reliability and professionalism of the model in the scenario of automated generation of medical image files.

[0116] In some embodiments, the above-described image report generation model is constructed in the following manner:

[0117] Obtain multiple training image files and multiple corresponding training image reports from the training database;

[0118] The text sequences corresponding to each training image report are length aligned, and the length-aligned text sequences are then segmented, deduplicated, and mapped using word encoding to generate training word vector sequences corresponding to the text sequences of each training image report. The word vectors obtained through word encoding mapping have the same encoding dimension as those encoded for the aforementioned training image files.

[0119] The above-mentioned training image report is input into the target neural network model to be trained, and the output image report prediction text sequence is generated. The above-mentioned training word vector sequence is used as training labels to train the above-mentioned target neural network model. Based on the trained target neural network model, the image report generation model is constructed. The above-mentioned target neural network model has the same structure as the above-mentioned image report generation model but different parameters.

[0120] To ensure the actual performance of the image report generation model built from the target neural network model in the image report generation task, a complete end-to-end training mechanism was constructed. This mechanism aims to maximize the matching degree between the model-generated reports and real reports, enabling the model to possess stable language expression capabilities and professional-domain image semantic understanding capabilities. During the training phase, the system uses data pairs (I, S) consisting of training image files and corresponding manually annotated training image reports (which are real reports) as the training set, where I represents the preprocessed training image file, and S = [Y1, Y2, ..., Y...]. t …, Y TLet T be the text sequence corresponding to the image report used for training. The training objective is to minimize the difference between the predicted text sequence of the image report and the actual report, i.e., to maximize the conditional probability. .

[0121] The negative log-likelihood loss function is used as the optimization objective.

[0122] (6)

[0123] Where θ represents the set of all learnable parameters of the target neural network model, including parameters from various modules / models such as the image semantic coding model 210, the attention module 220, the word embedding module 230, and the text sequence prediction and decoding model 240 (e.g., an LSTM model). By modeling conditional probabilities using the chain rule, the system models the text sequence as a product of multiple conditional probabilities:

[0124] (7)

[0125] The sequence prediction process is implemented recursively using an SLTM model, where the output word at each step is determined by the hidden state at the current time step. h t The probability distribution on the vocabulary is obtained through Softmax mapping. The prediction of each word is compared with the true word using the cross-entropy loss function, and the loss is accumulated over all training samples to drive parameter updates.

[0126] Specifically, the target neural network model is trained using an adaptive gradient descent algorithm; the target neural network model is considered to be trained when the loss function is less than a set threshold or the number of training iterations exceeds a set value; the loss function is used to represent the result of accumulating the cross-entropy loss between the image report prediction text sequence and the training label at each time step according to the time step.

[0127] Training based on adaptive gradient descent algorithms (such as Adam) allows for adaptive adjustment of the learning rate across different parameter dimensions, resulting in more stable convergence in scenarios with fewer samples. For example, when applied to the generation of medical image reports, it helps achieve more stable convergence in situations with small sample sizes of medical images.

[0128] In some embodiments, the training image files in the training database include at least some of the following types: video image files, picture image files, remote sensing image files, material testing image files, and medical image files. When the training image files include medical image files, these medical image files cover a variety of disease types and multiple image modalities.

[0129] For example, when the training image files include medical image files, by balancing the proportions of different types of diseases and different image modalities (such as CT, MRI, X-ray) during training, the model can be ensured to have good transfer and generalization capabilities in multiple medical scenarios.

[0130] In some embodiments, before performing length alignment on the text sequences, the method further includes: performing expression standardization on the text sequences corresponding to each training image report. This expression standardization includes at least one of the following operations: unifying terminology style, standardizing syntactic structure, and removing redundant modifiers. Expression standardization helps enhance the professionalism and consistency of the model output.

[0131] A second exemplary embodiment of this disclosure provides a method for constructing an image report generation model.

[0132] Figure 5 A flowchart illustrating a method for constructing an image report generation model according to an embodiment of the present disclosure is shown schematically.

[0133] Reference Figure 5 As shown, the method for constructing the image report generation model in this embodiment includes the following steps: S510, S520 and S530.

[0134] In step S510, multiple training image files and multiple corresponding training image reports are obtained from the training database.

[0135] In step S520, the text sequence corresponding to each training image report is length aligned, and the length-aligned text sequence is segmented, deduplicated, and encoded to generate the training word vector sequence corresponding to the text sequence of each training image report.

[0136] The word vectors obtained through lexical encoding mapping have the same encoding dimension as those encoded for the aforementioned training image files.

[0137] In step S530, the above-mentioned training image report is input into the target neural network model to be trained for image semantic encoding processing, generation of context vectors based on spatiotemporal attention mechanism and text sequence prediction decoding processing, outputting the image report prediction text sequence, using the above-mentioned training word vector sequence as training labels to train the above-mentioned target neural network model, and constructing the image report generation model based on the trained target neural network model.

[0138] The generation of context vectors based on the spatiotemporal attention mechanism includes: based on the results of image semantic encoding and text sequence prediction decoding, dynamically mapping and learning the correspondence between the importance of different image regions and the text prediction at each time step, based on the attention mechanism, to obtain the context vector for the global image at each time step; the above context vector is obtained by weighting the results of image semantic encoding with attention weights.

[0139] In some embodiments, the training image files in the training database include at least some of the following types: video image files, picture image files, remote sensing image files, material testing image files, and medical image files. When the training image files include medical image files, these medical image files cover a variety of disease types and multiple image modalities.

[0140] In some embodiments, before performing length alignment on the text sequences, the method further includes: performing expression standardization on the text sequences corresponding to each training image report. This expression standardization includes at least one of the following operations: unifying terminology style, standardizing syntactic structure, and removing redundant modifiers. Expression standardization helps enhance the professionalism and consistency of the model output.

[0141] More details and beneficial effects of this embodiment can be found in the relevant description of the first embodiment, which will not be repeated here.

[0142] A third exemplary embodiment of this disclosure provides an end-to-end image file processing apparatus.

[0143] The end-to-end image file processing device in this embodiment includes a file acquisition module and an image report automatic generation module.

[0144] The file acquisition module described above is used to acquire the image files to be processed.

[0145] The aforementioned image report automatic generation module is used to input the image files into a pre-built image report generation model for image semantic encoding processing, generation of context vectors based on spatiotemporal attention mechanism, and text sequence prediction and decoding processing, and output the image report text.

[0146] The generation of context vectors based on the spatiotemporal attention mechanism includes: based on the results of image semantic encoding and text sequence prediction decoding, dynamically mapping and learning the correspondence between the importance of different image regions and the text prediction at each time step, based on the attention mechanism, to obtain the context vector for the global image at each time step; the above context vector is obtained by weighting the results of image semantic encoding with attention weights.

[0147] In some embodiments, the image report generation model includes: an image semantic encoding model, an attention module, a word embedding module, and a text sequence prediction and decoding model. The image semantic encoding model performs image semantic encoding processing on the image file to obtain image semantic feature embedding vectors corresponding to each image region. The attention module performs the step of generating context vectors based on a spatiotemporal attention mechanism; wherein, during the dynamic mapping learning process for the context vector at each time step, the result of the image semantic encoding processing is: image semantic feature embedding vectors corresponding to each image region; the result of the text sequence prediction and decoding processing is: the hidden state output at the previous time step. The word embedding module maps the text predicted at the previous time step to word vectors. The text sequence prediction and decoding model predicts the output text at the current time step based on the word vectors, the context vectors, and the hidden state output at the previous time step, and performs multiple time steps to obtain an output text sequence; wherein, the image report text is obtained based on the output text sequence.

[0148] In some embodiments, the aforementioned image file includes a medical image file; the aforementioned image report generation model further includes a verification module. The verification module is used to match the lesion location and lesion type in the image described in the aforementioned output text sequence against a manually annotated clinical case file library, perform medical consistency verification between the aforementioned output text sequence and the matched target clinical case file, and output the aforementioned image report text if the verification passes.

[0149] In some embodiments, the image report generation model further includes an image region labeling module. This image region labeling module is used to associate corresponding image regions in the image file based on key information in the image report text and to perform visual labeling on the corresponding image regions. Associating corresponding image regions in the image file based on key information in the image report text includes: associating the image region corresponding to the maximum attention score from the attention module when generating the context vector at each time step with the image region corresponding to the maximum attention score.

[0150] In some embodiments, the network structure of the above-described image semantic coding model includes: a Region Convolutional Neural Network (R-CNN) with a deep convolutional neural network (VGG16) as its backbone. The network structure of the above-described text sequence prediction and decoding model includes: a Long Short-Term Memory (LSTM) network.

[0151] More details and beneficial effects of this embodiment can be found in the relevant description of the first embodiment, which will not be repeated here.

[0152] A fourth exemplary embodiment of this disclosure provides an apparatus for constructing an image report generation model.

[0153] The image report generation model construction device in this embodiment includes: a training data acquisition module, a label processing module, and a model construction module.

[0154] The training data acquisition module described above is used to acquire multiple training image files and multiple corresponding training image reports from the training database.

[0155] The aforementioned tag processing module is used to perform length alignment processing on the text sequences corresponding to each training image report, and to perform word segmentation, lexical deduplication, and lexical encoding mapping processing on the length-aligned text sequences to generate training word vector sequences corresponding to the text sequences of each training image report; wherein the word vectors obtained through lexical encoding mapping have the same encoding dimension as those encoded for the aforementioned training image files.

[0156] The aforementioned model construction module is used to input the training image report into the target neural network model to be trained for image semantic encoding processing, generation of context vectors based on a spatiotemporal attention mechanism, and text sequence prediction and decoding processing, outputting an image report prediction text sequence. The training word vector sequence is used as training labels to train the target neural network model, and an image report generation model is constructed based on the trained target neural network model. Specifically, generating the context vector based on the spatiotemporal attention mechanism includes: based on the results of image semantic encoding processing and text sequence prediction and decoding processing, dynamically mapping and learning the correspondence between the importance of different image regions and the text prediction at each time step based on an attention mechanism, obtaining a context vector for the global image at each time step; the aforementioned context vector is obtained by weighting the results of image semantic encoding processing with attention weights.

[0157] More details and beneficial effects of this embodiment can be found in the descriptions of the first and second embodiments, which will not be repeated here.

[0158] Any plurality of the functional modules included in the apparatus of the third embodiment or the apparatus of the fourth embodiment described above can be combined into one module, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules can be combined with at least part of the functionality of other modules and implemented in one module. At least one of the functional modules included in the apparatus of the third embodiment or the apparatus of the fourth embodiment described above can be at least partially implemented as hardware circuitry, such as field-programmable gate arrays (FPGAs), programmable logic arrays (PLAs), systems-on-a-chip, systems-on-a-substrate, systems-on-package, application-specific integrated circuits (ASICs), or implemented by any other reasonable means of integrating or packaging circuitry, or implemented in any one of software, hardware, and firmware methods, or in a suitable combination of any of these. Alternatively, at least one of the functional modules included in the apparatus of the third embodiment or the apparatus of the fourth embodiment described above can be at least partially implemented as a computer program module, which, when run, can perform corresponding functions.

[0159] The fifth exemplary embodiment of this disclosure provides an electronic device.

[0160] Figure 6 A schematic block diagram of an electronic device provided in an embodiment of the present disclosure is shown.

[0161] Reference Figure 6 As shown, the electronic device 600 provided in this embodiment includes a processor 601, a communication interface 602, a memory 603, and a communication bus 604. The processor 601, the communication interface 602, and the memory 603 communicate with each other through the communication bus 604. The memory 603 is used to store computer programs. When the processor 601 executes the program stored in the memory, it implements the end-to-end image file processing method or the image report generation model construction method as described above.

[0162] A sixth exemplary embodiment of this disclosure also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program that, when executed by a processor, implements the end-to-end image file processing method or the method for constructing an image report generation model as described above.

[0163] The computer-readable storage medium may be included in the device or apparatus described in the above embodiments; or it may exist independently and not assembled into the device or apparatus. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0164] According to embodiments of this disclosure, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0165] It should be noted that the collection, gathering, updating, analysis, processing, use, transmission, and storage of user personal information involved in the technical solutions provided in this disclosure comply with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken to prevent unauthorized access to user personal information data and to safeguard user personal information security and network security.

[0166] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0167] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. An end-to-end image file processing method, characterized in that, include: Obtain the image file to be processed; The image file is input into a pre-built image report generation model for image semantic encoding, context vector generation based on spatiotemporal attention mechanism and text sequence prediction and decoding, and the output is the image report text. The generation of context vectors based on the spatiotemporal attention mechanism includes: based on the results of image semantic encoding processing and text sequence prediction decoding processing, and based on the attention mechanism, dynamically mapping and learning the correspondence between the importance of different image regions and the text prediction at each time step to obtain the context vector for the global image at each time step; the context vector is obtained by weighting the results of image semantic encoding processing with attention weights. Based on an attention mechanism, a dynamic mapping learning is performed to learn the correspondence between the importance of different image regions and the text prediction at each time step, resulting in a context vector for each time step, including: For the current time step, based on the attention mechanism, importance analysis of different image regions is performed according to the image semantic feature embedding vectors corresponding to each image region and the hidden state output from the previous time step, so as to obtain the attention score of each image region for the current time step. The attention weights for each image region at the current time step are normalized to obtain the attention weights for each image region at the current time step. The image semantic feature embedding vectors corresponding to each image region are weighted and summed with the corresponding attention weights to obtain the context vector of the global image at the current time step.

2. The method according to claim 1, characterized in that, The image report generation model includes: an image semantic encoding model, an attention module, a word embedding module, and a text sequence prediction and decoding model; The image semantic coding model is used to perform image semantic coding processing on the image file to obtain the image semantic feature embedding vector corresponding to each image region; The attention module is used to perform the step of generating context vectors based on the spatiotemporal attention mechanism; wherein, during the dynamic mapping learning process for the context vector at each time step, the result of the image semantic encoding processing is: the image semantic feature embedding vector corresponding to each image region; the result of the text sequence prediction decoding processing is: the hidden state output at the previous time step; The word embedding module is used to map the text predicted in the previous time step into word vectors; The text sequence prediction and decoding model is used to predict the output text of the current time step based on the word vector, the context vector, and the hidden state output at the previous time step, and to execute multiple time steps to obtain the output text sequence; wherein, the image report text is obtained based on the output text sequence.

3. The method according to claim 2, characterized in that, The image files include: medical image files; the image report generation model also includes: The verification module is used to match the lesion location and lesion type in the image described in the output text sequence with a manually annotated clinical case file library, perform medical consistency verification between the output text sequence and the matched target clinical case file, and output the image report text if the verification is successful.

4. The method according to any one of claims 2-3, characterized in that, The image report generation model also includes: The image region marking module is used to associate the corresponding image region in the image file with the key information in the image report text and to perform visual marking on the corresponding image region. Among them, associating the corresponding image region in the image file with key information in the image report text includes: associating the image region corresponding to the maximum attention score of different image regions based on the maximum attention score of the attention module when generating the context vector at each time step.

5. The method according to any one of claims 2-3, characterized in that, The network structure of the image semantic coding model includes: a region convolutional neural network R-CNN with a deep convolutional neural network VGG16 as the backbone network structure. The network structure of the text sequence prediction and decoding model includes a long short-term memory network.

6. The method according to claim 1, characterized in that, The image report generation model is constructed in the following way: Obtain multiple training image files and multiple corresponding training image reports from the training database; The text sequences corresponding to each training image report are length aligned, and the length-aligned text sequences are then segmented, deduplicated, and mapped using word encoding to generate training word vector sequences corresponding to the text sequences of each training image report. The word vectors obtained through word encoding mapping have the same encoding dimension as those encoded for the training image files. The training image report is input into the target neural network model to be trained, and the output image report prediction text sequence is generated. The training word vector sequence is used as training labels to train the target neural network model. Based on the trained target neural network model, an image report generation model is constructed. The target neural network model and the image report generation model have the same structure but different parameters.

7. The method according to claim 6, characterized in that, The training database contains multiple training image files, including at least some of the following types: video image files, picture image files, remote sensing image files, material testing image files, and medical image files. In cases where the training image files include medical image files, the medical image files cover multiple disease types and multiple image modalities; Before performing length alignment on the text sequence, the method further includes: The text sequences corresponding to each training image report are subjected to expression standardization processing, which includes at least one of the following operations: unifying terminology style, standardizing syntactic structure, and removing redundant modifiers; The target neural network model is trained using an adaptive gradient descent algorithm. The target neural network model is considered to have completed training if the loss function is less than a set threshold or the number of training iterations exceeds a set value. The loss function is used to represent the result of accumulating the cross-entropy loss between the image report prediction text sequence and the training label at each time step according to the time step.

8. A method for constructing an image report generation model, characterized in that, include: Obtain multiple training image files and multiple corresponding training image reports from the training database; The text sequences corresponding to each training image report are length aligned, and the length-aligned text sequences are then segmented, deduplicated, and mapped using word encoding to generate training word vector sequences corresponding to the text sequences of each training image report. The word vectors obtained through word encoding mapping have the same encoding dimension as those encoded for the training image files. The training image report is input into the target neural network model to be trained for image semantic encoding, generation of context vectors based on spatiotemporal attention mechanism and text sequence prediction and decoding, and output of image report prediction text sequence. The training word vector sequence is used as training labels to train the target neural network model. Based on the trained target neural network model, an image report generation model is constructed. The generation of context vectors based on the spatiotemporal attention mechanism includes: based on the results of image semantic encoding processing and text sequence prediction decoding processing, and based on the attention mechanism, dynamically mapping and learning the correspondence between the importance of different image regions and the text prediction at each time step to obtain the context vector for the global image at each time step; the context vector is obtained by weighting the results of image semantic encoding processing with attention weights. Based on an attention mechanism, a dynamic mapping learning is performed to learn the correspondence between the importance of different image regions and the text prediction at each time step, resulting in a context vector for each time step, including: For the current time step, based on the attention mechanism, importance analysis of different image regions is performed according to the image semantic feature embedding vectors corresponding to each image region and the hidden state output from the previous time step, so as to obtain the attention score of each image region for the current time step. The attention weights for each image region at the current time step are normalized to obtain the attention weights for each image region at the current time step. The image semantic feature embedding vectors corresponding to each image region are weighted and summed with the corresponding attention weights to obtain the context vector of the global image at the current time step.

9. An end-to-end image file processing apparatus, characterized in that, include: The file acquisition module is used to acquire the image files to be processed. The image report automatic generation module is used to input the image file into a pre-built image report generation model for image semantic encoding processing, generating context vectors based on spatiotemporal attention mechanism and text sequence prediction and decoding processing, and outputting image report text. The generation of context vectors based on the spatiotemporal attention mechanism includes: based on the results of image semantic encoding processing and text sequence prediction decoding processing, and based on the attention mechanism, dynamically mapping and learning the correspondence between the importance of different image regions and the text prediction at each time step to obtain the context vector for the global image at each time step; the context vector is obtained by weighting the results of image semantic encoding processing with attention weights. Based on an attention mechanism, a dynamic mapping learning is performed to learn the correspondence between the importance of different image regions and the text prediction at each time step, resulting in a context vector for each time step, including: For the current time step, based on the attention mechanism, importance analysis of different image regions is performed according to the image semantic feature embedding vectors corresponding to each image region and the hidden state output from the previous time step, so as to obtain the attention score of each image region for the current time step. The attention weights for each image region at the current time step are normalized to obtain the attention weights for each image region at the current time step. The image semantic feature embedding vectors corresponding to each image region are weighted and summed with the corresponding attention weights to obtain the context vector of the global image at the current time step.

10. A device for constructing an image report generation model, characterized in that, include: The training data acquisition module is used to acquire multiple training image files and multiple corresponding training image reports from the training database. The tag processing module is used to perform length alignment processing on the text sequences corresponding to each training image report, and to perform word segmentation, word deduplication, and word encoding mapping processing on the length-aligned text sequences to generate training word vector sequences corresponding to the text sequences of each training image report; wherein the word vectors obtained by word encoding mapping have the same encoding dimension as the encoding of the training image file. The model building module is used to input the training image report into the target neural network model to be trained for image semantic encoding processing, generate context vectors based on spatiotemporal attention mechanism and text sequence prediction and decoding processing, output image report prediction text sequence, use the training word vector sequence as training labels to train the target neural network model, and build an image report generation model based on the trained target neural network model. The generation of context vectors based on the spatiotemporal attention mechanism includes: based on the results of image semantic encoding processing and text sequence prediction decoding processing, and based on the attention mechanism, dynamically mapping and learning the correspondence between the importance of different image regions and the text prediction at each time step to obtain the context vector for the global image at each time step; the context vector is obtained by weighting the results of image semantic encoding processing with attention weights. Based on an attention mechanism, a dynamic mapping learning is performed to learn the correspondence between the importance of different image regions and the text prediction at each time step, resulting in a context vector for each time step, including: For the current time step, based on the attention mechanism, importance analysis of different image regions is performed according to the image semantic feature embedding vectors corresponding to each image region and the hidden state output from the previous time step, so as to obtain the attention score of each image region for the current time step. The attention weights for each image region at the current time step are normalized to obtain the attention weights for each image region at the current time step. The image semantic feature embedding vectors corresponding to each image region are weighted and summed with the corresponding attention weights to obtain the context vector of the global image at the current time step.

11. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method of any one of claims 1-8.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1-8.

Images