Image feature extraction structure, network, training method and multi-modal prediction system

Abstract

The invention provides an image feature extraction structure, a network, a training method and a multi-modal prediction system, the structure comprises a plurality of paths for parallel processing and a feature fusion layer, the plurality of paths are configured to perform attention calculation on input features under different feature map scales, the input features of different feature map scales used by each path are determined by down-sampling original input features through different pooling step lengths; the feature fusion layer is configured to fuse the output features of the plurality of paths; in the plurality of paths, the scale of the input features on which attention calculation is carried out is in negative correlation with the number of the configured parallel attention calculation modules. According to the structure, for large-scale features needing fine local information, only a small number of attention calculation modules are allocated for processing; and for the small-scale features bearing the advanced semantic information, a large amount of attention is put into processing, so that the optimization of the calculation efficiency is realized on the algorithm level.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to an image feature extraction structure, a network, a training method and a multi-modal prediction system. BACKGROUND

[0002] In the field of pathological prognosis prediction, especially for the prediction of postoperative survival time of patients with advanced cancer, clinicians need to comprehensively analyze the physiological characteristics and pathological characteristics of patients. Physiological characteristics such as gender and age are relatively clear, while pathological characteristics are more complex, including disease site, chief complaint text description, and high-resolution medical images, etc. These multi-source heterogeneous data contain a large amount of redundancy and noise. Traditionally, doctors rely on manual screening and interpretation, which not only consumes time and effort, but also has a large subjective experience influence on the consistency of diagnosis. With the development of artificial intelligence technology, using deep neural networks to assist in prognosis analysis has become an important research direction.

[0003] At present, researchers have tried to build multi-modal neural network models to integrate image and text information to improve prediction accuracy. However, the image feature extraction backbone network in such models mostly follows conventional convolutional neural networks or early visual Transformer architecture. Convolutional neural networks are good at capturing local spatial features, but their inherent inductive bias limits their ability to model long-range dependencies, and they are inefficient in processing sequential or global context information. Standard visual Transformer models have advantages in feature expression ability due to their powerful global self-attention mechanism, but their computational complexity is proportional to the square of the input image size. When processing high-resolution medical images, the huge computational overhead results in slow model inference speed, making it difficult to meet the real-time requirements of clinical practice, and severely limiting its deployment and application in resource-constrained environments. SUMMARY

[0004] One of the purposes of the present application is to provide an image feature extraction structure to solve the technical problem that the image feature extraction method based on deep learning in the prior art is difficult to achieve an effective balance between high-precision prediction ability of the model and acceptable computational efficiency when applied to high-resolution image analysis.

[0005] To achieve the above-mentioned one of the purposes of the application, the application provides an image feature extraction structure, comprising a plurality of parallel processing paths and a feature fusion layer: the plurality of paths are configured to perform attention calculation on input features at different feature map scales respectively, wherein the input features at different feature map scales used by each path are determined by downsampling the original input features with different pooling steps; the feature fusion layer is configured to fuse the output features of the plurality of paths; wherein in the plurality of paths, the scale of the input features on which the attention calculation is performed is negatively correlated with the number of parallel attention calculation modules configured.

[0006] As a further improvement of an embodiment of the application, the number of the plurality of paths is three, and the feature map scales of the three paths decrease in turn.

[0007] As a further improvement of an embodiment of the application, in the path with the largest scale, the attention calculation is limited to be performed within a spatial local window.

[0008] As a further improvement of an embodiment of the application, the image feature extraction structure is an attention enhancement module, the attention enhancement module further comprises: a first layer normalization module, an input end of which accepts input features, and an output end of which is connected to an input end of the plurality of paths; a feedforward network module; a second layer normalization module connected between the feature fusion layer and the feedforward network module; wherein the output features of the feature fusion layer are added to the input end of the first layer normalization module through a first residual connection; the output features of the feedforward network module are added to the input end of the second layer normalization module through a second residual connection.

[0009] As a further improvement of an embodiment of the application, the feedforward network module comprises a first linear transformation layer, a nonlinear activation layer and a second linear transformation layer connected in turn.

[0010] As a further improvement of an embodiment of the application, the feedforward network module further comprises a depthwise convolution layer connected between the first linear transformation layer and the nonlinear activation layer.

[0011] As a further improvement of an embodiment of the application, the feedforward network module comprises a first fully connected layer, a depthwise convolution layer, an activation function layer, a first dropout layer, a second fully connected layer and a second dropout layer connected in turn.

[0012] As a further improvement of the embodiment of the present application, in the plurality of paths, each path comprises: a query generation module configured to generate a query vector based on input features of the path; a key generation module configured to generate a key vector based on the input features of the path; a value generation module configured to generate a value vector based on the input features of the path; and an attention calculation module, an input end of which is connected to output ends of the query generation module, the key generation module and the value generation module respectively, to accept the query vector, the key vector and the value vector and perform the attention calculation.

[0013] In order to achieve one of the above-mentioned purposes, the present application provides an image feature extraction network, comprising: an image block module for converting an input image into an image block sequence; at least one down-sampling and processing module, an input end of which is connected to an output end of the image block module, for performing feature processing on the input image block sequence; at least one image feature extraction structure according to any one of claims 1-8, an input end of which is connected to an output end of the last down-sampling and processing module; and a first classification module, an input end of which is connected to an output end of the image feature extraction structure.

[0014] As a further improvement of the embodiment of the present application, the sampling and processing module comprises a down-sampling module and a feature transformation module connected in sequence, and the feature transformation module comprises: a third layer normalization module, an input end of which accepts input features; and a feedforward network module, an input end of which is connected to an output end of the third layer normalization module; wherein an output feature of the feedforward network module is added to the input end of the third layer normalization module through a third residual connection.

[0015] In order to achieve one of the above-mentioned purposes, the present application provides a multi-modal prediction model, comprising: an image feature extraction network according to any one of the preceding description; a text feature extraction network; a feature fusion module, input ends of which are connected to output ends of the image feature extraction network and the text feature extraction network respectively; and a classification module, an input end of which is connected to an output end of the feature fusion module.

[0016] In order to achieve one of the above-mentioned purposes, the present application provides a training method of an image feature extraction network, the method comprising: obtaining a training image dataset containing labeled categories; inputting the training image dataset into the image feature extraction network for training to obtain predicted categories; determining a loss function value based on the predicted categories and corresponding labeled categories; and updating parameters of the image feature extraction network based on the loss function value until the function loss value converges, to obtain a trained image feature extraction network.

[0017] Compared with the prior art, the embodiments of the present application have at least one of the following beneficial effects: The application discloses an image feature extraction structure. By establishing parallel multi-scale processing paths, the "intensity of investment of computing resources" and the "computing cost of feature processing" are decoupled and inversely associated, that is, for large-scale features requiring fine local information, only a small number of attention computing modules are allocated for necessary processing; and for small-scale features bearing high-level semantic information, a large number of attention computing modules are invested for deep and global modeling. This "local conditions" resource scheduling strategy makes the expensive global attention calculation mainly occur in the path with the smallest computing base, thereby realizing optimization of computing efficiency at the algorithm level. Finally, the feature fusion layer organically integrates the local details and global semantics extracted by different paths, forming a feature representation that is both efficient and strong in expression. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 is a structural schematic diagram of an image feature extraction structure 100 in an embodiment of the application.

[0019] Figure 2 is a structural schematic diagram of a Transformer-based image feature extraction structure (MPT) 100 in an embodiment of the application.

[0020] Figure 3 is a structural schematic diagram of an attention enhancement module (Atten-Block) in an embodiment of the application.

[0021] Figure 4 is a structural schematic diagram of an image feature extraction network 300 in an embodiment of the application.

[0022] Figure 5 is a structural schematic diagram of a feature transformation module (MLP-Block) in an embodiment of the application.

[0023] Figure 6 is a structural schematic diagram of a multi-modal prediction model 400 in an embodiment of the application.

[0024] Figure 7 is a step schematic diagram of a training method of an image feature extraction network in an embodiment of the application. DETAILED DESCRIPTION

[0025] The application will be described in detail below with reference to the specific embodiments shown in the drawings. However, these embodiments do not limit the application, and the changes made by those of ordinary skill in the art in structure, method, or function based on these embodiments are all included in the protection scope of the application.

[0026] In the description of the present application, it should be noted that unless otherwise explicitly specified and limited, the terms "mounting", "connection", "connecting" should be understood in a broad sense, for example, can be fixed connection, can also be detachable connection, or integral connection; can be mechanical connection, can also be electrical connection; can be direct connection, can also be indirect connection through intermediate medium, can be internal communication of two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0027] It should be noted that the term "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or apparatus including a series of elements includes not only those elements, but also other elements not explicitly listed, or further includes elements inherent to such a process, method, article or apparatus. In addition, the terms "first", "second", etc. are only used for descriptive purposes and should not be construed as indicating or implying relative importance.

[0028] As shown in Figure 1 An embodiment of the present application provides an image feature extraction structure 100.

[0029] The image feature extraction structure 100 is a neural network module for receiving an image or a feature representation of the image, and outputting a feature vector representing the core information of the image through a series of processes (such as sampling and attention calculation).

[0030] The image feature extraction structure 100 includes multiple parallel processing paths 11, which are configured to perform attention calculation on input features at different feature map scales respectively, wherein the input features at different feature map scales used by each path are determined by downsampling the original input features with different pooling steps.

[0031] Among them, the feature map scale refers to the two-dimensional spatial size of the feature map, that is, the spatial resolution (height H Width W or the number of channels C). Different feature map scales also refer to different spatial resolutions, for example, [H, W], [H / 2, W / 2] and [H / 4, W / 4], which show the gradual halving of spatial size (or spatial scale), and the corresponding spatial resolution is also gradually decreasing. When the scale is large, the feature map size is large, which retains the details and local information of the image, and the calculation amount is large; when the scale is small, the feature map size is small, which retains the compressed and abstracted high-level semantic information, and the calculation amount is small.

[0032] In the present application, the input features of different feature map scales technically specifically refer to multiple feature maps with the same number of channels C but different spatial resolutions (H, W), and the different feature map scales refer to allowing different paths to process feature maps of different resolutions (i.e., different levels of detail) so as to capture comprehensive information from details to semantics.

[0033] In addition, downsampling refers to a process of reducing the spatial resolution (i.e., the size of the spatial dimension) of data (such as images, feature maps), and the downsampling can be completed by a pooling operation. Specifically, the values within a pooling window are combined into one value, the pooling window is used to determine the range of the combined area, and the pooling step is used to determine the distance of the movement of the pooling window each time. For example, the pooling window is 2 2, and the pooling step is 1, which means that the pooling window moves 2 pixels each time, and the height H and the width W of the output feature map are each reduced to half of the original, i.e., from [H, W] to [H / 2, W / 2].

[0034] In an embodiment, the multiple paths generate features of different scales through mean pooling layers with different parameters.

[0035] The original input features refer to the image features input to the image feature extraction structure 100 that have not yet been processed by the structure. Based on the position of the image feature extraction structure 100 in the network, it refers to different meanings.

[0036] In a specific embodiment, when the image feature extraction structure 100 is the first layer of the network, the original input features refer to the original image pixel blocks after preprocessing (for example, a sequence formed after the image is divided into a plurality of pixel blocks).

[0037] In a specific embodiment, when the image feature extraction structure 100 is an intermediate layer of the network, the original input features refer to the feature maps output by the previous layer of the network.

[0038] In an embodiment, the image feature extraction structure 100 is improved based on the form of neuron connection in the Transformer structure.

[0039] It should be noted that the input features of different feature map scales refer to multiple feature maps of different scales formed after the original input features are subjected to different preliminary processing (i.e., downsampling with different pooling steps) by the multiple paths.

[0040] Furthermore, in multiple paths 11, the scale of the input features used for attention computation is negatively correlated with the number of parallel attention computation modules configured. This means that when the input features on a path have a large scale in the spatial dimension, the computational resources allocated to it (i.e., the number of attention heads in the attention computation module) will be reduced accordingly; conversely, when the scale of the input features is low, more computational resources will be allocated to it.

[0041] In this way, shifting computationally intensive attention from high-cost, large-scale paths to low-cost, small-scale paths not only effectively avoids wasting resources at computational bottlenecks, but also enables more thorough and complex feature interactions at the abstract semantic level (small-scale features), thereby maximizing overall feature learning and representation capabilities while controlling the total computational budget.

[0042] The image feature extraction structure 100 includes a feature fusion layer 12. The feature fusion layer 12 is configured to fuse the output features of multiple paths 11.

[0043] In one embodiment, the feature fusion layer 12 uses a concate (dimension concatenation) operation to fuse multi-path output features. Assume the image feature extraction structure 100 includes parallel processing of upper-layer paths, middle-layer paths, and lower-layer paths. If the dimensions of the upper-layer path output Out1, the middle-layer path output Out2, and the lower-layer path output Out3 are all [B, L, D], where B is the batch size, L is the sequence length, and D is the feature dimension, then concatenation along the feature dimension D results in a fused feature dimension of [B, L, 3D].

[0044] In other embodiments, feature fusion can also be performed by element-wise summation or averaging (that is, directly adding or averaging multiple feature maps at corresponding spatial and channel positions), without any specific limitations.

[0045] like Figure 2 As shown, in one embodiment, each of the multiple paths 11 includes: a query generation module configured to generate a query vector based on the input features of the path; a key generation module configured to generate a key vector based on the input features of the path; a value generation module configured to generate a value vector based on the input features of the path; and an attention calculation module whose input is connected to the output of the query generation module, the key generation module, and the value generation module, respectively, to receive the corresponding query vector, key vector, and value vector and perform attention calculation.

[0046] Thus, by constructing the "query-key-value" on each path, a "scale-aware" feature interaction mechanism is realized, which enables the network to adaptively adjust its "attention granularity" and "interaction range" according to the input feature scale of different paths, thereby fundamentally avoiding the conflicts and waste caused by using a single and rigid interaction mode to process multi-scale information.

[0047] In this embodiment, each path takes its specific scale input feature as an independent starting point to generate a dedicated query (Q), key (K) and value (V) vector group. In the high-resolution path, Q, K and V carry rich spatial detail information, and their interaction mainly focuses on the fine correlation between pixels or feature points in the local area. In the low-resolution path, Q, K and V carry highly compressed semantic information, and their interaction is committed to mining the abstract logical relationship between different semantic concepts or object components.

[0048] On the one hand, this means that two types of tasks with different properties, local fine correlation and global semantic correlation, are efficiently completed in their respective dedicated "workspaces" without interference. On the other hand, it also means that the computational complexity of each path matches the reasonable range of the task it performs, avoiding waste of computing resources.

[0049] In a specific embodiment, in the path with the largest scale, the attention calculation is limited to be performed within a spatial local window.

[0050] In this way, by limiting the attention calculation of large scale to be performed within a spatial window, global calculation is avoided, and the computational complexity is reduced from global to window level, realizing the unification of high performance and high efficiency.

[0051] In a specific embodiment, the spatial local window is configured as 7x7, and the attention can be set as self-attention.

[0052] Continuing to refer to Figure 2 In a specific embodiment, based on the Transformer model, the image feature extraction structure 100 is determined. Specifically, the structure of the bottom layer in the model adopts three parallel calculation paths, forming an MPT (Multi-Path-Transformer) architecture.

[0053] In the MPT structure, it is assumed that there are three parallel paths of upper, middle and lower, and the feature map scales of the three paths decrease in turn. They all start from the input feature X and finally converge into the output feature Y. The input feature X enters the three paths after different processing, each path receives the input feature after different down-sampling processing, and independently performs self-attention calculation, and finally the output features of the three paths are fused through the Concate operation. For example, two matrices with sizes [16, 512] and [16, 512] are spliced into a [16, 1024] matrix.

[0054] Specifically, in the upper path, first, the input feature X1 is linearly transformed to generate a query vector (Q), a key vector (K) and a value vector (V), i.e. Q, K, V = Linear (X1); second, the attention weight Atten is obtained by calculating the product of the query vector (Q) and the key vector (K) and using the softmax function for layer normalization, i.e. Atten = softmax (QxK); then, the attention weight Atten is multiplied with the value vector (V) to update Atten, i.e. Atten = Atten x V; finally, the multiplication result is linearly transformed to obtain the output Out of the upper path, i.e. Out = Linear (Atten).

[0055] In the middle path and the lower path, the transformation process is the same as that of the upper path, except that the input feature of the middle path is X2 and the input feature of the lower path is X3. The specific process can be referred to the description of the upper path above, which will not be repeated here.

[0056] In a specific embodiment, the feature map scale of the input feature X2 of the middle path is equal to half of the input feature X1 of the upper path, i.e. X2 = 0.5 * X1; the feature map scale of the input feature X3 of the lower path is equal to half of the input feature X2 of the middle path, i.e. X3 = 0.5 * X2.

[0057] For example, if the dimension of X1 is [B, H, W, C], the feature map scale of the input feature X2 of the middle path is [B, H / 2, W / 2, C]; the feature map scale of the input feature X3 of the lower path is [B, H / 4, W / 4, C].

[0058] In a specific embodiment, the proportion of the number of attention calculation modules (i.e. attention heads) in the upper path, the middle path and the lower path to the total number of attention heads of the MPT module is 0.1, 0.3 and 0.6 respectively. That is, the larger the scale of the input feature, the fewer the number of attention heads, and the smaller the scale of the input feature, the more the number of attention heads.

[0059] For example, assuming the total number of attention heads in the MTP module is N, the largest upper-layer path X1 is assigned 0.1N attention heads, the medium-scale middle-layer path X2 is assigned 0.3N attention heads, and the smallest-scale lower-layer path X3 is assigned 0.6N attention heads.

[0060] This architecture fundamentally optimizes the connection and computation patterns of the standard Transformer by distributing computationally expensive attention operations across feature maps of different resolutions and dynamically allocating computational resources based on the feature map size, thereby achieving higher operating efficiency.

[0061] like Figure 3 As shown, in one embodiment, the image feature extraction structure 100 is an attention enhancement module (Atten-Block). The attention enhancement module Atten-Block includes a first-layer normalization module, a feedforward network module, and a second-layer normalization module. The output of the first-layer normalization module is connected to the input of multiple paths, and the second-layer normalization module is connected between the feature fusion layer and the feedforward network module.

[0062] Furthermore, the output features of the feature fusion layer are added to the input of the first layer normalization module through the first residual connection; the output features of the feedforward network module are added to the input of the second layer normalization module through the second residual connection.

[0063] Thus, by introducing residual connections and layer normalization layers, the powerful modeling capabilities of multi-scale attention are preserved, while fundamentally ensuring the training feasibility and convergence stability of this module when it is stacked in deep networks.

[0064] In this embodiment, the first-layer normalization module and the first residual connection form a "pre-normalized residual" structure, ensuring that the input features are stabilized before entering the complex parallel path and allowing the gradient to bypass attention calculations and be directly backpropagated, thus mitigating gradient vanishing. The second-layer normalization module and the second residual connection form the same stable structure around the feedforward network module. This dual-residual design ensures that there is always a shortcut of identity mapping between the module's input and output, allowing the network to focus on learning the residual (i.e., the changing part), greatly reducing the optimization difficulty and enabling the deep network containing this module to be successfully trained.

[0065] In one specific embodiment, the feedforward network module is an MLP (Multilayer Perceptron). The MLP includes a first linear transformation layer, a nonlinear activation layer, and a second linear transformation layer connected in sequence.

[0066] In another specific embodiment, the feedforward network module further includes a depthwise convolutional layer connected between the first linear transformation layer and the nonlinear activation layer.

[0067] In this way, by introducing the depth-wise convolution layer, the feedforward network module is facilitated to fuse the information of the local field around each position when processing the feature of each position, thereby enhancing the spatial coherence of the feature.

[0068] In another specific embodiment, the feedforward network module comprises, in sequence, a first fully connected layer (Linear), a depth-wise convolution layer (DWconv), an activation function layer (ReLU), a first dropout layer (DropOut), a second fully connected layer (Linear), and a second dropout layer (DropOut).

[0069] In this way, by introducing the dropout layer, the model can be effectively prevented from overfitting on the training data without excessively increasing the inference calculation burden, thereby prompting the model to learn features with better generalization ability.

[0070] In this embodiment, the first fully connected layer is configured to perform dimensionality increasing conversion on the input feature, the depth-wise convolution layer is configured to fuse the local spatial context information of the feature, the nonlinear activation function is used to introduce nonlinear transformation capability, the first dropout layer is used to regularize the feature at the training stage to prevent overfitting, the second fully connected layer is configured to restore the feature to the original dimension or target dimension, and the second dropout layer is configured to regularize the final output feature.

[0071] This structure, through the cascading processing procedure of "fully connected dimensionality increasing-local convolution enhancement-nonlinear activation-random dropout-fully connected dimensionality reduction-random dropout", ensures the generalization performance and training stability of the module while introducing strong nonlinear expression capability and spatial perception capability.

[0072] In one specific embodiment, as shown in FIG. 2, the attention enhancement module Atten-Block has the following specific structure: input feature→first layer normalization module (Nom1)→image feature extraction structure (MPT)→first residual connection (added to the original input feature)→second layer normalization module (Nom2)→feedforward network module (MLP)→second residual connection (added to the input feature of Norm2)→output feature. Figure 3

[0073] ​For ease of understanding, for example, assuming that the input feature is X, the first fully connected layer is processed to obtain X = Linear (X); then the processed output feature is processed by the depthwise convolution layer to obtain X = DWconv (X); then the processed output feature is processed by the activation function layer to obtain X = ReLU (X); then the processed output feature is processed by the first dropout layer to obtain X = DropOut (X); then the processed output feature is processed by the second fully connected layer to obtain X = Linear (X); then the processed output feature is processed by the second dropout layer to obtain the output feature Out = DropOut (X) of the feedforward network.

[0074] As shown in Figure 4 , an embodiment of the present application provides an image feature extraction network 300.

[0075] The image feature extraction network 300 is the final product form obtained by embedding the image feature extraction structure 100 described above into a larger and more complete processing flow.

[0076] The image feature extraction network 300 includes an image blocking module for converting an input image into an image block sequence.

[0077] The image feature extraction network 300 includes at least one downsampling and processing unit, the input end of which is connected to the output end of the image blocking module, for performing feature processing on the input image block sequence.

[0078] In an embodiment, the downsampling and processing unit includes a plurality of alternately stacked feature transformation modules (MLP-Block) and downsampling modules (Down), i.e., the partial structure of the backbone network shown in Figure 4 .

[0079] As shown in Figure 5 , in a specific embodiment, the feature transformation module (MLP-Block) includes a third layer normalization module (i.e., Norm) and a feedforward network module (i.e., MLP), the input end of the third layer normalization module receives an input feature, and the output end thereof is added to the input end of the third layer normalization model through a third residual connection.

[0080] The image feature extraction network 300 includes at least one image feature extraction structure 100, the input end of which is connected to the input end of the last downsampling and processing module.

[0081] The image feature extraction network 300 includes an output module, the input end of which is connected to the output end of the image feature extraction structure 100.

[0082] Continuing to refer to Figure 4 , the working process of the image feature extraction network 300 is described.

[0083] The network begins with an image segmentation module, which cuts and flattens the input 2D image grid into a sequence of image blocks. This sequence of image blocks is then passed through two consecutive feature transformation modules, MLP-Block (MLP-Block x2), to complete the basic feature transformation. Next, a downsampling layer (Down) halves the length and width of the feature map to expand the receptive field. It then passes through two more consecutive feature transformation modules, MLP-Block (MLP-Block x2), to further optimize the basic features. A second downsampling layer (Down) is then passed, and the downsampling result is input into six consecutive attention enhancement modules (Atten-Block x6) to extract high-level semantic features. A third downsampling layer (Down) is then passed, and the downsampling result is input into four consecutive attention enhancement modules (Atten-Block x4) to strengthen the global correlation of semantic features. Finally, the output features are fed into the classification head module.

[0084] The classification head module first stabilizes the feature distribution using a layer normalization layer (Norm), and then uses an adaptive average pooling layer (AvgPool) to transform the feature map of arbitrary spatial size for each channel into a vector of the target dimension. Finally, a fully connected layer (FC) performs a linear transformation on this vector, outputting the prediction score corresponding to each target category (such as the survival time interval), thus completing the end-to-end mapping from the original image to high-level prediction.

[0085] Compared to the Attention Enhancement Module (Atten-Block), the Feature Transformation Module (MLP-Block) does not contain an attention structure. Its corresponding number of parameters and computational cost are much lower than those of the Attention Enhancement Module. The combination of the two achieves a fine division of labor and dynamic balance between network capacity and computational efficiency, thereby ensuring the powerful representational ability of the model while concentrating limited computational resources on the most critical semantic interaction links.

[0086] like Figure 6 As shown, one embodiment of the present invention provides a multimodal prediction model 400.

[0087] A multimodal prediction model 400 refers to a machine learning model that can simultaneously receive and process two or more different types of data and fuse these data to make a final prediction. In this invention, the first input modality is an image, such as a patient's pathological image, and the second input modality is text information, such as a patient's pathological text description, such as the patient's location and chief complaint. By fusing image and text information, the category to which the patient's postoperative survival time belongs is predicted.

[0088] The multi-modal prediction model 400 comprises the image feature extraction network 300, the text feature extraction network and the feature fusion module as described above. The input end of the feature fusion module is connected with the output ends of the image feature extraction network and the text feature extraction network respectively.

[0089] The image feature extraction network is used to mine visual semantics from the image, the text feature extraction network is used to extract clinical semantics from the textual description, and the feature fusion module is used to align, interact and integrate the two kinds of semantics in a common space. The second classification module determines the final survival time interval classification according to the fused features.

[0090] In an embodiment, the text feature extraction network obtains multi-source text information before extracting text features, for example, receives multiple pieces of text from different sources (such as chief complaints, medical histories, pathological report fragments, etc.); extracts key information from each text piece, and performs word embedding operation on the extracted text to map each word unit or sub-word unit in the text sequence to a fixed-dimensional dense vector; the embedding vectors corresponding to all text pieces are spliced to form a long vector sequence. The long vector sequence is used as the input feature of the text feature extraction network.

[0091] In a specific embodiment, the text feature extraction network uses a text encoder in the CLIP (Contrastive Language-Image Pre-Training) model as its backbone network.

[0092] In a specific embodiment, the feature fusion module adopts a block structure based on Transformer, which is used to transform the fused features to a high-dimensional space and perform feature transformation in the high-dimensional space to extract high-dimensional semantic information of the fused features.

[0093] In a specific embodiment, the classification module comprises a LayerNorm layer, an adaptive average pooling layer and a fully connected layer connected in sequence. The LayerNorm layer is used to normalize the mean and variance of each sample of the input feature to stabilize the training process and accelerate convergence; the adaptive average pooling layer is used to compress a two-dimensional feature map of arbitrary spatial size to a fixed 1 1 spatial size through adaptive pooling operation, thereby aggregating the information of the spatial dimension to the channel dimension to generate a one-dimensional feature vector; the fully connected layer is used to linearly transform the one-dimensional vector after pooling, and the number of output neurons is set to the number of target categories (for example, 6 categories corresponding to different survival time intervals), and the subsequent Softmax function is used to convert the prediction probability of each category.

[0094] As Figure 7As shown, the training method of the image feature extraction network in an embodiment of the present application comprises the following steps.

[0095] Step S1, obtaining a training image dataset containing labeled categories; Step S2, inputting the training image dataset into the image feature extraction network for training to obtain predicted categories; Step S3, determining a loss function value based on the predicted categories and the corresponding labeled categories; Step S4, updating the parameters of the image feature extraction network based on the loss function value until the function loss value converges.

[0096] In this embodiment, the training method of the image feature extraction network comprises two stages of pre-training and fine-tuning. In the pre-training stage: first, a large-scale general image classification dataset (such as ImageNet, containing 1000 categories) is used to pre-train the image feature extraction network 300, the output dimension of the fully connected layer in the terminal classification module is configured to be consistent with the number of categories of the general dataset (for example, 1000 dimensions), and the network parameters are optimized to enable it to correctly classify each natural image in ImageNet.

[0097] In the pre-training process, the cross-entropy loss function is used as the optimization objective, and the corresponding calculation formula is shown in formula (1):

[0098] Where x represents the network prediction value, and class represents the category label of the input image. In the pre-training process, AdamW is used, wherein the numerical stability term parameter eps of the optimizer is set to 1e-8, the decay rate parameter betas (set to (0.9, 0.999)) of the gradient matrix estimation is set to 0.9, the initial learning rate is set to 5e-4. The training batch size (batch) is set to 1024, and a total of 300 epochs are trained. After training, the weight of the image feature extractor backbone network is obtained.

[0099] In the fine-tuning stage, the pre-trained weight is loaded into the image feature extraction network of the multi-modal prediction model, the parameters of the bottom layer (such as the first 2 MLP-Block) of the backbone network are frozen, and only the parameters of the upper Atten-Block, feature fusion module and classification head are trained; the pathological prognosis prediction dataset (containing image images, text features and survival time labels) is used for fine-tuning, and the final model is obtained after training until the loss function value converges.

[0100] In summary, the present application provides an image feature extraction structure, network, training method and multi-modal prediction system. The image feature extraction structure decouples and reversely correlates the "intensity of investment of computing resources" and the "computing cost of feature processing" by establishing parallel multi-scale processing paths, that is, for large-scale features that require fine local information, only a small number of attention computing modules are allocated for necessary processing; and for small-scale features that carry high-level semantic information, a large number of attention computing modules are invested for deep and global modeling. This "adjust measures to local conditions" resource scheduling strategy makes the expensive global attention calculation mainly occur in the path with the smallest calculation base, thereby realizing the optimization of computing efficiency at the algorithm level. Finally, the feature fusion layer organically integrates the local details and global semantics extracted by different paths, forming a feature representation that is both efficient and strong in expression.

[0101] It should be understood that although the present specification is described in terms of embodiments, not every embodiment contains only one independent technical solution, and the description of the specification is only for the sake of clarity, and those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that those skilled in the art can understand.

[0102] The series of detailed descriptions listed above are only specific descriptions of the feasible embodiments of the present application, and are not intended to limit the protection scope of the present application. Any equivalent embodiments or changes made without departing from the spirit of the present application shall be included in the protection scope of the present application.

Claims

1. An image feature extraction structure, characterized by, The multiple paths include parallel processing and a feature fusion layer: The multiple paths are configured to perform attention calculation on input features at different feature map scales respectively, wherein the input features at different feature map scales used by each path are determined by downsampling the original input features with different pooling steps; The feature fusion layer is configured to fuse output features of the multiple paths; In the multiple paths, the scale of the input features on which attention calculation is performed is negatively correlated with the number of parallel attention calculation modules configured for the path.

2. The image feature extraction structure according to claim 1, characterized in that, The number of the multiple paths is three, and the feature map scales of the three paths decrease in turn.

3. The image feature extraction structure according to claim 2, characterized in that, In the path with the largest scale, attention calculation is limited to be performed within a spatial local window.

4. The image feature extraction structure according to claim 1, characterized in that, The image feature extraction structure is an attention enhancement module, and the attention enhancement module further includes: A first normalization module, an input end of which accepts input features, and an output end of which is connected to an input end of the multiple paths; A feedforward network module; A second normalization module, which is connected between the feature fusion layer and the feedforward network module; An output feature of the feature fusion layer is added to an input end of the first normalization module through a first residual connection; and an output feature of the feedforward network module is added to an input end of the second normalization module through a second residual connection.

5. The image feature extraction structure according to claim 4, characterized in that, The feedforward network module includes a first linear transformation layer, a nonlinear activation layer and a second linear transformation layer connected in turn.

6. The image feature extraction structure according to claim 5, characterized in that, The feedforward network module further includes a depth-wise convolution layer connected between the first linear transformation layer and the nonlinear activation layer.

7. The image feature extraction structure according to claim 6, characterized in that, The feedforward network module includes a first fully connected layer, a depth-wise convolution layer, an activation function layer, a first dropout layer, a second fully connected layer and a second dropout layer connected in turn.

8. The image feature extraction structure according to claim 1, characterized in that, In the multiple paths, each path includes: A query generation module configured to generate a query vector based on input features of the path; A key generation module configured to generate a key vector based on the input features of the path; A value generation module configured to generate a value vector based on the input features of the path; An attention calculation module, an input end of which is connected to output ends of the query generation module, the key generation module and the value generation module respectively, to accept the query vector, the key vector and the value vector and perform the attention calculation.

9. An image feature extraction network characterized by, The image feature extraction structure includes: An image block module for converting an input image into a sequence of image blocks; At least one downsampling and processing module, an input end of which is connected to an output end of the image block module, for performing feature processing on the input sequence of image blocks; At least one image feature extraction structure according to any one of claims 1-8, an input end of which is connected to an output end of the last downsampling and processing module; A first classification module, an input end of which is connected to an output end of the image feature extraction structure.

10. The image feature extraction network of claim 9, wherein, The downsampling and processing module includes a downsampling module and a feature transformation module connected in turn, and the feature transformation module includes: A third normalization module, an input end of which accepts input features; A feedforward network module, an input end of which is connected to an output end of the third normalization module; The output feature of the feedforward network module is added to the input end of the third layer normalization module through a third residual connection.

11. A multi-modal prediction model, characterized in that, The method comprises: The image feature extraction network according to any one of claims 9-10; The text feature extraction network; The feature fusion module is connected with the output ends of the image feature extraction network and the text feature extraction network respectively; The second classification module is connected with the output end of the feature fusion module.

12. A method of training an image feature extraction network as claimed in any one of claims 9-10, characterized in that, The method comprises: An image data set containing labeled categories is obtained; The image data set is input into the image feature extraction network for training to obtain a predicted category; Based on the predicted category and the corresponding labeled category, a loss function value is determined; Based on the loss function value, the parameters of the image feature extraction network are updated until the function loss value converges, and a trained image feature extraction network is obtained.

Images