Radiology report generation method based on hierarchical interactive fusion

Abstract

The invention discloses a radiology report generation method based on hierarchical interactive fusion, and relates to the technical field of medical image intelligent processing, and the method comprises the steps: collecting original image data, carrying out the intensity normalization, obtaining normalized image data, carrying out the multi-level visual feature extraction and feature dimension unification of the normalized image data, and carrying out the multi-level visual feature extraction and feature dimension unification of the normalized image data; forming a hierarchical visual feature set; coding processing is carried out on the hierarchical visual feature set, cross-layer attention relations between a shallow layer and a deep layer and between a middle layer and the deep layer are constructed in the coding process, a shallow layer two-dimensional biased field and a middle layer two-dimensional biased field are formed, and a migration smoothness index and a consistency index are obtained after nonlinear resampling; and in the decoding stage, a multi-path cross attention structure is constructed based on the coded hierarchical visual feature set, an abnormal priori graph is constructed according to the shallow two-dimensional biased field and the middle two-dimensional biased field, attention bias is formed, and a cross-hierarchical cross attention aggregation vector is generated. According to the invention, association expression of multi-level visual features is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical image intelligent processing, in particular to a radiology report generation method based on hierarchical interaction fusion. BACKGROUND

[0002] Radiology report is the core carrier for imaging diagnosis department to record examination conclusion, diagnosis thought and follow-up suggestion. With the wide application of X-ray, computer tomography and other medical image examinations in clinic, the quantity and complexity of image data continue to grow, which promotes the development of automatic report generation technology. Radiology report generation method based on deep learning has gradually become a research hotspot. Related work usually jointly models large-scale image and corresponding report samples, learns the implicit mapping relationship between image content and text expression through neural network, and is used to assist the realization of structured expression of report content and consistency control of language style.

[0003] In the existing radiology report generation scheme, image feature coding mainly focuses on modeling the overall semantic representation, and the multi-level interaction relationship between shallow detail information, organ structure hierarchical information and high-level lesion semantics is relatively rough, which is difficult to form more detailed hierarchical relationship in spatial correspondence and deformation association. At the same time, the mapping of visual features to text sequences often relies on unified attention or fixed fusion mechanism, and less combines structured priors such as abnormal region distribution and cross-layer consistency to dynamically adjust attention distribution and fusion strength. In the scene of complex lesion morphology or long diagnosis description, the focusing and organization ability of key diagnostic clues still has further space for improvement. SUMMARY

[0004] In view of the above existing problems, the present application is proposed.

[0005] Therefore, the present application provides a radiology report generation method based on hierarchical interaction fusion, which solves the problems of insufficient multi-level visual feature correlation modeling and lack of dynamic attention adjustment mechanism combined with structured priors.

[0006] To solve the above technical problems, the present application provides the following technical scheme: The present application provides a radiology report generation method based on hierarchical interaction fusion, which comprises: collecting original image data and performing intensity normalization to obtain normalized image data; performing multi-level visual feature extraction and feature dimension unification on the normalized image data to form a hierarchical visual feature set; The hierarchical visual feature set is encoded, and the cross-layer attention relationship between the shallow layer and the deep layer and the cross-layer attention relationship between the middle layer and the deep layer are constructed in the encoding process to form a shallow two-dimensional offset field and a middle two-dimensional offset field, and the offset smoothness index and the consistency index are obtained after nonlinear resampling; In the decoding stage, a multi-path cross attention structure is constructed based on the processed hierarchical visual feature set, an abnormal prior map is constructed according to the shallow two-dimensional offset field and the middle two-dimensional offset field, and an attention bias is formed to generate a cross-level cross attention aggregation vector; The cross-level cross attention aggregation vector is processed by feedforward to generate a fusion vector, the fusion vector is subjected to semantic enhancement processing to form an initial enhanced vector, the initial enhanced vector is converged into a final decoder hidden state vector through iterative enhancement, and a radiology report text is generated based on the final decoder hidden state vector.

[0007] As a preferred scheme of the radiology report generation method based on hierarchical interaction fusion, the multi-level visual feature extraction on the normalized image data comprises adopting a pre-trained ResNet101 model as a visual feature extractor to extract shallow features, middle features and deep features on the normalized image data at different visual depths.

[0008] As a preferred scheme of the radiology report generation method based on hierarchical interaction fusion, the feature dimension unification and the formation of the hierarchical visual feature set comprise adopting a single-layer linear function to perform linear mapping of the shallow features, the middle features and the deep features at the spatial grid positions to generate shallow visual features, middle visual features and deep visual features, and combining the shallow visual features, the middle visual features and the deep visual features to form the hierarchical visual feature set.

[0009] As a preferred scheme of the radiology report generation method based on hierarchical interaction fusion, the formation of the shallow two-dimensional offset field and the middle two-dimensional offset field comprises encoding the shallow visual features, the middle visual features and the deep visual features in the hierarchical visual feature set by a Transformer encoder to obtain shallow visual encoding features, middle visual encoding features and deep visual encoding features; According to the cross-level attention relationship between the shallow visual encoding features, the middle visual encoding features and the deep visual encoding features, a set of deep visual encoding feature grid positions is selected at each grid position, and the geometric center of the shallow related deep spatial coordinates and the geometric center of the middle related deep spatial coordinates are respectively calculated according to the spatial coordinates of the set of deep visual encoding feature grid positions; The difference between the geometric center of the shallow related deep spatial coordinates and the current shallow grid position coordinate is taken as the shallow two-dimensional offset vector, the difference between the geometric center of the middle related deep spatial coordinates and the current middle grid position coordinate is taken as the middle two-dimensional offset vector, and the shallow two-dimensional offset field and the middle two-dimensional offset field are respectively arranged.

[0010] As a preferred scheme of the radiology report generation method based on hierarchical interactive fusion provided in the application, wherein: the obtaining of the offset smoothness index and the consistency index after the non-linear resampling comprises: performing non-linear resampling on the shallow two-dimensional offset field and the middle two-dimensional offset field to obtain the shallow resampled deep visual coding feature and the middle resampled deep visual coding feature; The offset smoothness index is obtained by performing summation processing on the difference vectors of the shallow two-dimensional offset field and the middle two-dimensional offset field in the spatial grid coordinate direction respectively. The consistency index is obtained by performing square difference processing on the difference values of the shallow resampled deep visual coding feature and the shallow visual coding feature in the spatial grid position and performing square difference processing on the difference values of the middle resampled deep visual coding feature and the middle visual coding feature in the spatial grid position.

[0011] As a preferred scheme of the radiology report generation method based on hierarchical interactive fusion provided in the application, wherein: the constructing of the multi-path cross attention structure based on the hierarchical visual feature set after the encoding processing in the decoding stage comprises: representing the hidden state of the Transformer decoder as a decoder hidden state vector, and generating a decoder query vector by performing a linear transformation on the decoder hidden state vector; The decoder query vector is respectively associated with the shallow visual coding feature, the middle visual coding feature and the deep visual coding feature to form the shallow cross attention structure, the middle cross attention structure and the deep cross attention structure. In the shallow cross attention structure, the middle cross attention structure and the deep cross attention structure, linear transformation is performed on the shallow visual coding feature, the middle visual coding feature and the deep visual coding feature to generate the shallow key vector, the shallow value vector, the middle key vector, the middle value vector, the deep key vector and the deep value vector.

[0012] As a preferred scheme of the radiology report generation method based on hierarchical interactive fusion provided in the application, wherein: the constructing of the abnormal prior map and forming of the attention bias according to the shallow two-dimensional offset field and the middle two-dimensional offset field comprises: calculating the two-norm of the difference vector of the shallow two-dimensional offset vector and the middle two-dimensional offset vector in the spatial grid coordinate, and performing normalization on the two-norm results of all spatial grid coordinates to obtain the abnormal prior map; The abnormal prior map, the offset smoothness index and the consistency index are combined in numerical value in the spatial grid coordinate to form the attention bias.

[0013] As a preferred scheme of the radiology report generation method based on hierarchical interaction fusion provided in the application, the generation of the cross-level cross-attention aggregation vector comprises: based on the shallow cross-attention structure, the middle cross-attention structure and the deep cross-attention structure, calculating the shallow cross-attention weight, the middle cross-attention weight and the deep cross-attention weight according to the decoder query vector, the shallow key vector, the middle key vector, the deep key vector and the attention bias; The shallow cross-attention weight, the middle cross-attention weight and the deep cross-attention weight are respectively multiplied with the shallow value vector, the middle value vector and the deep value vector at all positions of the spatial grid to obtain the shallow cross-attention aggregation vector, the middle cross-attention aggregation vector and the deep cross-attention aggregation vector. The shallow cross-attention aggregation vector, the middle cross-attention aggregation vector and the deep cross-attention aggregation vector are spliced in the channel direction to form the cross-level cross-attention aggregation vector.

[0014] As a preferred scheme of the radiology report generation method based on hierarchical interaction fusion provided in the application, the generation of the cross-level cross-attention aggregation vector comprises: based on the shallow cross-attention structure, the middle cross-attention structure and the deep cross-attention structure, calculating the shallow cross-attention weight, the middle cross-attention weight and the deep cross-attention weight according to the decoder query vector, the shallow key vector, the middle key vector, the deep key vector and the attention bias; The cross-level cross-attention aggregation vector is input into the feedforward network structure of the Transformer decoder to generate the cross-level difference quantity by sequentially performing the first linear transformation, the nonlinear activation processing and the second linear transformation. According to the cross-level difference quantity, the decoder query vector and the consistency index, a three-element combined vector is formed to generate a judgment factor. The fusion vector is selected from the low-complexity fusion vector, the medium-complexity fusion vector and the high-complexity fusion vector according to the judgment factor, and is spliced with the deep global semantic representation vector obtained by the deep visual coding feature and the word embedding vector in the channel dimension to form the initial enhanced vector.

[0015] As a preferred scheme of the radiology report generation method based on hierarchical interaction fusion provided in the application, the generation of the cross-level cross-attention aggregation vector comprises: based on the shallow cross-attention structure, the middle cross-attention structure and the deep cross-attention structure, calculating the shallow cross-attention weight, the middle cross-attention weight and the deep cross-attention weight according to the decoder query vector, the shallow key vector, the middle key vector, the deep key vector and the attention bias; The linear mapping and the exponential normalization operation are performed on the final decoder hidden state vector to generate a word probability distribution, and the word identification with the maximum probability in the word probability distribution is selected and arranged in time sequence to form the radiology report text.

[0016] The present application has the beneficial effects that: by constructing the cross-layer attention relationship in the encoding process, the correlation expression between the shallow visual encoding features, the middle visual encoding features and the deep visual encoding features is realized, so that the hierarchical visual features form a unified representation in spatial correspondence and semantic connection; by constructing the multi-path cross-attention structure and introducing the attention bias, the differentiated focusing of different depth visual encoding features is realized, so that the cross-layer cross-attention aggregation vector can strengthen the semantic expression of the key area. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0018] Figure 1 The flowchart of the radiology report generation method based on hierarchical interaction fusion.

[0019] Figure 2 The flowchart of multi-level visual feature extraction and feature dimension unification.

[0020] Figure 3 The flowchart of encoding processing and cross-layer attention relationship construction.

[0021] Figure 4 The flowchart of multi-path cross-attention structure and report generation.

[0022] Figure 5 The spatial response schematic diagram of multi-level visual features. DETAILED DESCRIPTION

[0023] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application will be described in detail below with reference to the drawings of the specification.

[0024] In the following description, many specific details are set forth in order to provide a thorough understanding of the present application, but the present application can also be implemented in other ways different from those described herein, and those skilled in the art can make similar generalizations without departing from the connotation of the present application, therefore the present application is not limited to the specific embodiments disclosed below.

[0025] Secondly, the "one embodiment" or "embodiment" referred to herein means that the specific features, structures or characteristics can be included in at least one implementation of the present application. "In one embodiment" appearing in different places in the specification does not mean the same embodiment, nor is it an independent or selective embodiment that excludes other embodiments.

[0026] Reference Figures 1-5 This is one embodiment of the present invention, which provides a method for generating radiological reports based on hierarchical interactive fusion, including the following steps: S1. Collect raw image data and normalize the intensity to obtain normalized image data. Extract multi-level visual features and unify feature dimensions on the normalized image data to form a hierarchical visual feature set.

[0027] Furthermore, raw image data generated by medical imaging equipment is acquired. Depending on the imaging method, the raw image data may exist in the form of a two-dimensional pixel matrix, such as X-ray images, or in the form of a three-dimensional voxel matrix, such as computed tomography images.

[0028] Each pixel or voxel in the raw image data contains a raw intensity value, which is used to characterize energy decay, material density or other physical image parameters detected during the imaging process.

[0029] The original intensity values ​​of all pixels or voxels in the original image data are read one by one. By continuously comparing all the original intensity values, the minimum and maximum intensity values ​​in the original image data are determined.

[0030] Based on the linear proportional relationship between the minimum and maximum intensity values, the original intensity value of each pixel or voxel in the original image data is mapped proportionally to the intensity range of zero to one, and normalized image data is generated after all the original intensity values ​​have been mapped.

[0031] Furthermore, a ResNet101 model pre-trained on the ImageNet dataset is used as a visual feature extractor to perform multi-level visual feature extraction on the normalized image data. The visual features include shallow features, mid-level features, and deep features.

[0032] The pre-trained ResNet101 model is a deep convolutional network structure consisting of 101 convolutional layers and residual connections. Through supervised training on large-scale natural images on the ImageNet dataset, it learns multi-level convolutional representations ranging from low-level textures to high-level semantics.

[0033] The pre-trained ResNet101 model consists of multiple consecutively stacked residual convolutional stages, where each residual convolutional stage receives the output of the previous convolutional stage and preserves the identity mapping path between multiple convolutional blocks, enabling the residual convolutional stages to learn deeper representations while maintaining gradient stability.

[0034] After normalization, the image data is expanded or resized according to the input size requirements of the ResNet101 model and then input into the pre-trained ResNet101 model. During the layer-by-layer convolution process on the normalized image data, the pre-trained ResNet101 model forms multiple abstract levels of convolution output in the residual convolution stage at different depths.

[0035] To obtain clearly defined visual features, shallow, mid-level, and deep features were extracted from the last three residual convolutional stages of the pre-trained ResNet101 model. Shallow features originate from the output of the shallow residual convolutional stages of the pre-trained ResNet101 model. These stages contain a small number of convolutional blocks and represent low-level visual information, including image edges, texture details, and local brightness variations in the normalized image data. Mid-level features originate from the output of the mid-level residual convolutional stages of the pre-trained ResNet101 model. The convolutional stage contains a medium-depth stack of convolutions. The mid-level features are medium-level visual information and contain local structural composition information of the normalized image data, including organ contours and region shapes formed by multiple continuous regions. The deep features are derived from the output of the deep residual convolutional stage of the pre-trained ResNet101 model. The deep residual convolutional stage contains a higher number of convolutional stacks. The deep features are high-level visual information and contain semantic-level representations of the normalized image data, including semantic activation responses of lesion areas and combinational relationships between semantic regions.

[0036] Shallow, mid-level, and deep features all exist in the form of grid features. The grid feature form ensures that each spatial location has a corresponding channel vector, which is used to represent the visual content formed at different convolution depths.

[0037] Furthermore, a single-layer linear function is used to unify the feature dimensions of shallow, mid-level, and deep features.

[0038] A single-layer linear function contains a channel mapping matrix and a channel bias vector. The parameters of the channel mapping matrix and the channel bias vector are updated according to the loss function through backpropagation during network training, enabling the linear transformation to apply additional weights to subtle information in low-response channels while maintaining a uniform number of channels.

[0039] A single-layer linear function performs a linear transformation on the input channel vector at each spatial grid position, so that the shallow, middle, and deep features are transformed into shallow visual features, middle visual features, and deep visual features, respectively, after processing by the single-layer linear function, as follows: ; in, It is shallow. For middle level, For deeper layers, These are shallow visual features resulting from the application of a single-layer linear function. These are the mid-level visual features resulting from the application of a single-layer linear function. These are the deep visual features resulting from the application of a single-layer linear function. It is a single-layer linear function, which can be implemented using a fully connected layer structure or a one-to-one convolution structure. For normalized image data, These are shallow features obtained from the shallow residual convolution stage of a pre-trained ResNet101 model. These are mid-level features obtained from the residual convolution stage of a pre-trained ResNet101 model. These are deep features obtained from the deep residual convolution stage of the pre-trained ResNet101 model.

[0040] Shallow visual features, mid-level visual features, and deep visual features represent visual content using a unified channel dimension and together constitute a hierarchical visual feature set.

[0041] It should be noted that, in order to visually demonstrate the differences between multiple levels of visual features, you can refer to... Figure 5 As shown, the visual responses formed at different feature layer depths differ in spatial distribution.

[0042] Figure 5 Layer 1, Layer 2, and Layer 3 correspond to the visualization responses of shallow, mid-level, and deep visual features, respectively, and are used to show the spatial distribution differences of visual features at different levels. Figure 5 The terms Right, Lung, pleural, breast, seen, and volumes are region cues generated by the visualization tool, representing the right side region, lung region, pleural region, breast region, visible region, and volume region, respectively. They are used to indicate the approximate location of the visual response in the image and are for illustrative purposes only.

[0043] S2. Encode the hierarchical visual feature set. During the encoding process, construct cross-layer attention relationships between shallow and deep layers and between middle and deep layers to form shallow two-dimensional offset fields and middle two-dimensional offset fields. After nonlinear resampling, obtain the offset smoothness index and consistency index.

[0044] Furthermore, the shallow visual features, mid-level visual features, and deep visual features from the hierarchical visual feature set are respectively input into a Transformer encoder containing six coding blocks. Each coding block in the Transformer encoder contains a multi-head attention structure and a feedforward network structure.

[0045] The shallow, medium, and deep visual features are expanded into a spatial grid sequence before being input into the Transformer encoder. Each grid position in the spatial grid corresponds to a channel vector.

[0046] In each encoding block, the Transformer encoder performs a linear mapping three times on each channel vector in the spatial grid sequence. The results of the three linear mappings are used as the encoder query vector, encoder key vector, and encoder value vector, respectively. The encoder query vector and encoder key vector are multiplied by a dot product in the channel dimension. The output of the dot product is normalized to form the multi-head attention weights. The multi-head attention weights and encoder value vectors are weighted and combined in the channel dimension to form the attention aggregation result from each spatial grid position. The attention aggregation result is processed by two layers of linear transformation and nonlinear activation function in the feedforward network to replace the channel vector at the corresponding spatial grid position. The spatial grid sequence of shallow visual features, mid-level visual features, and deep visual features is updated sequentially in the six encoding blocks.

[0047] After processing by the Transformer encoder, the shallow visual features, mid-level visual features, and deep visual features are respectively formed into shallow visual coding features, mid-level visual coding features, and deep visual coding features, represented as: ; in, These are shallow visual encoding features. These are mid-level visual encoding features. For deep visual encoding features, This is a Transformer encoder containing six coding blocks.

[0048] Furthermore, there are spatial differences between shallow and deep visual coding features, and between intermediate and deep visual coding features.

[0049] To achieve cross-level spatial deformation consistency during the encoding stage, a cross-level attention relationship is constructed between shallow and deep visual coding features, and between mid-level and deep visual coding features.

[0050] The cross-layer attention relationship is constructed as follows: At each spatial grid position of a shallow visual encoding feature, the channel vector of the shallow visual encoding feature at the current spatial grid position is represented as a shallow spatial vector, and the channel vectors of the deep visual encoding feature at all spatial grid positions are represented as a set of deep spatial vectors. The similarity metric between the shallow spatial vector and each deep spatial vector in the set of deep spatial vectors is calculated, and the similarity metric is normalized to form a shallow cross-layer attention weight distribution. At each spatial grid position of a mid-layer visual encoding feature, the channel vector of the mid-layer visual encoding feature at the current spatial grid position is represented as a mid-layer spatial vector, and the channel vectors of the deep visual encoding feature at all spatial grid positions are represented as a set of deep spatial vectors. The similarity metric between the mid-layer spatial vector and each deep spatial vector in the set of deep spatial vectors is calculated, and the similarity metric is normalized to form a mid-layer cross-layer attention weight distribution.

[0051] For each shallow visual encoding feature grid location, a set of deep visual encoding feature grid locations is selected based on the shallow cross-layer attention weight distribution.

[0052] The set of deep visual encoding feature grid locations is formed by sorting the shallow cross-layer attention weights from largest to smallest and selecting the top-ranked deep spatial grid locations. The set of deep visual encoding feature grid locations contains a set of deep spatial coordinates. By averaging the abscissa and ordinate of the deep spatial coordinates, the geometric center of the shallow-related deep spatial coordinates is formed.

[0053] The difference between the geometric center of the shallow layer's related deep spatial coordinates and the current shallow layer grid position coordinates is used as the shallow layer's two-dimensional offset vector.

[0054] At each mid-layer visual coding feature grid location, the deep-layer visual coding feature grid location set is selected in the same way according to the mid-layer cross-layer attention weight distribution, and the geometric center of the spatial coordinates of the deep-layer visual coding feature grid location set is calculated to form the geometric center of the mid-layer related deep-layer spatial coordinates; the difference between the geometric center of the mid-layer related deep-layer spatial coordinates and the current mid-layer grid location coordinates is used as the mid-layer two-dimensional offset vector.

[0055] All shallow two-dimensional offset vectors are arranged according to the position of the shallow visual coding feature grid to form a shallow two-dimensional offset field, and all middle two-dimensional offset vectors are arranged according to the position of the middle visual coding feature grid to form a middle two-dimensional offset field.

[0056] Furthermore, the shallow 2D offset field and the middle 2D offset field are input into the nonlinear resampling structure. At each spatial grid position of the shallow 2D offset field, the nonlinear resampling structure determines a floating-point spatial sampling coordinate in the deep visual coding features based on the shallow 2D offset vector. Bilinear interpolation is then performed at the four adjacent deep visual coding feature grid positions of the floating-point spatial sampling coordinate to obtain the shallow resampled deep visual coding features. At each spatial grid position of the middle 2D offset field, a floating-point spatial sampling coordinate is determined in the deep visual coding features based on the middle 2D offset vector. Bilinear interpolation is then performed at the four adjacent deep visual coding feature grid positions of the floating-point spatial sampling coordinate to obtain the middle resampled deep visual coding features.

[0057] In shallow resampling, deep visual coding features maintain a correspondence with shallow visual coding features in terms of spatial grid positions; in mid-level resampling, deep visual coding features maintain a correspondence with mid-level visual coding features in terms of spatial grid positions. This nonlinear resampling process is represented as: ; in, This represents shallow resampling of deep visual encoding features. This represents the deep visual encoding features resampled at the mid-level. This is a nonlinear resampling structure based on bilinear interpolation. It is a shallow two-dimensional migration field. This is a two-dimensional offset field in the middle layer.

[0058] By summing the difference vectors of the shallow and middle-layer two-dimensional migration fields along the spatial grid coordinates, respectively, the migration smoothness index is obtained, expressed as: ; in, For offset smoothness index, For shallow two-dimensional offset fields in spatial grid coordinates Two-dimensional offset vector, For the mid-level two-dimensional offset field in spatial grid coordinates Two-dimensional offset vector, For the vertical coordinate index of the spatial grid, For the horizontal coordinate index of the spatial grid, Let L be the L2 norm of a two-dimensional vector. This refers to the number of grid cells in the vertical direction of the spatial grid. This refers to the number of grid cells in the horizontal direction of the spatial grid. The number of spatial grid locations participating in the summation. This is the spatial scale normalization coefficient, with the same unit as the two-dimensional offset vector. It is used to normalize the length of the offset vector to a dimensionless value. For example, the value can be the maximum value of the L2 norm of the two-dimensional offset vector in the shallow two-dimensional offset field and the middle two-dimensional offset field.

[0059] By performing squared difference processing on the differences between shallow resampled deep visual coding features and shallow visual coding features at spatial grid positions, and on the differences between mid-layer resampled deep visual coding features and mid-layer visual coding features at spatial grid positions, a consistency index is obtained, expressed as: ; in, As a consistency indicator, To resample deep visual encoded features in spatial grid coordinates The channel vector, For mid-level resampling of deep visual encoding features in spatial grid coordinates The channel vector, Encoding shallow visual features in spatial grid coordinates The channel vector, Encoding features of mid-level visual layers in spatial grid coordinates The channel vector.

[0060] S3. In the decoding stage, a multi-path cross-attention structure is constructed based on the hierarchical visual feature set after encoding. Anomaly prior maps are constructed based on the shallow two-dimensional offset field and the middle two-dimensional offset field, and attention bias is formed to generate cross-level cross-attention aggregation vectors.

[0061] Furthermore, during the decoding phase, the Transformer decoder is time-stepped... The hidden state is represented as the decoder hidden state vector.

[0062] By performing a linear transformation on the decoder hidden state vector, a decoder query vector is generated for cross-level attention computation.

[0063] To enable the decoder query vector to establish correspondences with shallow visual coding features, mid-level visual coding features, and deep visual coding features respectively, the cross-attention structure in the Transformer decoder, which originally contained only one path, is expanded into three parallel paths: shallow cross-attention structure, mid-level cross-attention structure, and deep cross-attention structure.

[0064] In the shallow cross-attention structure, the channel vectors of the shallow visual encoding features at each spatial grid position are input into a linear transformation process to generate shallow key vectors and shallow value vectors. In the mid-level cross-attention structure, the channel vectors of the mid-level visual encoding features at each spatial grid position are input into a linear transformation process to generate mid-level key vectors and mid-level value vectors. In the deep cross-attention structure, the channel vectors of the deep visual encoding features at each spatial grid position are input into a linear transformation process to generate deep key vectors and deep value vectors.

[0065] The shallow key vector, shallow value vector, middle key vector, middle value vector, deep key vector, and deep value vector all maintain the same spatial grid arrangement as the original visual encoding features, and are used to establish cross-level attention relationships with the decoder query vector in the three cross-attention paths.

[0066] Furthermore, anomaly prior maps are constructed based on shallow and mid-level two-dimensional migration fields.

[0067] In each spatial grid coordinate The L2 norm of the difference vector between the shallow and middle layer 2D offset vectors is calculated, and the L2 norm results are normalized across all spatial grid coordinates to obtain the anomaly prior map. The possible values ​​of are represented as follows: ; in, For the anomalous prior map in spatial grid coordinates The value of , This is the coordinate index of the spatial grid traversed during the normalization process.

[0068] Represent the anomalous prior map in spatial grid coordinates The combination of the values ​​of the values, the offset smoothness index, and the consistency index constitutes the attention bias, which is applied to the spatial grid coordinates. The possible values ​​of are represented as follows: ; in, To focus attention on spatial grid coordinates The value of .

[0069] It should be noted that, in a preferred embodiment, in order to adaptively adjust the influence of the values ​​of the anomaly prior map in spatial grid coordinates, the offset smoothness index, and the consistency index on the attention bias construction process under different image scenes, a bias construction coefficient set can be constructed. Each bias construction coefficient in the bias construction coefficient set corresponds to a numerical component in the attention bias construction process. During the training phase, the anomaly prior map is mapped in spatial grid coordinates... The numerical values ​​of the bias composition coefficient, the offset smoothness index, and the consistency index are combined with their corresponding bias composition coefficients, and the combined results are input into the bias learning structure. The bias learning structure learns the bias composition coefficients through an iterative update method based on a loss function, enabling the bias composition coefficients to differentiate and adjust the various numerical components of the attention bias composition process according to the statistical regularities in the training data. After training, the set of bias composition coefficients serves as one of the bases for generating the attention bias combination method, and is correlated with the anomaly prior map in spatial grid coordinates. The values ​​of the metric, the offset smoothness metric, and the consistency metric together form the attention bias.

[0070] Furthermore, the shallow cross-attention structure, the middle cross-attention structure, and the deep cross-attention structure at different time steps... Based on the decoder query vector, shallow key vector, mid-level key vector, deep key vector, and attention bias, the shallow cross-attention weights, mid-level cross-attention weights, and deep cross-attention weights are calculated respectively, and expressed as follows: ; ; ; in, The shallow cross-attention weights at time step and spatial grid coordinates The value, For the cross-attention weights of the middle layer at time step and spatial grid coordinates The value, For deep cross-attention weights at time step and spatial grid coordinates The value, In time step The decoder query vector, Encoding shallow visual features in spatial grid coordinates Shallow key vectors, Encoding features of mid-level visual layers in spatial grid coordinates The middle layer key vector, Encoding features for deep visual representation in spatial grid coordinates The deep key vector, This represents the common dimension of the shallow key vector, middle key vector, deep key vector, and decoder query vector along the channel direction. This is the exponential normalization operation performed at all spatial grid locations.

[0071] The shallow, medium, and deep cross-attention weights are multiplied point-by-point with the shallow, medium, and deep value vectors, respectively, at all locations on the spatial grid. The results of these point-by-point multiplications are then summed along the spatial dimension to obtain the shallow, medium, and deep cross-attention aggregation vectors, as follows: ; ; ; in, In time step Shallow cross-attention aggregation vector, In time step The mid-level cross-attention aggregation vector, In time step Deep cross-attention aggregation vector, The shallow value vector in spatial grid coordinates The channel vector, The mid-level value vector in spatial grid coordinates The channel vector, For the deep value vector in spatial grid coordinates The channel vector.

[0072] The shallow, mid-level, and deep cross-attention aggregation vectors are concatenated along the channel direction to form a cross-level cross-attention aggregation vector, represented as: ; in, In time step Cross-level cross-attention aggregation vector, This refers to vector concatenation operations along the channel dimension.

[0073] S4. The cross-level attention aggregation vector is fed forward to generate a fusion vector. The fusion vector is semantically enhanced to form an initial enhancement vector. The initial enhancement vector is converged into the final decoder hidden state vector through iterative enhancement. The radiology report text is generated based on the final decoder hidden state vector.

[0074] Furthermore, the cross-level attention aggregation vector is input into the feedforward network structure of the Transformer decoder, which contains two layers of linear transformation and nonlinear activation function.

[0075] The feedforward network structure sequentially performs a first linear transformation, a nonlinear activation process, and a second linear transformation to generate cross-layer difference quantities through continuous transformations. Specifically, the first linear transformation is performed on the cross-layer cross-attention aggregation vector to generate an intermediate vector of the linear transformation in the channel dimension; the nonlinear activation process is performed on the intermediate vector of the linear transformation to form an activation vector in the channel dimension; the second linear transformation is performed on the activation vector to combine the activated values ​​in the channel dimension to obtain the cross-layer difference quantities.

[0076] Furthermore, the cross-layer difference quantity, decoder query vector label, and consistency index are combined into a ternary vector in the numerical dimension. The ternary vector is then subjected to a first linear transformation, a nonlinear activation process, and a second linear transformation in sequence. Through continuous linear mapping and nonlinear compression, the joint numerical features of the ternary vector are extracted layer by layer to generate a decision factor between zero and one.

[0077] Specifically, by performing a first linear transformation on the ternary combination vector, the first linear transformation output is obtained in the channel dimension; a nonlinear activation function is used to perform a nonlinear mapping on the first linear transformation output to form a ternary combination activation vector; then a second linear transformation is performed on the ternary combination activation vector to compress and combine the activated channel values ​​to obtain the decision factor.

[0078] The determination factor is used to characterize the combined influence of cross-layer feature differences, the smoothness of cross-layer offset changes, and the consistency of cross-layer correspondences.

[0079] Furthermore, to select fusion structures of different complexities based on the comprehensive performance of cross-layer difference, offset smoothness index, and consistency index during cross-layer fusion, the decision factors corresponding to multiple image scenes are statistically analyzed during the training phase. In the numerical distribution statistics, all decision factors are arranged in ascending order of value. The first quartile of the decision factor numerical distribution is determined as the first threshold, and the third quartile of the decision factor numerical distribution is determined as the second threshold. The first quartile reflects the typical distribution location of the smaller region of the decision factor, which is suitable for image scenes with smaller cross-layer difference, smoother offset field, and higher consistency. The third quartile reflects the typical distribution location of the larger region of the decision factor, which is suitable for image scenes with larger cross-layer difference, significant offset field fluctuation, or decreased consistency. The first threshold and the second threshold satisfy the condition that the first threshold is less than the second threshold and both are between zero and one, which is used to divide the value range of the decision factor into three fusion complexity intervals.

[0080] Based on the decision factor obtained at the current time step, the value range of the decision factor is divided into three fusion complexity ranges. When the decision factor is not greater than the first threshold, the decision factor is classified into the low complexity range; when the decision factor is greater than the first threshold but not greater than the second threshold, the decision factor is classified into the medium complexity range; when the decision factor is greater than the second threshold, the decision factor is classified into the high complexity range.

[0081] In the low-complexity range, the cross-level cross-attention aggregation vector is processed by a linear combination mainly based on channel gating (the channel gating parameters are learned parameters during the training phase and are automatically optimized through backpropagation of the loss function). By adjusting the retention and suppression ratios of each channel value in the channel dimension, a low-complexity fusion vector is obtained. In the medium-complexity range, a linear transformation is performed on the cross-level cross-attention aggregation vector, and nonlinear activation processing is performed in the channel dimension, so that the cross-level cross-attention aggregation vector forms a medium-complexity fusion vector with higher discriminativeness after a nonlinear mapping. In the high-complexity range, continuous linear transformation, nonlinear activation processing, and feature compression operations are performed on the cross-level cross-attention aggregation vector. Through multi-level combination transformation, the cross-level semantic information expression in the channel dimension is enhanced, resulting in a high-complexity fusion vector.

[0082] Furthermore, the channel vectors of the deep visual coding features at all spatial grid positions are numerically summed, and the summation result is averaged according to the number of spatial grid positions to form a deep global semantic representation vector.

[0083] Combine any one of the low-complexity, medium-complexity, or high-complexity fusion vectors (selected according to the interval to which the decision factor belongs) with the deep global semantic representation vector and the time step. The word embedding vectors are concatenated along the channel dimension to form a time step. The initial enhancement vector is expressed as: ; in, For time step The initial enhancement vector, For time step The fusion vector, This is a deep global semantic representation vector. For time step Word embedding vectors.

[0084] The word embedding vector is a fixed-length vector used to describe the semantic features of a word. It is obtained by performing a vector lookup on the discrete number of the current word in the word embedding matrix. The word embedding matrix is ​​a real-valued matrix learned during the language model training phase. Each row of the matrix corresponds to the vector representation of a word in the continuous semantic space. The word embedding vector can represent the semantic category, semantic similarity, and contextual relationship of a word in the numerical space.

[0085] Furthermore, for the initial enhancement vector at time step Iterative enhancement processing is performed; each round of iterative enhancement includes linear transformation, nonlinear activation processing, and numerical normalization, gradually adjusting the numerical distribution of the initial enhancement vector along the channel dimension and gradually aligning it with cross-layer fusion semantics, deep global semantics, and language context; as the number of iterations increases, the initial enhancement vector gradually converges in continuous transformations, forming a time step. The final decoder hidden state vector.

[0086] During the training phase, to ensure the stability of the iterative enhancement process under different medical imaging scenarios, the gradient vector of the parameters between two adjacent updates is differentially divided and normalized to obtain the gradient fluctuation, which is expressed as: ; in, For gradient fluctuations, For the first The gradient vector obtained during the next update For the first The gradient vector at the next update To prevent extremely small positive numbers with a denominator of zero.

[0087] The gradient fluctuation is used to adjust the learning rate used in the current iteration of augmentation, so that a larger gradient fluctuation results in a smaller learning rate, thus avoiding unstable updates; a smaller gradient fluctuation results in a learning rate closer to its initial value, which is beneficial for fast convergence; the learning rate update method is expressed as: ; in, For the first The learning rate used in the next iteration of the update is enhanced. This is the initial learning rate.

[0088] The initial learning rate is a configurable hyperparameter that serves as the baseline update magnitude for all update steps at the start of iterative augmentation. Example value range: .

[0089] After multiple rounds of iterative enhancement processing, the final decoder hidden state vector is obtained at the current time step. A linear mapping operation is performed on the final decoder hidden state vector to generate a word score vector in the vocabulary dimension. Exponential normalization is performed on the word score vector to form the word probability distribution at the current time step in the vocabulary dimension.

[0090] At all time steps, based on the word probability distribution formed at each time step, the word identifier with the highest probability in the word probability distribution is selected, and the word identifiers of all time steps are arranged in chronological order to form a radiographic report text used to represent visual content.

[0091] It should also be noted that during the training phase, medical images and corresponding radiological report texts are collected to form a training sample set. The normalized image data undergoes visual feature extraction and cross-level encoding processes to form shallow visual features, mid-level visual features, deep visual features, shallow visual coding features, mid-level visual coding features, deep visual coding features, shallow two-dimensional offset fields, mid-level two-dimensional offset fields, offset smoothness index, and consistency index. The target radiological report text is converted into a word sequence as a supervision signal. At each time step, the word embedding vector of the current time step and the hidden state generated in the previous time step are used together in the decoding calculation to obtain the cross-level cross-attention aggregation vector, cross-level difference quantity, decision factor, fusion vector, and the final decoder hidden state vector.

[0092] The hidden state vector is passed through the output mapping layer to obtain the word probability distribution, and a cross-entropy loss function is constructed based on the target lexical units. During the training phase, backpropagation is performed on the loss function to generate gradients for the parameters in the visual feature extraction process, the parameters in the cross-level encoding process, the parameters in each linear transformation in the decoding stage, the parameters in the three-element combination vector mapping, the parameters in the three types of fusion structures, the coefficients used in the attention bias construction, and the parameters in the output mapping layer. All gradients are passed through the network backpropagation path, and combined with the learning rate update method based on gradient fluctuation, the parameters are iteratively updated so that the visual feature representation, cross-level spatial relationship modeling, cross-level attention aggregation, cross-level fusion representation, and language generation process gradually converge on the training sample set, realizing end-to-end training.

[0093] In summary, this invention achieves the relevance between shallow, mid-level, and deep visual coding features by constructing cross-layer attention relationships during the encoding process, thus forming a unified representation of hierarchical visual features in terms of spatial correspondence and semantic connection; and by constructing a multi-path cross-attention structure and introducing attention bias, it achieves differentiated focusing of visual coding features at different depths, enabling cross-layer cross-attention aggregation vectors to strengthen the semantic expression of key regions.

[0094] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for generating radiological reports based on hierarchical interactive fusion, characterized in that: include, Raw image data is collected and intensity is normalized to obtain normalized image data. Multi-level visual feature extraction and feature dimension unification are performed on the normalized image data to form a hierarchical visual feature set. The hierarchical visual feature set is encoded, and cross-layer attention relationships are constructed between shallow and deep layers and between middle and deep layers during the encoding process to form shallow two-dimensional offset fields and middle two-dimensional offset fields. After nonlinear resampling, offset smoothness index and consistency index are obtained. In the decoding stage, a multi-path cross-attention structure is constructed based on the hierarchical visual feature set after encoding. Anomaly prior maps are constructed based on shallow two-dimensional offset fields and middle two-dimensional offset fields, and attention biases are formed to generate cross-level cross-attention aggregation vectors. The cross-level attention aggregation vector is fed forward to generate a fusion vector. The fusion vector is then semantically enhanced to form an initial enhancement vector. Through iterative enhancement, the initial enhancement vector is converged to the final decoder hidden state vector. Finally, the radiology report text is generated based on the final decoder hidden state vector.

2. The radiology report generation method based on hierarchical interactive fusion as described in claim 1, characterized in that: The multi-level visual feature extraction of the normalized image data includes using a pre-trained ResNet101 model as a visual feature extractor to extract shallow, mid-level, and deep features at different visual depths from the normalized image data.

3. The radiology report generation method based on hierarchical interactive fusion as described in claim 2, characterized in that: The unified feature dimension, forming a hierarchical visual feature set, includes using a single-layer linear function to perform linear mapping on shallow, medium, and deep features at spatial grid positions to generate shallow, medium, and deep visual features, and combining the shallow, medium, and deep visual features to form a hierarchical visual feature set.

4. The radiology report generation method based on hierarchical interactive fusion as described in claim 3, characterized in that: The formation of the shallow two-dimensional offset field and the middle two-dimensional offset field includes encoding the shallow visual features, middle visual features and deep visual features in the hierarchical visual feature set through a Transformer encoder to obtain shallow visual encoded features, middle visual encoded features and deep visual encoded features. Based on the cross-layer attention relationship between shallow visual coding features, mid-layer visual coding features and deep visual coding features, a set of deep visual coding feature grid positions is selected at each grid position, and the geometric center of the shallow-layer related deep spatial coordinates and the mid-layer related deep spatial coordinates are obtained respectively based on the spatial coordinates of the deep visual coding feature grid position set. The difference between the geometric center of the shallow-layer related deep spatial coordinates and the current shallow-layer grid position coordinates is used as the shallow-layer two-dimensional offset vector, and the difference between the geometric center of the middle-layer related deep spatial coordinates and the current middle-layer grid position coordinates is used as the middle-layer two-dimensional offset vector. These are then arranged to form the shallow-layer two-dimensional offset field and the middle-layer two-dimensional offset field, respectively.

5. The radiology report generation method based on hierarchical interactive fusion as described in claim 4, characterized in that: The process of obtaining the offset smoothness index and consistency index after nonlinear resampling includes performing nonlinear resampling on the shallow two-dimensional offset field and the middle two-dimensional offset field to obtain the shallow resampled deep visual coding features and the middle resampled deep visual coding features. The migration smoothness index is obtained by summing the difference vectors of the shallow two-dimensional migration field and the middle two-dimensional migration field in the spatial grid coordinate direction. The consistency index is obtained by performing squared difference processing on the difference between the shallow resampled deep visual coding features and the shallow visual coding features at the spatial grid position, and on the difference between the mid-layer resampled deep visual coding features and the mid-layer visual coding features at the spatial grid position.

6. The radiology report generation method based on hierarchical interactive fusion as described in claim 5, characterized in that: The construction of a multi-path cross-attention structure based on the encoded hierarchical visual feature set in the decoding stage includes representing the hidden state of the Transformer decoder as a decoder hidden state vector, and generating a decoder query vector by performing a linear transformation on the decoder hidden state vector. The decoder query vector is associated with shallow visual coding features, medium visual coding features and deep visual coding features respectively, forming shallow cross-attention structure, medium cross-attention structure and deep cross-attention structure; In the shallow, medium, and deep cross-attention structures, linear transformations are performed on the shallow, medium, and deep visual encoding features to generate shallow key vectors, shallow value vectors, medium key vectors, medium value vectors, deep key vectors, and deep value vectors.

7. The radiology report generation method based on hierarchical interactive fusion as described in claim 6, characterized in that: The step of constructing an anomaly prior map and forming an attention bias based on the shallow two-dimensional offset field and the middle two-dimensional offset field includes calculating the L2 norm of the difference vector between the shallow two-dimensional offset vector and the middle two-dimensional offset vector on the spatial grid coordinates, and performing normalization on the L2 norm results on all spatial grid coordinates to obtain the anomaly prior map. An attention bias is formed by numerically combining the anomaly prior map, offset smoothness index, and consistency index on spatial grid coordinates.

8. The radiology report generation method based on hierarchical interactive fusion as described in claim 7, characterized in that: The generation of cross-level cross-attention aggregation vectors includes, based on shallow cross-attention structures, mid-level cross-attention structures, and deep cross-attention structures, calculating shallow cross-attention weights, mid-level cross-attention weights, and deep cross-attention weights according to decoder query vectors, shallow key vectors, mid-level key vectors, deep key vectors, and attention biases. The shallow cross-attention weights, mid-level cross-attention weights, and deep cross-attention weights are multiplied point-by-point with the shallow, mid-level, and deep value vectors at all locations on the spatial grid, respectively. The point-by-point multiplication results are then summed in the spatial dimension to obtain the shallow cross-attention aggregation vector, mid-level cross-attention aggregation vector, and deep cross-attention aggregation vector. The shallow cross-attention aggregation vector, the mid-level cross-attention aggregation vector, and the deep cross-attention aggregation vector are concatenated along the channel direction to form a cross-level cross-attention aggregation vector.

9. The radiology report generation method based on hierarchical interactive fusion as described in claim 8, characterized in that: The step of generating a fusion vector from the cross-level cross-attention aggregation vector through feedforward processing and performing semantic enhancement processing on the fusion vector to form an initial enhancement vector includes inputting the cross-level cross-attention aggregation vector into the feedforward network structure of the Transformer decoder, and sequentially performing a first linear transformation, a nonlinear activation process, and a second linear transformation to generate a cross-level difference quantity. A ternary vector is formed by combining cross-layer differences, decoder query vectors, and consistency indicators, and a decision factor is generated. Based on the decision factor, a fusion vector is selected from the low-complexity fusion vector, medium-complexity fusion vector, and high-complexity fusion vector, and then concatenated with the deep global semantic representation vector obtained from the deep visual encoding features and the word embedding vector in the channel dimension to form the initial enhancement vector.

10. The radiology report generation method based on hierarchical interactive fusion as described in claim 9, characterized in that: The step of iteratively enhancing the initial enhancement vector to converge into the final decoder hidden state vector and generating radiological report text based on the final decoder hidden state vector includes performing multiple rounds of iterative enhancement processing on the initial enhancement vector, performing linear transformation, nonlinear activation processing and numerical normalization in each round of iterative enhancement in sequence to obtain the final decoder hidden state vector. Linear mapping and exponential normalization operations are performed on the hidden state vector of the final decoder to generate a word probability distribution. The word identifier with the highest probability in the word probability distribution is selected and arranged in chronological order to form the radiological report text.

Images