A system, medium, and computer device for medical image processing

Abstract

The application provides a system, medium and computer device for medical image processing, comprising an image preprocessing module, a graph convolution network module, a convolutional neural network module and a multi-modal feature fusion module; the image preprocessing module performs morphological operation on a tumor core area of a first image to obtain a second image; the graph convolution network module outputs a first feature vector which collects information of the tumor core area and a peritumoral neighborhood area; the convolutional neural network module performs convolution to extract deep features of the image to obtain a second feature vector; the first feature vector and the second feature vector are fused to obtain a first output and a second output, and the first output and the second output determine a final tumor identification result through a multi-layer perceptron. The system of the application performs image processing on an input breast ultrasound image, judges based on related features of the tumor core area and the peritumoral neighborhood area, and realizes classification of benign and malignant types of breast tumors.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of medical image processing, and in particular to a system, medium and computer device for medical image processing. BACKGROUND

[0002] According to statistical data, breast cancer has become the most common malignant tumor in women and one of the most common causes of cancer death in women. Early detection and diagnosis can provide the best guidance for tumor treatment plan, which is the key to reducing mortality and has important clinical research value. X-ray, ultrasound and magnetic resonance are the main screening methods for breast cancer detection, especially ultrasound, which is widely used because of its convenience, non-invasiveness, no radiation, real-time and good tolerance. However, to ensure the diagnostic accuracy of breast ultrasound, radiologists need to have rich experience and proficiency, in addition, medical image experts need to interpret the results which may be highly subjective. However, for the huge ultrasound image data generated in the clinic, medical image experts need to review the film one by one and give the corresponding judgment results, which is a huge workload, which leads to easy omission of small features on the image and misjudgment caused by subjective factors.

[0003] In view of the above problems, a computer-aided diagnosis system is needed to assist in identifying based on medical images. SUMMARY

[0004] The purpose of the present application is to provide a system, medium and computer device for medical image processing, which can automatically identify the benign and malignant of breast tumors based on ultrasound images.

[0005] To achieve the above purpose, the present application provides a system for medical image processing, which comprises an image preprocessing module, a graph convolution network module, a convolutional neural network module and a multi-modal feature fusion module.

[0006] The image preprocessing module is used for receiving a plurality of first images, the tumor core area of the first image is labeled with a mask image, a plurality of second images including an inflation area are obtained by performing morphological operations on the tumor core area of a plurality of first images, the difference set of the inflation area and the tumor core area is taken as a peritumoral area, the peritumoral area is evenly divided into eight peritumoral neighborhood areas along the circumferential direction, and a plurality of high-throughput imageomic features are extracted from the tumor core area and each peritumoral neighborhood area.

[0007] The graph convolution network module integrates the related information of the images of each peritumoral neighborhood area and the tumor core area to obtain the related information between each peritumoral neighborhood area and the tumor core area, and outputs a first feature vector that collects the information of the tumor core area and the peritumoral neighborhood area.

[0008] The convolutional neural network module is configured to normalize the plurality of second images to obtain a plurality of third images with uniform resolution, the convolutional neural network module comprising a convolutional block with a three-layer bottleneck structure, and the convolutional neural network module is configured to extract deep features of the third images to obtain a second feature vector.

[0009] The multi-modal feature fusion module is configured to fuse the first feature vector and the second feature vector to obtain a first output and a second output, and the first output and the second output are added after passing through a multi-layer perception machine to determine a final tumor identification result.

[0010] Optionally, the image preprocessing module extracts high-throughput image features from the tumor core region and each peritumoral neighborhood region, including 16 gray scale features, 53 texture features and 276 wavelet transform features.

[0011] Optionally, the 53 texture features include 22 gray level co-occurrence matrix features, 13 gray level run-length matrix features, 13 gray level zone matrix features and 5 neighborhood gray level difference matrix features.

[0012] Optionally, the convolutional neural network module comprises an input layer, a plurality of convolutional blocks with a three-layer bottleneck structure, an average pooling layer and an output layer.

[0013] Optionally, the convolutional block with a three-layer bottleneck structure comprises a 1x1 convolutional layer, a 3x3 convolutional layer and a 1x1 convolutional layer connected in sequence.

[0014] Optionally, the convolutional neural network module comprises four convolutional blocks with a three-layer bottleneck structure.

[0015] Optionally, the multi-modal feature fusion module fuses the first feature vector and the second feature vector to obtain a first output and a second output, specifically comprising:

[0016] The first feature vector is mapped to an intermediate representation space as Query1, the second feature vector is mapped to an intermediate representation space as Key1 and Value1, and an attention score is calculated:

[0017]

[0018] In the formula, Q1 represents Query1, K1 represents Key1, and d represents the dimension of Q1 and K1.

[0019] The attention score and Value1 are combined to guide the feature representation of Value1, as shown in the following formula:

[0020]

[0021] In the formula, alpha represents an optional scaling factor for controlling the degree of influence of the attention score; V1 represents Value1; and output1 represents the first output.

[0022] The second feature vector is mapped to an intermediate representation space as Query2, the first feature vector is mapped to the intermediate representation space as Key2 and Value2, and an attention score thereof is calculated:

[0023]

[0024] In the formula, Q2 represents Query2, K2 represents Key2, and d represents the dimension of Q2 and K2;

[0025] The attention score and Value2 are combined to guide the feature representation of Value2, as shown in the following formula:

[0026]

[0027] In the formula, alpha represents an optional scaling factor for controlling the degree of influence of the attention score; V2 represents Value2; and output2 represents the second output.

[0028] The application further provides a computer readable storage medium, which stores the system for medical image processing.

[0029] The application further provides a computer device, which comprises the storage medium.

[0030] Compared with the prior art, the system, medium and computer device for medical image processing have the following beneficial effects:

[0031] The system, medium and computer device for medical image processing realize the benign and malignant classification of breast tumors by performing image processing on the input breast ultrasound image, judging based on the related features of the tumor core area and the peritumoral neighborhood area, and automatically completing the process by an algorithm without manual intervention to extract the identification parameters, and have strong robustness. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 FIG. 1 is a structural schematic diagram of the system for medical image processing.

[0033] Figure 2 FIG. 2 is a flowchart of the breast cancer benign and malignant identification method based on the combination of the internal and external texture structure features and the spatial position features of the system for medical image processing.

[0034] Figure 3A flowchart of a system identification method for medical image processing.

[0035] Figure 4 A receiver operating characteristic curve diagram. DETAILED DESCRIPTION

[0036] The technical solutions, structural features, purposes achieved and effects of the embodiments of the present application will be described in detail below with reference to the accompanying drawings. Figure 1 ~Appendix Figure 4 The technical solutions, structural features, purposes achieved and effects of the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0037] It should be noted that the drawings are very simplified and all use non-precise proportions, only to facilitate, clear, auxiliary purpose of explaining the purpose of the present application, and not to limit the implementation of the present application, the conditions, therefore, not have the technical essence, any modification, proportion relationship change or size adjustment, without affecting the effect and the purpose of the present application can be achieved, still fall within the scope of the disclosed technology can cover.

[0038] It should be noted that in the present application, such as the first and second relationship terms are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations. Moreover, the term "include", "contain" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements not only includes the explicitly listed elements, but also includes other elements not explicitly listed, or includes the elements inherent in such process, method, article or equipment.

[0039] As shown in the figure, the present application provides a system for medical image processing, which includes an image preprocessing module, a graph convolution network module, a convolutional neural network module and a multi-modal feature fusion module. Figures 1-3

[0040] The division of tumor core area and peritumoral neighborhood area is the premise and basis for subsequent feature extraction and classification identification. First, the original image is manually drawn or the mask of the tumor core area is obtained through the tumor segmentation system, wherein the mask is used to highlight or isolate the core part of the tumor for further analysis or processing. The tumor core area refers to the most representative part inside the tumor that may contain the most malignant cells. This is usually the most active growth area or the part with the most abundant blood flow in the tumor.

[0041] ​The image preprocessing module receives a plurality of first images, and the first images are tumor core area mask marked images. A plurality of second images including an inflation area are obtained by performing morphological operations on the tumor core area of the plurality of first images. Specifically, the morphological operations include taking the mask area as an intratumoral area, using the cv2.dilate function, setting the kernel parameter to 4x4, and setting the iteration parameter to 10, to perform morphological operations on the mask area to obtain the inflation area. The difference set of the inflation area and the tumor core area is taken as a peritumoral area. A rectangular coordinate system is established with the tumor center as the origin, and each quadrant is then divided into two parts, so that eight sector areas can be obtained. The intersection part of the peritumoral area and one of the sector areas is recorded as a peritumoral neighborhood area, and finally eight peritumoral neighborhood areas are obtained.

[0042] In order to effectively utilize the spatial position correlation between the peritumoral neighborhood areas, the graph convolution network module integrates the relevant information of the images of the peritumoral neighborhood areas and the tumor core area to obtain the relevant information between the peritumoral neighborhood areas and the tumor core area, and outputs a first feature vector that collects the tumor core area and the peritumoral neighborhood area.

[0043] Specifically, the graph convolution network module extracts a plurality of high-throughput imageomics features from the tumor core area and each peritumoral neighborhood area, specifically a total of 345 high-throughput imageomics features, including 16 gray level features, 53 texture features, and 276 wavelet transform features. In this way, each image can be represented by a 9x345 feature vector to represent its information.

[0044] The 53 texture features include 22 gray level co-occurrence matrix features, 13 gray level run length matrix features, 13 gray level zone matrix features, and 5 neighborhood gray level difference matrix features.

[0045] The 16 gray level features are specifically: 1, energy; 2, histogram entropy; 3, peak value; 4, mean absolute error; 5, mean value; 6, median value; 7, gray level range; 8, root mean square; 9, skewness; 10, standard deviation; 11, histogram uniformity; 12, variance; 13, histogram mean; 14, histogram variance; 15, histogram skewness; and 16, histogram peak value.

[0046] The 22 gray level co-occurrence matrix features are specifically: 1, energy; 2, entropy; 3, difference; 4, contrast; 5, inverse difference moment; 6, correlation; 7, homogeneity; 8, auto-correlation; 9, cluster shade; 10, cluster prominence; 11, maximum probability; 12, sum of squares; 13, sum average; 14, total variance; 15, sum entropy; 16, difference variance; 17, difference entropy; 18, correlation-normalized; 19, co-occurrence-normalized; 20, maximum correlation coefficient; 21, inverse contrast-normalized; and 22, inverse difference-normalized.

[0047] The 13 gray scale run matrix features are: 23, short-term emphasis; 24, long-term emphasis; 25, gray scale non-uniformity; 26, long line non-uniformity; 27, long line percentage; 28, line metric of low gray scale value; 29, line length of high gray scale value; 30, short line low gray scale value line metric; 31, short line high gray scale value line metric; 32, long line low gray scale value line metric; 33, long line high gray scale value line metric; 34, gray scale value variance; and 35, long line variance.

[0048] The 13 gray scale region matrix features are: 36, small block metric; 37, large block metric; 38, gray scale non-uniformity; 39, block non-uniformity; 40, block percentage; 41, block metric of low gray scale value; 42, block metric of high gray scale value; 43, small region low gray scale value block metric; 44, small region high gray scale value block metric; 45, large region low gray scale value block metric; 46, large region high gray scale value block metric; 47, gray scale value variance; and 48, block size variance.

[0049] The 5 neighborhood gray scale difference matrix features are: 49, roughness; 50, contrast; 51, busyness; 52, complexity; and 53, intensity.

[0050] The 276 wavelet transform features are components obtained by separately analyzing four sub-bands obtained by wavelet decomposition. The wavelet transform features are used to represent information that cannot be directly reflected by gray scale and texture features.

[0051] The graph convolution network module includes a graph convolution network. The core idea of the graph convolution network is that each node will receive messages from its adjacent nodes, fuse the features of the adjacent nodes, and update the representation of the node. The graph can be represented as G=(V, E), where V and E represent the nodes and adjacent edges of the graph convolution network.

[0052] Each node in the graph represents one of the regions obtained by the segmentation, and the relationship between the nodes is dynamically updated with the training of the network. Since the nine regions obtained by the segmentation have an intuitive connection relationship, the connection relationship is used as the basis for constructing the adjacency matrix, that is, if two regions are connected, the adjacent edge value between the nodes of the two regions is 1, and if the two regions are not connected, the adjacent edge value is 0. The 9x345 feature vector obtained through the above steps is first normalized to the 0-1 interval, and then the representation of the feature vector is updated through the graph convolution network. Finally, a feature vector with a dimension of 9x512 is obtained. The graph convolution process on the graph can be described as:

[0053]

[0054] In the formula, the matrix H l represents the node features of the lth layer, H 0 = X input . σ represents a nonlinear activation function, and Wl-1 is a learnable weight matrix, denotes a normalized adjacency matrix, which can be calculated as:

[0055]

[0056] where I is an identity matrix, is the degree matrix of A. The whole process can be seen as the propagation and aggregation of information on the graph. The graph convolutional network can finally obtain a feature vector that comprehensively integrates the information of the tumor core region and the tumor peripheral region by learning the complex interactions between nodes.

[0057] In the present application, a convolutional neural network is constructed as a parallel branch to extract the local structure texture features of the tumor. Since a large part of the information in the breast ultrasound image is irrelevant to the gland and tissue, in order to reduce the influence of redundant information on the classification accuracy, the convolutional neural network module performs normalization processing on multiple second images to obtain multiple third images with uniform resolution. In order to facilitate subsequent unified processing, the resolution of the third image is selected to be 512x512.

[0058] The convolutional neural network module includes three layers of bottleneck structure convolutional blocks. The convolutional neural network module extracts deep features of the third image by convolution to obtain a second feature vector FC, wherein the output size of the second feature vector FC is 8x8x1024.

[0059] The convolutional neural network module includes an input layer, four layers of convolutional blocks containing three layers of bottleneck structure, an average pooling layer and an output layer. Among them, the first layer of convolution stage preliminarily processes the input image, extracts some basic features, and at the same time performs a certain degree of down-sampling to reduce the resolution of the image. For example, the size of the input image is reduced by half, and the number of channels is moderately increased; the second layer of convolution stage further extracts features on the basis of the first layer, the resolution continues to decrease, and the number of channels further increases. At this stage, the features become more abstract and representative; the third layer of convolution stage continues to deepen the feature extraction, the resolution is further reduced, the number of channels continues to increase, and the information in the feature map is more focused on the key features and patterns of the image; the fourth layer of convolution stage finally extracts the features in depth, the resolution reaches the minimum, and the number of feature channels reaches a relatively high value. At this time, the extracted features are deep features of the image, which contain rich semantic information and structural information. Overall, the convolutional neural network gradually reduces the resolution and increases the number of feature channels through the four layers of convolutional blocks to extract the deep features of the image, and finally obtains a feature vector containing local structure texture information through the average pooling layer.

[0060] The stage composed of the three-layer bottleneck structure convolution block includes sequentially connecting a 1*1 convolution layer, a 3*3 convolution layer and a 1*1 convolution layer. Among them, the first layer 1*1 convolution layer is responsible for dimension reduction operation on the input feature map, reducing the number of channels, reducing the complexity and parameter amount of subsequent calculation; the second layer 3*3 convolution layer is the main feature extraction layer, which is used to capture local features of the image, such as edge, texture and other information. Due to the dimension reduction processing of the first layer, the calculation amount of this layer is relatively small; the third layer is also a 1*1 convolution layer, which is used to restore the number of channels of the feature map to a suitable number so as to connect with the subsequent layer. Such a structure can ensure that the filter number of the 3*3 convolution layer is independent of the upper and lower layers when reducing or expanding the dimension of the feature map, which not only maintains the accuracy of the model, but also reduces the number of parameters. At the same time, the residual structure is added, which can speed up the convergence of the network and avoid gradient disappearance.

[0061] The multi-modal feature fusion module fuses the first feature vector FG (9*512) and the second feature vector FC (8*8*1024) to obtain a first output output1 and a second output output2, and the first output output1 and the second output output2 are added after passing through a multi-layer perception to determine the final tumor identification result.

[0062] The multi-modal feature fusion module fuses the first feature vector FG and the second feature vector FC to obtain a first output output1 and a second output output2, and the multi-modal feature fusion module fuses the first feature vector FG and the second feature vector FC to obtain a first output output1 and a second output output2.

[0063] The first feature vector FG is mapped to an intermediate representation space as Query1, and the second feature vector FC is mapped to an intermediate representation space as Key1 and Value1, and the attention score is calculated:

[0064]

[0065] In the formula, Q1 represents the vector Query1, K1 represents the vector Key1, and d represents the dimension of Q1 and K1.

[0066] The attention score and Value1 are combined, specifically, the attention score is multiplied by Value1 and weighted by a learnable parameter a, and then added to Value1, which can improve the stability of the module to guide the feature representation of Value1, and the process is as follows:

[0067]

[0068] In the formula, a represents a selectable scaling factor for controlling the influence degree of the attention score; V1 represents Value1; output1 represents the first output;

[0069] The second feature vector FC is mapped to the intermediate representation space as Query2, the first feature vector FG is mapped to the intermediate representation space as Key2 and Value2, and the attention score thereof is calculated:

[0070]

[0071] In the formula, Q2 represents Query2, K2 represents Key2, and d represents the dimension of Q2 and K2.

[0072] The attention score and Value2 are combined to guide the feature representation of Value2, as shown in the following formula:

[0073]

[0074] In the formula, alpha represents an optional scaling factor for controlling the degree of influence of the attention score; V2 represents Value2; and output2 represents the second output.

[0075] Finally, ACC (accuracy), SEN (sensitivity), SPE (specificity), AUC (Area Under the Curve), PPV (Positive Predictive Value), and NPV (Negative Predictive Value) are calculated to quantitatively evaluate the discrimination performance.

[0076] The network parameter setting of the system for medical image processing of the application is set as follows: the network is trained using the PyTorch deep learning framework on Nvidia GTX Titan X, the batchsize is set to 8, the initial learning rate of the Adam optimization algorithm is 1e-4, the binary cross-entropy loss function is used, and the definition is as follows:

[0077] L = ∑ [Y·logP + (1-Y)·log(1-P)]

[0078] In the formula, P is the probability value of the sample identified by the model, and the overall loss function of the network is designed as:

[0079] L total = λ1·L QKV + λ2·L G + λ3·L C

[0080] Wherein L G and L C represent the outputs of the graph convolution network and the convolutional neural network respectively, and L QKVThe output of the feature fusion module is represented, and λ1, λ2, and λ3 represent the weights of the respective losses.

[0081] The system of medical image processing of the application is experimented on an ultrasound breast cancer dataset and the effect is verified in a five-fold cross-validation manner. A public dataset is adopted, which is composed of 437 benign cases and 210 malignant cases, with an average size of 500*500 pixels, and corresponding segmentation masks from 600 female patients aged 25 to 75 years old. These data are collected by a LOGIQ E9 US system and a LOGIQ E9 Agile US system, and the sensor frequency of the linear array probe is 1-5 MHz.

[0082] As shown in Figure 4 The application performs a comparative experiment to prove the effectiveness of the method. The method of the application is compared with four existing methods, including Logistic Regression, Resnest18, Vision Transformer, and a literature method. Table 1 below gives the results of differentiating benign and malignant breast cancer on dataset A using different methods. It can be seen that the method of the application achieves the best performance. Compared with the other four methods, the highest accuracy of 96.66% is achieved, in addition, the AUC is 0.9839, the SEN is 92.58%, the SPE is 98.86%, the PPV is 97.71%, and the NPV is 96.14%. It can be seen that compared with the existing methods, the method of the application has overall improvement, better feature extraction capability, and illustrates the interaction of the intratumor and peritumoral region information.

[0083] Table 1. Differentiating results of benign and malignant breast cancer using different methods

[0084]

[0085] The application uses ultrasound grayscale images to differentiate benign and malignant breast cancer and achieves satisfactory results, which embodies the practicality and universality of the method of the application. In addition, since the entire identification process of the method of the application is automatically completed by the algorithm without manual intervention to extract identification parameters, the method has strong robustness in actual application.

[0086] The system for medical image processing of the application first constructs a graph convolution network module to extract the potential connection between the tumor core area and the peritumoral neighborhood area as the global feature of the image, secondly, constructs a convolutional neural network module to extract the structural texture feature of the tumor area as the local feature of the image, and finally establishes a multi-modal feature fusion module for interactive learning of the above global feature and local feature, to further improve the accuracy of benign and malignant tumor identification.

[0087] The difference between benign and malignant tumors not only exists in the core region of the tumor, but also exists in the difference of the peritumoral region. Therefore, the tumor core region and the peritumoral region are segmented first, the high flux features of the image of each region are extracted, the tumor images of different volumes and shapes are effectively converted into high-dimensional data depicting tumor information, and a graph convolution network model is constructed. According to the graph node features and the graph structure, the features are screened and learned, not only the relevance of the features of different regions is enhanced, but also the identification accuracy is improved. A convolutional neural network is established to extract deep detail features of the tumor region as another modal feature complement, and a feature fusion framework is established from the potential relationship between the multi-modal features, which can further improve the performance of classification and identification. The whole process is automatically completed, without manual extraction of parameters, and has strong robustness.

[0088] The application further provides a computer readable storage medium, which stores the system for medical image processing.

[0089] The application further provides a computer device, which comprises the storage medium.

[0090] Although the content of the application has been described in detail through the above preferred embodiments, it should be recognized that the above description should not be considered as a limitation of the application. After reading the above content, various modifications and alternatives of the application will be apparent to those skilled in the art. Therefore, the protection scope of the application should be defined by the appended claims.

Claims

1. A system for medical image processing, characterized in that, The system includes an image preprocessing module, a graph convolutional network module, a convolutional neural network module, and a multimodal feature fusion module; The image preprocessing module is used to receive multiple first images, the tumor core region of the first images having a masked marker image, and to obtain multiple second images including an inflated region by performing morphological operations on the tumor core region of the multiple first images. The difference between the inflated region and the tumor core region is used as the peritumoral region. The peritumoral region is divided into eight peritumoral neighborhood regions along the circumferential direction. Multiple high-throughput radiomics features are extracted from the tumor core region and each of the peritumoral neighborhood regions. The graph convolutional network module integrates the relevant information of the images of each of the peritumoral neighborhood regions and the tumor core region to obtain the relevant information between each of the peritumoral neighborhood regions and the tumor core region, and outputs a first feature vector that combines the information of the tumor core region and the peritumoral neighborhood regions. The convolutional neural network module is used to normalize multiple second images to obtain multiple third images with uniform resolution. The convolutional neural network module includes a three-layer bottleneck structure convolutional block. The convolutional neural network module performs convolution on the third images to extract deep image features to obtain a second feature vector. The multimodal feature fusion module fuses the first feature vector and the second feature vector to obtain a first output and a second output. The first output and the second output are then added together after passing through a multilayer perceptron to determine the final tumor identification result.

2. The system for medical image processing as described in claim 1, characterized in that, The image preprocessing module extracts high-throughput image features from the tumor core area and each of the surrounding tumor regions, including 16 grayscale features, 53 texture features, and 276 wavelet transform features.

3. The system for medical image processing as described in claim 2, characterized in that, The 53 texture features include 22 gray-level co-occurrence matrix features, 13 gray-level run-length matrix features, 13 gray-level region matrix features, and 5 neighborhood gray-level difference matrix features.

4. The system for medical image processing as described in claim 1, characterized in that, The convolutional neural network module includes an input layer, multiple layers of convolutional blocks containing three bottleneck structures, an average pooling layer, and an output layer.

5. The system for medical image processing as described in claim 4, characterized in that, The three-layer bottleneck structure convolutional block includes a 1×1 convolutional layer, a 3×3 convolutional layer, and a 1×1 convolutional layer connected in sequence.

6. The system for medical image processing as described in claim 4, characterized in that, It includes four layers of convolutional blocks, each containing a three-layer bottleneck structure.

7. The system for medical image processing as described in claim 1, characterized in that, The multimodal feature fusion module fuses the first feature vector and the second feature vector to obtain a first output and a second output, specifically including: Map the first feature vector to the intermediate representation space as Query1, and map the second feature vector to the intermediate representation space as Key1 and Value1, then calculate their attention scores: In the formula, Q1 represents Query1, K1 represents Key1, and d represents the dimensions of Q1 and K1; The attention score and Value1 are combined to guide the feature representation of Value1, as shown in the following formula: In the formula, α represents an optional scaling factor used to control the degree of influence of the attention score; V1 represents Value1; and output1 represents the first output. Map the second feature vector to the intermediate representation space as Query2, and map the first feature vector to the intermediate representation space as Key2 and Value2, then calculate their attention scores: In the formula, Q2 represents Query2, K2 represents Key2, and d represents the dimensions of Q2 and K2; The attention score and Value2 are combined to guide the feature representation of Value2, as shown in the following equation: In the formula, α represents an optional scaling factor used to control the degree of influence of the attention score; V2 represents Value2; and output2 represents the second output.

Images