Rheumatoid arthritis diagnosis and treatment auxiliary system and method

Through deep learning technology, semantic encoding and feature extraction of inspection information, medical record text information and joint image data in the rheumatoid arthritis diagnosis and treatment system, the multi-dimensional information is integrated to solve the problems of data singularity and diagnostic dependence of existing systems. Improves the level of automation and accuracy of diagnosis.

CN118866338BActive Publication Date: 2024-11-26JILIN UNIVERSITY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411319425.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-23
Publication Date
2024-11-26
Estimated Expiration
2044-09-23

AI Technical Summary

Technical Problem

The existing rheumatoid arthritis diagnosis and treatment system is limited by the singularity of data types when processing clinical data, failing to make full use of multi-dimensional information, resulting in limited diagnosis depth and breadth, and over-reliance on the personal experience of doctors, increasing work burdens and may lead to bias and inconsistency in diagnostic results.

Method used

Using deep learning-based information processing and image analysis technology, semantic encoding of inspection information and medical record text information, feature extraction of joint image data, and through cross-domain heterogeneous Transformer multimodal encoding module and attention-based mechanism significance Combined with the perception module, the joint image characteristics are integrated with the constraints of the examination information and medical record text information, and automatically determine whether you have rheumatoid arthritis.

Benefits of technology

A comprehensive evaluation of various types of clinical data has been achieved, which reduces the dependence on doctors' personal experience, improves the automation level of diagnosis and treatment assistance for rheumatoid arthritis, and enhances the accuracy and consistency of diagnosis.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118866338B_ABST
    Figure CN118866338B_ABST
Patent Text Reader

Abstract

The present application provides a rheumatoid arthritis diagnosis and treatment assistance system and method, which relates to the field of intelligent diagnosis and treatment assistance, and uses information processing and image analysis technology based on deep learning to semantically encode the inspection information and medical record text information of the patient object, and extract features from the joint image data of the patient object, so as to automatically judge whether the patient suffers from rheumatoid arthritis based on the significant joint perception features between the joint image fusion features under the constraints of the inspection information and the joint image fusion features under the constraints of the medical record text information. In this way, by processing multiple types of clinical data, the patient's condition can be automatically and comprehensively evaluated, while reducing the dependence on the doctor's personal experience and judgment, which is conducive to improving the automation level of rheumatoid arthritis diagnosis and treatment assistance.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of intelligent diagnosis and treatment assistance, and more specifically, to a rheumatoid arthritis diagnosis and treatment assistance system and method. Background Art

[0002] Rheumatoid arthritis (RA) is a systemic immune rheumatic disease characterized by polyarticular, symmetrical, erosive joint inflammation of the small joints of the hands and feet. The incidence of RA is as high as 0.3% to 1%. It is a difficult-to-treat disease with a recurrent course, lingering and high disability rate. It can affect multiple systems throughout the body and seriously endanger human health. In particular, the diagnosis of rheumatoid arthritis is often complicated because it may have similar symptoms to other diseases, such as osteoarthritis. Therefore, diagnosis is crucial to controlling disease progression and preventing permanent joint damage.

[0003] However, when processing clinical data, traditional rheumatoid arthritis diagnosis and treatment systems are often limited by the singleness of data types, such as focusing too much on laboratory test results or only analyzing medical record texts, while failing to fully utilize multi-dimensional information such as imaging, genetics, and lifestyle. This single perspective may ignore the overall picture of the disease and limit the depth and breadth of diagnosis. At the same time, due to the lack of efficient automated tools, these systems overly rely on doctors' personal experience and instant judgment in information analysis, which not only increases the workload of doctors, but may also lead to deviations and inconsistencies in diagnostic results.

[0004] Therefore, an optimized auxiliary program for the diagnosis and treatment of rheumatoid arthritis is needed. Summary of the invention

[0005] In order to solve the above-mentioned technical problems, the present application is proposed. The present application provides a rheumatoid arthritis diagnosis and treatment assistance system and method, which uses deep learning-based information processing and image analysis technology to semantically encode the test information and medical record text information of the patient, and extract features from the joint image data of the patient, so as to automatically judge whether the patient suffers from rheumatoid arthritis based on the significant joint perception features between the joint image fusion features under the constraints of the test information and the joint image fusion features under the constraints of the medical record text information. In this way, by processing multiple types of clinical data, the patient's condition can be automatically and comprehensively assessed, while reducing the reliance on the doctor's personal experience and judgment, which is conducive to improving the level of automation of rheumatoid arthritis diagnosis and treatment assistance.

[0006] According to one aspect of the present application, a rheumatoid arthritis diagnosis and treatment auxiliary system is provided, comprising:

[0007] A patient-related data collection module is used to obtain the patient's test information, medical record text information and joint imaging data;

[0008] A semantic coding module for test information and medical record text information, used for semantically coding the test information and the medical record text information to obtain a semantic coding feature vector for the test information and a semantic coding feature vector for the medical record text information;

[0009] A joint image feature extraction module, used for inputting the joint image data into a joint image feature extractor to obtain a joint image feature map;

[0010] The inspection information guides the joint image multimodal fusion module, which is used to input the inspection information semantic encoding feature vector and the joint image feature map into the cross-domain heterogeneous Transformer multimodal encoding module to obtain the joint image multimodal fusion feature map under the inspection information constraint;

[0011] A medical record text information guided joint image multimodal fusion module, used to input the medical record text information semantic encoding feature vector and the joint image feature map into the cross-domain heterogeneous Transformer multimodal encoding module to obtain a joint image multimodal fusion feature map under the constraint of medical record text information;

[0012] A joint state multimodal mixed saliency module, used for inputting the joint image multimodal fusion feature map under the inspection information constraint and the joint image multimodal fusion feature map under the medical record text information constraint into a salient joint perception module based on an attention mechanism to obtain a joint state multimodal mixed salient joint perception feature map as a joint state multimodal mixed salient joint perception feature;

[0013] The auxiliary diagnosis result generation module is used to obtain an auxiliary diagnosis result based on the multimodal mixed significant joint perception features of the joint state, and the auxiliary diagnosis result is used to indicate whether rheumatoid arthritis is caused.

[0014] In the above-mentioned rheumatoid arthritis diagnosis and treatment auxiliary system, the joint image feature extraction module is used to: input the joint image data into a joint image feature extractor based on a void convolutional neural network model to obtain the joint image feature map.

[0015] In the above-mentioned rheumatoid arthritis diagnosis and treatment auxiliary system, the inspection information guided joint image multimodal fusion module includes: an inspection information feature linear transformation shape reshaping unit, which is used to perform linear transformation and feature shape reshaping on the inspection information semantic coding feature vector to obtain the inspection information semantic coding modulation matrix; a joint image feature linear transformation unit, which is used to perform a first linear transformation and a second linear transformation on the joint image feature map to obtain a first linear transformation joint image feature map and a second linear transformation joint image feature map; a linear transformation joint image feature shape reshaping unit, which is used to perform feature shape reshaping on the first linear transformation joint image feature map and the second linear transformation joint image feature map to obtain a first linear transformation joint image feature matrix and a second linear transformation joint image feature matrix; an inspection information attention calculation unit, which is used to use the first linear transformation joint image feature matrix and the second linear transformation joint image feature matrix as a key matrix and a value matrix and the inspection information semantic coding modulation matrix as a query matrix, and input the query matrix, the key matrix and the value matrix into Tr ansformer structure to obtain the inspection information attention query joint image matrix; a joint image matrix shape reshaping convolution encoding unit, used to reshape and point convolution encoding the inspection information attention query joint image matrix to obtain the inspection information attention query joint image feature map; a joint image multimodal fusion feature generation unit under the inspection information constraint, used to multiply the inspection information attention query joint image feature map with the joint image feature map by position point to obtain the joint image multimodal fusion feature map under the inspection information constraint; wherein the inspection information attention calculation unit is used to: calculate the matrix product between the transposed matrix of the inspection information semantic coding modulation matrix and the first linear transformation joint image feature matrix and then input it into the softmax function to obtain the inspection information-joint image feature association matrix; use the inspection information-joint image feature association matrix as the weight matrix, calculate the matrix product between the weight matrix and the second linear transformation joint image feature matrix to obtain the inspection information attention query joint image matrix.

[0016] In the above-mentioned rheumatoid arthritis diagnosis and treatment auxiliary system, the test and inspection information feature linear transformation reshape reshaping unit is used to: matrix multiply the test and inspection information semantic coding feature vector with the test and inspection information modulation matrix, and then add the test and inspection information bias vector by position to obtain the test and inspection information semantic coding modulation bias vector; input the test and inspection information semantic coding modulation bias vector into the Sigmoid function for activation processing to obtain the test and inspection information semantic coding modulation activation vector; reshape the test and inspection information semantic coding modulation activation vector to obtain the test and inspection information semantic coding modulation matrix.

[0017] In the above-mentioned rheumatoid arthritis diagnosis and treatment auxiliary system, the joint image feature linear transformation unit is used to: multiply each feature matrix along the channel dimension in the joint image feature map by position with the first joint image modulation matrix, and then add them by position with the first joint image bias matrix to obtain a first joint image bias feature map; input the first joint image bias feature map into the Sigmoid function for activation processing to obtain the first linear transformation joint image feature map; multiply each feature matrix along the channel dimension in the joint image feature map by position with the second joint image modulation matrix, and then add them by position with the second joint image bias matrix to obtain a second joint image bias feature map; input the second joint image bias feature map into the Sigmoid function for activation processing to obtain the second linear transformation joint image feature map.

[0018] In the above-mentioned rheumatoid arthritis diagnosis and treatment auxiliary system, the joint state multimodal hybrid saliency module includes: a joint image multimodal fusion feature shape reshaping unit, which is used to reshape the joint image multimodal fusion feature map under the inspection and examination information constraint and the joint image multimodal fusion feature map under the medical record text information constraint to obtain the joint image multimodal fusion feature shape reshaping matrix under the inspection and examination information constraint and the joint image multimodal fusion feature shape reshaping matrix under the medical record text information constraint; an inspection and examination-medical record text channel-by-channel interactive perception unit, which is used to reshape the joint image multimodal fusion feature shape reshaping matrix under the inspection and examination information constraint and the joint image multimodal fusion feature shape reshaping matrix under the medical record text information constraint. The joint image multimodal fusion feature shape reshaping matrix is ​​input into the feature channel-by-channel interactive perception module to obtain the joint image dependency matrix under the inspection and medical record text constraints and the joint image dependency matrix under the medical record text-inspection and inspection constraints; the joint image dependency matrix pruning unit is used to input the joint image dependency matrix under the inspection and medical record text constraints and the joint image dependency matrix under the medical record text-inspection and inspection constraints into the random inactivation module to obtain the pruned joint image dependency matrix under the inspection and inspection-medical record text constraints and the pruned joint image dependency matrix under the medical record text-inspection and inspection constraints; the joint image multimodal fusion feature optimization unit under the medical record text information constraints, using The pruned joint image dependency matrix under the inspection and examination text constraint is matrix-multiplied with the joint image multimodal fusion feature shape reshaping matrix under the medical record text information constraint to obtain the multimodal fusion feature matrix of joint images under the dependency optimization medical record text information constraint; the multimodal fusion feature optimization unit of joint images under the inspection and examination information constraint is used to matrix-multiply the joint image dependency matrix under the pruned medical record text-inspection and examination constraint with the multimodal fusion feature shape reshaping matrix of joint images under the inspection and examination information constraint to obtain the multimodal fusion feature matrix of joint images under the dependency optimization inspection and examination information constraint; the multimodal fusion feature optimization unit of joint images under the inspection and examination information constraint is used to matrix-multiply the joint image dependency matrix under the pruned medical record text-inspection and examination constraint with the multimodal fusion feature shape reshaping matrix of joint images under the inspection and examination information constraint to obtain the multimodal fusion feature matrix of joint images under the dependency optimization inspection and examination information constraint; A shape reshaping unit is used to perform feature shape reshaping on the multimodal fusion feature matrix of joint images under the constraint of the dependency optimized medical record text information and the multimodal fusion feature matrix of joint images under the constraint of the dependency optimized inspection and examination information to obtain the multimodal fusion feature map of joint images under the constraint of the optimized medical record text information and the multimodal fusion feature map of joint images under the constraint of the optimized inspection and examination information; a joint state multimodal mixed significant joint perception feature generation unit is used to calculate the weighted sum of the multimodal fusion feature map of joint images under the constraint of the optimized inspection and examination information and the multimodal fusion feature map of joint images under the constraint of the optimized medical record text information to obtain the multimodal mixed significant joint perception feature map of joint states.

[0019] In the above-mentioned rheumatoid arthritis diagnosis and treatment auxiliary system, the test and examination-medical record text channel-by-channel interactive perception unit is used to: calculate the matrix product between the transposed matrix of the joint image multimodal fusion feature shape reshaping matrix under the test and examination information constraint and the joint image multimodal fusion feature shape reshaping matrix under the medical record text information constraint to obtain a first related feature matrix; calculate the positional division of the scale of the first related feature matrix and the first related feature matrix to obtain a first related feature modulation matrix; calculate the matrix product between the transposed matrix of the joint image multimodal fusion feature shape reshaping matrix under the medical record text information constraint and the joint image multimodal fusion feature shape reshaping matrix under the test and examination information constraint to obtain a second related feature matrix; calculate the positional division of the scale of the second related feature matrix and the second related feature matrix to obtain a second related feature modulation matrix; input the first related feature modulation matrix and the second related feature modulation matrix into the softmax function respectively to obtain the joint image dependency matrix under the test and examination-medical record text constraint and the joint image dependency matrix under the medical record text-test and examination constraint.

[0020] In the above-mentioned rheumatoid arthritis diagnosis and treatment auxiliary system, the auxiliary diagnosis result generation module is used to: input the joint state multimodal mixed significant joint perceptual feature map into the classifier-based RA auxiliary diagnosis module to obtain the auxiliary diagnosis result, and the auxiliary diagnosis result is used to indicate whether the patient suffers from rheumatoid arthritis.

[0021] According to another aspect of the present application, a method for assisting diagnosis and treatment of rheumatoid arthritis is provided, comprising:

[0022] Obtaining the examination information, medical record text information and joint imaging data of the patient;

[0023] Performing semantic coding on the test information and the medical record text information to obtain a test information semantic coding feature vector and a medical record text information semantic coding feature vector;

[0024] Inputting the joint image data into a joint image feature extractor to obtain a joint image feature map;

[0025] Inputting the inspection information semantic encoding feature vector and the joint image feature map into a cross-domain heterogeneous Transformer multimodal encoding module to obtain a joint image multimodal fusion feature map under the inspection information constraint;

[0026] Inputting the semantic encoding feature vector of the medical record text information and the joint image feature map into the cross-domain heterogeneous Transformer multimodal encoding module to obtain a joint image multimodal fusion feature map under the constraint of the medical record text information;

[0027] Inputting the multimodal fusion feature map of joint images under the constraint of the inspection information and the multimodal fusion feature map of joint images under the constraint of the medical record text information into a significant joint perception module based on the attention mechanism to obtain a multimodal mixed significant joint perception feature map of joint states as a multimodal mixed significant joint perception feature of joint states;

[0028] Based on the multimodal mixed significant joint perception features of the joint status, an auxiliary diagnosis result is obtained, and the auxiliary diagnosis result is used to indicate whether rheumatoid arthritis is present.

[0029] In the above-mentioned rheumatoid arthritis diagnosis and treatment auxiliary method, the joint image data is input into a joint image feature extractor to obtain a joint image feature map, including: inputting the joint image data into a joint image feature extractor based on a void convolutional neural network model to obtain the joint image feature map.

[0030] Compared with the prior art, the rheumatoid arthritis diagnosis and treatment assistance system and method provided by the present application uses information processing and image analysis technology based on deep learning to semantically encode the inspection information and medical record text information of the patient object, and extracts features from the joint image data of the patient object, so as to automatically determine whether the patient suffers from rheumatoid arthritis based on the significant joint perception features between the joint image fusion features under the constraints of the inspection information and the joint image fusion features under the constraints of the medical record text information. In this way, by processing multiple types of clinical data, the patient's condition can be automatically and comprehensively evaluated, while reducing the dependence on the doctor's personal experience and judgment, which is conducive to improving the automation level of rheumatoid arthritis diagnosis and treatment assistance. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0032] Figure 1 This is a system block diagram of a rheumatoid arthritis diagnosis and treatment assistance system according to an embodiment of the present application.

[0033] Figure 2 Schematic diagram of data flow of the rheumatoid arthritis diagnosis and treatment assistance system according to an embodiment of the present application.

[0034] Figure 3 Flow chart of a method for assisting diagnosis and treatment of rheumatoid arthritis according to an embodiment of the present application. DETAILED DESCRIPTION

[0035] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0036] Rheumatoid Arthritis (RA) is a systemic immune disease characterized by symmetrical and erosive inflammation of the small joints of the hands and feet, which usually affects multiple joints. The prevalence of RA is between 0.3% and 1%. It is a difficult-to-treat disease with a recurrent and chronic course. It can easily lead to permanent damage to the joints and may affect multiple systems throughout the body, posing a serious threat to human health. It is worth noting that the diagnosis of RA is relatively complicated because its symptoms may be similar to those of other diseases such as osteoarthritis. Therefore, accurate diagnosis is crucial for disease management and prevention of joint damage.

[0037] However, existing rheumatoid arthritis diagnosis and treatment systems are often limited by the limitations of data types when processing clinical data. These systems may rely too much on a single type of data, such as laboratory test results or only medical record text analysis, without fully utilizing multi-dimensional information including imaging, genetics, and lifestyle. This one-sided use of data may ignore the comprehensiveness of the disease, thereby limiting the depth and breadth of diagnosis. In addition, due to the lack of efficient automated tools, these systems rely too much on doctors' personal experience and instant judgment in information processing, which not only increases the workload of doctors, but may also cause deviations and inconsistencies in diagnostic results.

[0038] In response to the above technical problems, the technical concept of the present application is to obtain the test information, medical record text information and joint image data of the patient, and use information processing and image analysis technology based on deep learning to semantically encode the test information and medical record text information, and extract features from the joint image data, so as to automatically judge whether the patient suffers from rheumatoid arthritis based on the significant joint perception features between the joint image fusion features under the constraints of the test information and the joint image fusion features under the constraints of the medical record text information. In this way, multiple types of clinical data can be processed at the same time, and the fusion of multi-dimensional data is realized, so that the patient's condition can be evaluated more comprehensively. In addition, the deep learning technology is used to automatically encode and extract features of various clinical data, which reduces the dependence on the doctor's personal experience and judgment, and improves the automation level of rheumatoid arthritis diagnosis and treatment assistance.

[0039] Figure 1 This is a system block diagram of a rheumatoid arthritis diagnosis and treatment assistance system according to an embodiment of the present application. Figure 2 FIG. 1 is a data flow diagram of a rheumatoid arthritis diagnosis and treatment assistance system according to an embodiment of the present application. Figure 1 and Figure 2 As shown, in the rheumatoid arthritis diagnosis and treatment auxiliary system 100, it includes: a patient object related data acquisition module 110, which is used to obtain the test information, medical record text information and joint image data of the patient object; a test information and medical record text information semantic encoding module 120, which is used to semantically encode the test information and the medical record text information to obtain the test information semantic encoding feature vector and the medical record text information semantic encoding feature vector; a joint image feature extraction module 130, which is used to input the joint image data into a joint image feature extractor to obtain a joint image feature map; a test information guided joint image multimodal fusion module 140, which is used to input the test information semantic encoding feature vector and the joint image feature map into a cross-domain heterogeneous Transformer multimodal encoding module to obtain a joint image multimodal fusion feature map under the test information constraint. Figure; a medical record text information guided joint image multimodal fusion module 150, which is used to input the medical record text information semantic encoding feature vector and the joint image feature map into the cross-domain heterogeneous Transformer multimodal encoding module to obtain a joint image multimodal fusion feature map under the constraint of medical record text information; a joint state multimodal mixed saliency module 160, which is used to input the joint image multimodal fusion feature map under the constraint of inspection information and the joint image multimodal fusion feature map under the constraint of medical record text information into the attention mechanism-based salient joint perception module to obtain a joint state multimodal mixed salient joint perception feature map as a joint state multimodal mixed salient joint perception feature; an auxiliary diagnosis result generation module 170, which is used to obtain an auxiliary diagnosis result based on the joint state multimodal mixed salient joint perception feature, and the auxiliary diagnosis result is used to indicate whether the patient suffers from rheumatoid arthritis.

[0040] In the embodiment of the present application, the patient object related data acquisition module 110 is used to obtain the inspection information, medical record text information and joint image data of the patient object. It should be understood that the inspection information of the patient object usually includes blood test results for rheumatoid factor, anti-cyclic citrullinated peptide antibody, erythrocyte sedimentation rate and C-reactive protein. Specifically, anti-cyclic citrullinated peptide antibody is a highly specific marker for rheumatoid arthritis, and a positive result is strongly correlated with rheumatoid arthritis; while rheumatoid factor, erythrocyte sedimentation rate and C-reactive protein are usually elevated in patients with rheumatoid arthritis. The medical record text information of the patient object is usually a description of the patient's main complaint symptoms, specifically the symptoms that appear, the duration of the symptoms, the severity of the symptoms, etc. The joint image data of the patient object usually includes X-ray images that can show changes in joint structure and MRI images that provide more detailed joint soft tissue and bone marrow status. In general, the inspection information, medical record text information and joint image data each provide important clinical information, which are interrelated in different ways to support the diagnosis of rheumatoid arthritis. Through comprehensive analysis of these data, it is possible to more accurately determine whether the patient suffers from rheumatoid arthritis, thereby improving the diagnostic accuracy of rheumatoid arthritis. In particular, in a specific implementation of the embodiment of the present application, the test information, medical record text information and joint imaging data of the patient object can be obtained from the patient information management system of the hospital.

[0041] In an embodiment of the present application, the test information and medical record text information semantic encoding module 120 is used to semantically encode the test information and the medical record text information to obtain a test information semantic encoding feature vector and a medical record text information semantic encoding feature vector. Accordingly, considering that both the test information and the medical record text information contain state semantic feature information about the patient object, therefore, in the technical solution of the present application, the test information and the medical record text information are semantically encoded to capture and extract the key descriptive semantic content in the information, and obtain a test information semantic encoding feature vector and a medical record text information semantic encoding feature vector.

[0042] In an embodiment of the present application, the joint image feature extraction module 130 is used to input the joint image data into a joint image feature extractor to obtain a joint image feature map. Specifically, in an embodiment of the present application, the joint image feature extraction module is used to: input the joint image data into a joint image feature extractor based on a hole convolutional neural network model to obtain the joint image feature map. It should be understood that the joint image data contains multi-level joint detail information and morphological features, and the hole convolution can capture features of different scales, which helps to extract multi-level detail information from the joint image, including local microstructures and global morphological features. Based on this, in the technical solution of the present application, the joint image data is input into a joint image feature extractor based on a hole convolutional neural network model to obtain a joint image feature map. In particular, the essence of the joint image feature extractor based on the hole convolutional neural network model here is a hole convolutional neural network model.

[0043] In an embodiment of the present application, the test information guides the joint image multimodal fusion module 140, which is used to input the test information semantic coding feature vector and the joint image feature map into the cross-domain heterogeneous Transformer multimodal coding module to obtain the joint image multimodal fusion feature map under the test information constraint. Accordingly, it is considered that the test information semantic coding feature vector is a representation of the patient's test results (such as blood indicators) after semantic coding. The joint image feature map captures the representation of the visual features of the morphological changes of the joints. In order to integrate the data of the above two different modalities to create a joint feature representation, in the technical solution of the present application, the test information semantic coding feature vector and the joint image feature map are cross-modally encoded and fused by using a cross-domain heterogeneous Transformer multimodal coding module to obtain the joint image multimodal fusion feature map under the test information constraint.

[0044] Specifically, the cross-domain heterogeneous Transformer multimodal encoding module first generates a semantic coding modulation matrix of inspection information by linearly transforming and reshaping the feature shape of the inspection information semantic coding feature vector. At the same time, the joint image feature map is linearly transformed twice and the feature shape is reshaped to obtain a first linear transformation joint image feature matrix and a second linear transformation joint image feature matrix. Then, the two sets of linear transformation feature matrices are used as key matrices and value matrices, and the inspection information semantic coding modulation matrix is ​​used as a query matrix, and input into the Transformer structure to utilize the Transformer's self-attention mechanism to achieve the fusion and mutual correlation of the cross-modal features of the inspection information and joint features, thereby capturing the complex relationship between different modalities. Next, the result matrix obtained by the attention mechanism encoding is reshaped and dot convolution encoded to generate the inspection information attention query joint image feature map. Finally, by calculating the positional point product between the inspection information attention query joint image feature map and the joint image feature map, a multimodal fusion feature map of joint images under the inspection information constraint containing inspection information and joint information is generated.

[0045] Specifically, in an embodiment of the present application, the inspection information guided joint image multimodal fusion module includes: an inspection information feature linear transformation shape reshaping unit, which is used to perform linear transformation and feature shape reshaping on the inspection information semantic coding feature vector to obtain an inspection information semantic coding modulation matrix; a joint image feature linear transformation unit, which is used to perform a first linear transformation and a second linear transformation on the joint image feature map to obtain a first linear transformation joint image feature map and a second linear transformation joint image feature map; a linear transformation joint image feature shape reshaping unit, which is used to perform feature shape reshaping on the first linear transformation joint image feature map and the second linear transformation joint image feature map to obtain a first linear transformation joint image feature matrix and a second linear transformation joint image feature matrix; an inspection information attention calculation unit, which is used to use the first linear transformation joint image feature matrix and the second linear transformation joint image feature matrix as a key matrix and a value matrix and the inspection information semantic coding modulation matrix as a query matrix, and input the query matrix, the key matrix and the value matrix into a Tran sformer structure to obtain the inspection information attention query joint image matrix; a joint image matrix shape reshaping convolution encoding unit, used to reshape and point convolution encoding the inspection information attention query joint image matrix to obtain the inspection information attention query joint image feature map; a joint image multimodal fusion feature generation unit under the inspection information constraint, used to multiply the inspection information attention query joint image feature map with the joint image feature map by position point to obtain the joint image multimodal fusion feature map under the inspection information constraint; wherein the inspection information attention calculation unit is used to: calculate the matrix product between the transposed matrix of the inspection information semantic coding modulation matrix and the first linear transformation joint image feature matrix and then input it into the softmax function to obtain the inspection information-joint image feature association matrix; use the inspection information-joint image feature association matrix as the weight matrix, calculate the matrix product between the weight matrix and the second linear transformation joint image feature matrix to obtain the inspection information attention query joint image matrix.

[0046] More specifically, in an embodiment of the present application, the inspection and check information feature linear transformation reshape reshaping unit is used to: perform matrix multiplication of the inspection and check information semantic coding feature vector and the inspection and check information modulation matrix, and then add the matrix multiplication and the inspection and check information bias vector by position to obtain the inspection and check information semantic coding modulation bias vector; input the inspection and check information semantic coding modulation bias vector into a Sigmoid function for activation processing to obtain the inspection and check information semantic coding modulation activation vector; and reshape the inspection and check information semantic coding modulation activation vector to obtain the inspection and check information semantic coding modulation matrix.

[0047] More specifically, in an embodiment of the present application, the joint image feature linear transformation unit is used to: multiply each feature matrix along the channel dimension in the joint image feature map by position with the first joint image modulation matrix, and then add them by position with the first joint image bias matrix to obtain a first joint image bias feature map; input the first joint image bias feature map into the Sigmoid function for activation processing to obtain the first linear transformation joint image feature map; multiply each feature matrix along the channel dimension in the joint image feature map by position with the second joint image modulation matrix, and then add them by position with the second joint image bias matrix to obtain a second joint image bias feature map; input the second joint image bias feature map into the Sigmoid function for activation processing to obtain the second linear transformation joint image feature map.

[0048] In an embodiment of the present application, specifically, the inspection information guides the joint image multimodal fusion module, which is used to: input the inspection information semantic coding feature vector and the joint image feature map into the cross-domain heterogeneous Transformer multimodal coding module, and process them with the following cross-domain coding formula to obtain the joint image multimodal fusion feature map under the inspection information constraint; wherein, the cross-domain coding formula is:

[0049] ;

[0050] ;

[0051] ;

[0052] ;

[0053] ;

[0054] ;

[0055] ;

[0056] ;

[0057] ;

[0058] in, a semantically encoded feature vector for the inspection information, for said test checking information modulation matrix, checking information bias vector for said test, is the Sigmoid function, modulating an activation vector for semantic coding of the inspection information, For the reshape operation, a semantic coding modulation matrix for said inspection information, is the joint image feature map, and are respectively the first joint image modulation matrix and the second joint image modulation matrix, and are the first joint image offset matrix and the second joint image offset matrix respectively, and are respectively the first linear transformation joint image feature map and the second linear transformation joint image feature map, and are respectively the first linear transformation joint image feature matrix and the second linear transformation joint image feature matrix, is the transposed matrix of the semantic coding modulation matrix of the inspection information, for function, is the matrix product, Attentionally querying the joint image matrix for said inspection information, is the point convolution operation, Attentionally querying the joint image feature map for the inspection information, For point-by-position multiplication, It is the joint image multimodal fusion feature map under the inspection information constraint. Similarly, the encoding method of the medical record text information semantic encoding feature vector and the joint image feature map is also as shown in the above encoding.

[0059] In an embodiment of the present application, the medical record text information guides the joint image multimodal fusion module 150, which is used to input the medical record text information semantic encoding feature vector and the joint image feature map into the cross-domain heterogeneous Transformer multimodal encoding module to obtain the joint image multimodal fusion feature map under the constraint of the medical record text information. It should be understood that in order to perform cross-modal joint interactive fusion of the medical record text information semantic encoding feature vector and the joint image feature map, so as to comprehensively represent the fusion feature information between the medical record information and the joint image features, in the technical solution of the present application, the medical record text information semantic encoding feature vector and the joint image feature map are input into the cross-domain heterogeneous Transformer multimodal encoding module to obtain the joint image multimodal fusion feature map under the constraint of the medical record text information. That is, the fusion processing through the cross-domain heterogeneous Transformer multimodal encoding module can integrate data from different sources (medical record text information and joint image data) together, so as to simultaneously use the semantic information in the text and the visual information in the image to better represent the patient's condition.

[0060] In an embodiment of the present application, the joint state multimodal hybrid saliency module 160 is used to input the joint image multimodal fusion feature map under the inspection information constraint and the joint image multimodal fusion feature map under the medical record text information constraint into the salient joint perception module based on the attention mechanism to obtain the joint state multimodal hybrid salient joint perception feature map as the joint state multimodal hybrid salient joint perception feature. Considering that the joint image multimodal fusion feature map under the inspection information constraint and the joint image multimodal fusion feature map under the medical record text information constraint respectively contain the understanding of the joint state from different angles. Therefore, in order to integrate these two kinds of information to obtain a more comprehensive joint state representation, in the technical solution of the present application, the joint image multimodal fusion feature map under the inspection information constraint and the joint image multimodal fusion feature map under the medical record text information constraint are input into the salient joint perception module based on the attention mechanism to obtain the joint state multimodal hybrid salient joint perception feature map. In particular, the salient joint perception module based on the attention mechanism uses a dual interactive attention mechanism to enhance the feature interaction between the two, thereby enriching the expression ability of the feature. Specifically, the joint image multimodal fusion feature map under the constraint of inspection information and the joint image multimodal fusion feature map under the constraint of medical record text information are reshaped to maintain the spatial and channel information of the features. Then, the feature channel-by-channel interaction perception module is used to quantify and reveal the interaction between the two to obtain two dependency matrices. Then, the two dependency matrices are input into the random dropout module for pruning to achieve weight sparsification. In particular, the random dropout pruning operation is a regularization technique that selectively ignores or suppresses part of the feature interaction by introducing randomness, that is, the dropout process helps prevent the model from over-relying on a specific feature combination, thereby reducing the sensitivity to specific noise or outliers in the training data to obtain the pruned dependency matrix. Subsequently, the pruned features and the original features are weighted to strengthen the important parts of each original feature. Finally, the weighted features are reshaped and weighted fused to further enhance the key information between the two, generating a richer multimodal mixed salient joint perception feature map of joint state.

[0061] Specifically, in an embodiment of the present application, the joint state multimodal hybrid saliency module includes: a joint image multimodal fusion feature shape reshaping unit, which is used to perform feature shape reshaping on the joint image multimodal fusion feature map under the inspection and examination information constraint and the joint image multimodal fusion feature map under the medical record text information constraint to obtain a joint image multimodal fusion feature shape reshaping matrix under the inspection and examination information constraint and a joint image multimodal fusion feature shape reshaping matrix under the medical record text information constraint; a inspection and examination-medical record text channel-by-channel interactive perception unit, which is used to perform feature shape reshaping on the joint image multimodal fusion feature shape reshaping matrix under the inspection and examination information constraint and the joint image multimodal fusion feature shape reshaping matrix under the medical record text information constraint. The modal fusion feature shape reshaping matrix is ​​input into the feature channel-by-channel interactive perception module to obtain the joint image dependency matrix under the inspection and medical record text constraint and the joint image dependency matrix under the medical record text-inspection and inspection constraint; the joint image dependency matrix pruning unit is used to input the joint image dependency matrix under the inspection and medical record text constraint and the joint image dependency matrix under the medical record text-inspection and inspection constraint into the random inactivation module to obtain the pruned joint image dependency matrix under the inspection and inspection-medical record text constraint and the pruned joint image dependency matrix under the medical record text-inspection and inspection constraint; the joint image multimodal fusion feature optimization unit under the medical record text information constraint is used to The pruned inspection and examination-medical record text constraint joint image dependency matrix is ​​matrix-multiplied with the medical record text information constraint joint image multimodal fusion feature shape reshaping matrix to obtain the dependency optimized medical record text information constraint joint image multimodal fusion feature matrix; the inspection and examination information constraint joint image multimodal fusion feature optimization unit is used to matrix-multiply the pruned medical record text-inspection and examination constraint joint image dependency matrix with the inspection and examination information constraint joint image multimodal fusion feature shape reshaping matrix to obtain the dependency optimized medical record text constraint joint image multimodal fusion feature matrix; the inspection and examination information constraint joint image multimodal fusion feature shape reshaping unit is used to matrix-multiply the medical record text-inspection and examination constraint joint image dependency matrix with the inspection and examination information constraint joint image multimodal fusion feature matrix to obtain the dependency optimized inspection and examination information constraint joint image multimodal fusion feature matrix; the inspection and examination information constraint joint image multimodal fusion feature shape reshaping unit is used to matrix-multiply the medical record text-inspection and examination constraint joint image dependency matrix with the inspection and examination information constraint joint image multimodal fusion feature matrix A reshaping unit is used to reshape the feature shape of the joint image multimodal fusion feature matrix under the dependency optimization medical record text information constraint and the joint image multimodal fusion feature matrix under the dependency optimization inspection and examination information constraint to obtain the joint image multimodal fusion feature map under the optimization medical record text information constraint and the joint image multimodal fusion feature map under the optimization inspection and examination information constraint; a joint state multimodal mixed significant joint perception feature generation unit is used to calculate the weighted sum of the joint image multimodal fusion feature map under the optimization inspection and examination information constraint and the joint image multimodal fusion feature map under the optimization medical record text information constraint to obtain the joint state multimodal mixed significant joint perception feature map.

[0062] More specifically, in an embodiment of the present application, the test examination-medical record text channel-by-channel interactive perception unit is used to: calculate the matrix product between the transposed matrix of the joint image multimodal fusion feature shape reshaping matrix under the test examination information constraint and the joint image multimodal fusion feature shape reshaping matrix under the medical record text information constraint to obtain a first related feature matrix; calculate the positional division of the scale of the first related feature matrix and the first related feature matrix to obtain a first related feature modulation matrix; calculate the matrix product between the transposed matrix of the joint image multimodal fusion feature shape reshaping matrix under the medical record text information constraint and the joint image multimodal fusion feature shape reshaping matrix under the test examination information constraint to obtain a second related feature matrix; calculate the positional division of the scale of the second related feature matrix and the second related feature matrix to obtain a second related feature modulation matrix; input the first related feature modulation matrix and the second related feature modulation matrix into the softmax function respectively to obtain the joint image dependency matrix under the test examination-medical record text constraint and the joint image dependency matrix under the medical record text-test examination constraint.

[0063] In the embodiment of the present application, specifically, the joint state multimodal hybrid saliency module is used to: input the joint image multimodal fusion feature map under the inspection information constraint and the joint image multimodal fusion feature map under the medical record text information constraint into the salient joint perception module based on the attention mechanism, and process them with the following salient joint perception formula to obtain the joint state multimodal hybrid salient joint perception feature map; wherein the salient joint perception formula is:

[0064] ;

[0065] ;

[0066] ;

[0067] ;

[0068] ;

[0069] ;

[0070] ;

[0071] ;

[0072] ;

[0073] ;

[0074] ;

[0075] in, and They are respectively a multimodal fusion feature map of joint images under the constraint of the inspection information and a multimodal fusion feature map of joint images under the constraint of the medical record text information, For the reshape operation, and They are respectively the joint image multimodal fusion feature shape reshaping matrix under the inspection information constraint and the joint image multimodal fusion feature shape reshaping matrix under the medical record text information constraint, is the matrix product, and is the transposed matrix of the joint image multimodal fusion feature shape reshaping matrix under the inspection information constraint and the joint image multimodal fusion feature shape reshaping matrix under the medical record text information constraint, and are the scales of the first correlation feature matrix and the second correlation feature matrix, respectively. for function, and is the joint image dependency matrix under the inspection and examination-medical record text constraint and the joint image dependency matrix under the medical record text-inspection and examination constraint, For pruning, and are the pruned inspection-medical record text constraint joint image dependency matrix and the pruned medical record text-inspection constraint joint image dependency matrix, respectively. and The dependency relationship optimizes the joint image multimodal fusion feature matrix under the constraint of medical record text information and the dependency relationship optimizes the joint image multimodal fusion feature matrix under the constraint of inspection information, respectively. and They are respectively the multimodal fusion feature map of joint images under the constraints of the optimized inspection information and the multimodal fusion feature map of joint images under the constraints of the optimized medical record text information. and is the weight hyperparameter, It is the multimodal mixed salient joint perception feature map of the joint state.

[0076] In an embodiment of the present application, the auxiliary diagnosis result generation module 170 is used to obtain an auxiliary diagnosis result based on the joint state multimodal mixed significant joint perception feature, and the auxiliary diagnosis result is used to indicate whether rheumatoid arthritis is suffered from. Specifically, in an embodiment of the present application, the auxiliary diagnosis result generation module is used to: input the joint state multimodal mixed significant joint perception feature map into the RA auxiliary diagnosis module based on the classifier to obtain the auxiliary diagnosis result, and the auxiliary diagnosis result is used to indicate whether rheumatoid arthritis is suffered from. That is, the joint state multimodal mixed significant joint perception feature obtained by performing significant perception fusion using the joint image multimodal fusion feature map under the inspection information constraint and the joint image multimodal fusion feature map under the medical record text information constraint is classified and processed, so as to automatically determine whether rheumatoid arthritis is suffered from. In this way, multiple types of clinical data can be processed simultaneously, and the fusion of multidimensional data is realized, so that the patient's condition can be evaluated more comprehensively. And the deep learning technology is used to automatically encode and extract features of various clinical data, which reduces the dependence on the doctor's personal experience and judgment, and improves the automation level of rheumatoid arthritis diagnosis and treatment assistance.

[0077] In particular, considering that the multimodal fusion feature map of joint images under the constraint of the inspection and examination information and the multimodal fusion feature map of joint images under the constraint of the medical record text information respectively represent the image semantic features of joint image data through the cross-domain heterogeneous multimodal encoding features of the encoded text semantic features of the inspection and examination information and the medical record text information, when further performing the significant joint perception based on the attention mechanism, the encoding semantic differences will also be constrained to cause the significant joint perception aggregation alignment conflict based on the multimodal semantic attention weights, thereby causing the loss of aggregation key information, affecting the expression effect of the multimodal mixed significant joint perception feature map of the joint state.

[0078] Based on this, in a preferred example, the joint state multimodal mixed significant joint perception feature map is input into a classifier-based RA auxiliary diagnosis module to obtain an auxiliary diagnosis result, including the steps of: clustering all feature values ​​of the joint state multimodal mixed significant joint perception feature map based on the distance between feature values, such as the difference absolute value distance, and determining the ratio of the number of features of the cluster features to the number of features of the joint state multimodal mixed significant joint perception feature map; arranging the cluster features into a joint state multimodal mixed significant joint perception cluster vector, and dividing the second norm of the joint state multimodal mixed significant joint perception cluster vector by the second norm of the joint state multimodal mixed significant joint perception feature vector obtained after the joint state multimodal mixed significant joint perception feature map is expanded to obtain a first joint state multimodal mixed significant joint perception clustering related weight value; dividing a first power value of a norm of the joint state multimodal mixed significant joint perception cluster vector exponentially by the ratio value by a second power value of a norm of the joint state multimodal mixed significant joint perception feature vector exponentially by the ratio value to obtain a second joint state multimodal mixed significant joint perception cluster. Cluster-related weight value; for each eigenvalue of the joint state multimodal mixed significant joint perception clustering vector, multiply it by the inverse of the difference between the first joint state multimodal mixed significant joint perception clustering-related weight value and the second joint state multimodal mixed significant joint perception clustering-related weight value to obtain the optimized eigenvalue of the joint state multimodal mixed significant joint perception clustering vector; for each eigenvalue outside the cluster in the joint state multimodal mixed significant joint perception feature map, multiply it by the difference between the first joint state multimodal mixed significant joint perception clustering-related weight value and the second joint state multimodal mixed significant joint perception clustering-related weight value The inverse of the sum of the relevant weight values ​​of the second joint state multimodal mixture significant joint perception clustering is obtained to obtain the optimized out-of-class eigenvalue of the joint state multimodal mixture significant joint perception feature map; the optimized eigenvalue of the joint state multimodal mixture significant joint perception clustering vector and the optimized out-of-class eigenvalue of the joint state multimodal mixture significant joint perception feature map are combined to form an optimized joint state multimodal mixture significant joint perception feature map; the optimized joint state multimodal mixture significant joint perception feature map is input into the classifier-based RA auxiliary diagnosis module to obtain the auxiliary diagnosis result.

[0079] Among them, the optimization expression of the joint state multimodal mixed significant joint perception feature map is:

[0080] ;

[0081] in, For each eigenvalue of the joint state multimodal mixture significant joint perception clustering vector, is the multi-modal mixed significant joint perception feature vector of the joint state, is the joint state multimodal mixture significant joint perception clustering vector, is the number of features of the joint state multimodal mixed significant joint perception feature vector, is the number of features of the joint state multimodal mixed significant joint perception clustering vector, is the ratio of the number of features of the joint state multimodal mixed significant joint perception clustering vector to the number of features of the joint state multimodal mixed significant joint perception feature vector, represents the cluster feature set corresponding to the multi-modal mixed significant joint perception cluster vector of the joint state, and Represents the binary norm and the uninorm of the vector respectively. Power, is each eigenvalue of the optimized joint-state multimodal mixed salient joint perceptual feature map.

[0082] In order to avoid the loss of key suffix semantic information relative to the original feature set as a whole due to aggregation conflict because the joint state multimodal mixed significant joint perception feature map is based on aggregated features, the clustering ratio of the number of features of the joint state multimodal mixed significant joint perception clustering vector relative to the number of features of the joint state multimodal mixed significant joint perception feature map is used as a decision function to perform an adversarial decision on the set absolute representation of the joint state multimodal mixed significant joint perception clustering vector and the one-norm of the joint state multimodal mixed significant joint perception feature vector, and the joint state multimodal mixed significant joint perception clustering vector and the joint state multimodal mixed significant joint perception feature vector are respectively compared with the joint state multimodal mixed significant joint perception clustering vector and the joint state multimodal mixed significant joint perception feature map. The clustering intrinsic conflict representation of the two norm of the joint state multimodal mixed significant joint perception feature vector is positively and negatively interacted to construct the optimized joint state multimodal mixed significant joint perception feature map based on the aggregated features and the solid alignment guardrail of the original feature set as a whole, so as to achieve the mitigation of the harmful information loss intention of the optimized joint state multimodal mixed significant joint perception feature map based on the aggregated risk transferability, and improve the expression effect of the optimized joint state multimodal mixed significant joint perception feature map, so as to improve the accuracy of the auxiliary diagnosis result obtained by the joint state multimodal mixed significant joint perception feature map input based on the classifier-based RA auxiliary diagnosis module. In this way, multiple types of clinical data can be processed at the same time, and the fusion of multi-dimensional data can be realized, so that the patient's condition can be evaluated more comprehensively. In addition, the deep learning technology is used to automatically encode and extract features of various clinical data, which reduces the dependence on the doctor's personal experience and judgment, and improves the automation level of rheumatoid arthritis diagnosis and treatment assistance.

[0083] In summary, the rheumatoid arthritis diagnosis and treatment assistance system 100 based on the embodiment of the present application is explained, which uses information processing and image analysis technology based on deep learning to semantically encode the test information and medical record text information of the patient object, and extract features from the joint image data of the patient object, so as to automatically judge whether the patient suffers from rheumatoid arthritis based on the significant joint perception features between the joint image fusion features under the constraints of the test information and the joint image fusion features under the constraints of the medical record text information. In this way, by processing various types of clinical data, the patient's condition can be automatically and comprehensively evaluated, while reducing the dependence on the doctor's personal experience and judgment, which is conducive to improving the automation level of rheumatoid arthritis diagnosis and treatment assistance.

[0084] As described above, the rheumatoid arthritis diagnosis and treatment assistance system 100 according to the embodiment of the present application can be implemented in various terminal devices, such as a server for rheumatoid arthritis diagnosis and treatment assistance. In one example, the rheumatoid arthritis diagnosis and treatment assistance system 100 according to the embodiment of the present application can be integrated into the terminal device as a software module and / or a hardware module. For example, the rheumatoid arthritis diagnosis and treatment assistance system 100 can be a software module in the operating system of the terminal device, or it can be an application developed for the terminal device; of course, the rheumatoid arthritis diagnosis and treatment assistance system 100 can also be one of the many hardware modules of the terminal device.

[0085] Alternatively, in another example, the rheumatoid arthritis diagnosis and treatment assistance system 100 and the terminal device may also be separate devices, and the rheumatoid arthritis diagnosis and treatment assistance system 100 may be connected to the terminal device via a wired and / or wireless network and transmit interactive information in accordance with an agreed data format.

[0086] Figure 3 Flow chart of the auxiliary method for diagnosis and treatment of rheumatoid arthritis according to an embodiment of the present application. Figure 3As shown, in the rheumatoid arthritis diagnosis and treatment auxiliary method, it includes: S110, obtaining the test information, medical record text information and joint image data of the patient object; S120, semantically encoding the test information and the medical record text information to obtain the test information semantic encoding feature vector and the medical record text information semantic encoding feature vector; S130, inputting the joint image data into the joint image feature extractor to obtain the joint image feature map; S140, inputting the test information semantic encoding feature vector and the joint image feature map into the cross-domain heterogeneous Transformer multimodal encoding module to obtain the joint image multimodal fusion feature map under the test information constraint; S150, inputting the disease The semantic encoding feature vector of the medical record text information and the joint image feature map are input into the cross-domain heterogeneous Transformer multimodal encoding module to obtain a multimodal fusion feature map of joint images under the constraint of the medical record text information; S160, the multimodal fusion feature map of joint images under the constraint of the inspection and examination information and the multimodal fusion feature map of joint images under the constraint of the medical record text information are input into the salient joint perception module based on the attention mechanism to obtain a joint state multimodal mixed salient joint perception feature map as a joint state multimodal mixed salient joint perception feature; S170, based on the joint state multimodal mixed salient joint perception feature, an auxiliary diagnosis result is obtained, and the auxiliary diagnosis result is used to indicate whether the patient suffers from rheumatoid arthritis.

[0087] Here, those skilled in the art will appreciate that the specific operations of each step in the above-mentioned rheumatoid arthritis diagnosis and treatment auxiliary method have been described in the above reference. Figure 1 to Figure 2 It has been described in detail in the description of the rheumatoid arthritis diagnosis and treatment assistance system, and therefore, its repeated description will be omitted.

[0088] In summary, the rheumatoid arthritis diagnosis and treatment assistance method based on the embodiment of the present application is explained, which uses information processing and image analysis technology based on deep learning to semantically encode the test information and medical record text information of the patient object, and extract features from the joint image data of the patient object, so as to automatically judge whether the patient suffers from rheumatoid arthritis based on the significant joint perception features between the joint image fusion features under the constraints of the test information and the joint image fusion features under the constraints of the medical record text information. In this way, by processing various types of clinical data, the patient's condition can be automatically and comprehensively evaluated, while reducing the dependence on the doctor's personal experience and judgment, which is conducive to improving the automation level of rheumatoid arthritis diagnosis and treatment assistance.

[0089] It should also be noted that in the system and method of the present application, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present application.

[0090] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present application. Therefore, the present application is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features of the present application.

[0091] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present application to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A rheumatoid arthritis diagnosis and treatment auxiliary system, characterized in that: include: The patient data collection module is used to obtain the patient's test information, medical record text information and joint imaging data; A semantic coding module for test information and medical record text information, configured to perform semantic coding on the test information and the medical record text information to obtain a semantic coding feature vector for the test information and a semantic coding feature vector for the medical record text information; a joint image feature extraction module, configured to input the joint image data into a joint image feature extractor to obtain a joint image feature map; The inspection information guides the joint image multimodal fusion module, which is used to input the inspection information semantic encoding feature vector and the joint image feature map into the cross-domain heterogeneous Transformer multimodal encoding module to obtain the joint image multimodal fusion feature map under the inspection information constraint; A medical record text information guided joint image multimodal fusion module, used to input the medical record text information semantic encoding feature vector and the joint image feature map into the cross-domain heterogeneous Transformer multimodal encoding module to obtain a joint image multimodal fusion feature map under the constraint of medical record text information; a joint state multimodal mixed saliency module, configured to input the joint image multimodal fusion feature map under the inspection information constraint and the joint image multimodal fusion feature map under the medical record text information constraint into a salient joint perception module based on an attention mechanism to obtain a joint state multimodal mixed salient joint perception feature map as a joint state multimodal mixed salient joint perception feature; An auxiliary diagnosis result generation module is used to obtain an auxiliary diagnosis result based on the multimodal mixed significant joint perception features of the joint state, and the auxiliary diagnosis result is used to indicate whether rheumatoid arthritis is caused.

2. The rheumatoid arthritis diagnosis and treatment support system according to claim 1, characterized in that: The joint image feature extraction module is used to: input the joint image data into a joint image feature extractor based on a void convolutional neural network model to obtain the joint image feature map.

3. The rheumatoid arthritis diagnosis and treatment support system according to claim 2, characterized in that: The inspection information guides the joint image multimodal fusion module, including: An inspection and testing information feature linear transformation and reshape reshaping unit is used to perform linear transformation and feature reshape on the inspection and testing information semantic coding feature vector to obtain an inspection and testing information semantic coding modulation matrix; a joint image feature linear transformation unit, configured to perform a first linear transformation and a second linear transformation on the joint image feature map to obtain a first linear transformation joint image feature map and a second linear transformation joint image feature map; a linear transformation joint image feature shape reshaping unit, configured to perform feature shape reshaping on the first linear transformation joint image feature map and the second linear transformation joint image feature map to obtain a first linear transformation joint image feature matrix and a second linear transformation joint image feature matrix; an inspection information attention calculation unit, configured to use the first linear transformation joint image feature matrix and the second linear transformation joint image feature matrix as a key matrix and a value matrix and use the inspection information semantic coding modulation matrix as a query matrix, input the query matrix, the key matrix and the value matrix into a Transformer structure to obtain an inspection information attention query joint image matrix; A joint image matrix reshape convolution encoding unit is used to reshape and perform point convolution encoding on the inspection information attention query joint image matrix to obtain an inspection information attention query joint image feature map; A joint image multimodal fusion feature generation unit under inspection and examination information constraints, configured to perform position point multiplication on the inspection and examination information attention query joint image feature map and the joint image feature map to obtain the joint image multimodal fusion feature map under inspection and examination information constraints; The inspection information attention calculation unit is used to: Calculating the matrix product between the transposed matrix of the inspection information semantic coding modulation matrix and the first linear transformation joint image feature matrix and inputting the matrix product into a softmax function to obtain an inspection information-joint image feature association matrix; The inspection information-joint image feature association matrix is ​​used as a weight matrix, and the matrix product between the weight matrix and the second linear transformation joint image feature matrix is ​​calculated to obtain the inspection information attention query joint image matrix.

4. The rheumatoid arthritis diagnosis and treatment support system according to claim 3, characterized in that: The inspection information feature linear transformation shape reshaping unit is used to: Performing matrix multiplication on the inspection information semantic coding feature vector and the inspection information modulation matrix, and then adding the result to the inspection information bias vector according to position to obtain the inspection information semantic coding modulation bias vector; Inputting the inspection information semantic coding modulation bias vector into a Sigmoid function for activation processing to obtain the inspection information semantic coding modulation activation vector; The inspection information semantic coding modulation activation vector is reshaped to obtain the inspection information semantic coding modulation matrix.

5. The rheumatoid arthritis diagnosis and treatment support system according to claim 4, characterized in that: The joint image feature linear transformation unit is used to: Multiplying each feature matrix along the channel dimension in the joint image feature map by the first joint image modulation matrix by position, and then adding the matrix by position to the first joint image bias matrix to obtain the first joint image bias feature map; Inputting the first joint image bias feature map into the Sigmoid function for activation processing to obtain the first linear transformation joint image feature map; Multiplying each feature matrix along the channel dimension in the joint image feature map by position with the second joint image modulation matrix and then adding the matrix by position with the second joint image bias matrix to obtain a second joint image bias feature map; The second joint image bias feature map is input into the Sigmoid function for activation processing to obtain the second linear transformation joint image feature map.

6. The rheumatoid arthritis diagnosis and treatment support system according to claim 5, characterized in that: The joint state multimodal hybrid saliency module includes: a joint image multimodal fusion feature shape reshaping unit, configured to perform feature shape reshaping on the joint image multimodal fusion feature map under the inspection and examination information constraint and the joint image multimodal fusion feature map under the medical record text information constraint to obtain a joint image multimodal fusion feature shape reshaping matrix under the inspection and examination information constraint and a joint image multimodal fusion feature shape reshaping matrix under the medical record text information constraint; The inspection and examination-medical record text channel-by-channel interactive perception unit is used to input the joint image multimodal fusion feature shape reshaping matrix under the inspection and examination information constraint and the joint image multimodal fusion feature shape reshaping matrix under the medical record text information constraint into the feature channel-by-channel interactive perception module to obtain the joint image dependency matrix under the inspection and examination-medical record text constraint and the joint image dependency matrix under the medical record text-inspection and examination constraint; a joint image dependency matrix pruning unit, configured to input the joint image dependency matrix under the inspection and examination-medical record text constraint and the joint image dependency matrix under the medical record text-inspection and examination constraint into a random dropout module to obtain a pruned joint image dependency matrix under the inspection and examination-medical record text constraint and a pruned joint image dependency matrix under the medical record text-inspection and examination constraint; A joint image multimodal fusion feature optimization unit under the constraint of medical record text information is used to perform matrix multiplication on the pruned inspection-joint image dependency matrix under the constraint of medical record text information and the joint image multimodal fusion feature shape reshaping matrix under the constraint of medical record text information to obtain a dependency-optimized joint image multimodal fusion feature matrix under the constraint of medical record text information; a joint image multimodal fusion feature optimization unit under inspection and examination information constraints, configured to perform matrix multiplication on the pruned medical record text-joint image dependency matrix under inspection and examination information constraints and the joint image multimodal fusion feature shape reshaping matrix under inspection and examination information constraints to obtain a dependency-optimized joint image multimodal fusion feature matrix under inspection and examination information constraints; The joint image multimodal fusion feature shape reshaping unit under information constraint is used to perform feature shape reshaping on the joint image multimodal fusion feature matrix under the dependency relationship optimization medical record text information constraint and the joint image multimodal fusion feature matrix under the dependency relationship optimization inspection and examination information constraint to obtain the joint image multimodal fusion feature map under the dependency relationship optimization medical record text information constraint and the joint image multimodal fusion feature map under the dependency relationship optimization inspection and examination information constraint; The joint state multimodal mixed significant joint perception feature generation unit is used to calculate the weighted sum of the joint image multimodal fusion feature map under the optimization inspection information constraint and the joint image multimodal fusion feature map under the optimization medical record text information constraint to obtain the joint state multimodal mixed significant joint perception feature map.

7. The rheumatoid arthritis diagnosis and treatment support system according to claim 6, characterized in that: The test and examination-medical record text channel-by-channel interactive perception unit is used to: Calculating the matrix product between the transposed matrix of the joint image multimodal fusion feature shape reshaping matrix under the constraint of the inspection information and the joint image multimodal fusion feature shape reshaping matrix under the constraint of the medical record text information to obtain a first correlation feature matrix; calculating a position-wise division of the first correlation feature matrix and the scale of the first correlation feature matrix to obtain a first correlation feature modulation matrix; Calculating the matrix product between the transposed matrix of the joint image multimodal fusion feature shape reshaping matrix under the constraint of the medical record text information and the joint image multimodal fusion feature shape reshaping matrix under the constraint of the inspection information to obtain a second correlation feature matrix; calculating a position-wise division of the second correlation feature matrix and the scale of the second correlation feature matrix to obtain a second correlation feature modulation matrix; The first correlation feature modulation matrix and the second correlation feature modulation matrix are respectively input into the softmax function to obtain the joint image dependency matrix under the inspection and examination-medical record text constraint and the joint image dependency matrix under the medical record text-inspection and examination constraint.

8. The rheumatoid arthritis diagnosis and treatment support system according to claim 7, characterized in that: The auxiliary diagnosis result generation module is used to: input the joint state multimodal mixed significant joint perception feature map into the classifier-based RA auxiliary diagnosis module to obtain the auxiliary diagnosis result, and the auxiliary diagnosis result is used to indicate whether the patient has rheumatoid arthritis.

9. A method for assisting the diagnosis and treatment of rheumatoid arthritis, characterized in that: include: Obtaining the patient's test and examination information, medical record text information, and joint imaging data; Performing semantic coding on the test information and the medical record text information to obtain a test information semantic coding feature vector and a medical record text information semantic coding feature vector; Inputting the joint image data into a joint image feature extractor to obtain a joint image feature map; Inputting the inspection information semantic encoding feature vector and the joint image feature map into a cross-domain heterogeneous Transformer multimodal encoding module to obtain a joint image multimodal fusion feature map under the inspection information constraint; Inputting the semantic encoding feature vector of the medical record text information and the joint image feature map into the cross-domain heterogeneous Transformer multimodal encoding module to obtain a joint image multimodal fusion feature map under the constraint of the medical record text information; Inputting the multimodal fusion feature map of joint images under the constraint of the inspection information and the multimodal fusion feature map of joint images under the constraint of the medical record text information into a salient joint perception module based on the attention mechanism to obtain a joint state multimodal mixed salient joint perception feature map as a joint state multimodal mixed salient joint perception feature; Based on the multimodal mixed significant joint perception features of the joint status, an auxiliary diagnosis result is obtained, and the auxiliary diagnosis result is used to indicate whether rheumatoid arthritis is present.

10. The rheumatoid arthritis diagnosis and treatment auxiliary method according to claim 9, characterized in that: The joint image data is input into a joint image feature extractor to obtain a joint image feature map, including: inputting the joint image data into a joint image feature extractor based on a void convolutional neural network model to obtain the joint image feature map.