Medical image auxiliary diagnosis and treatment method and device, equipment and medium

Abstract

The invention discloses a medical image auxiliary diagnosis and treatment method, device and equipment and a medium, and relates to the technical field of image recognition, and the method comprises the steps: employing a local data set of a hospital user side to train a medical image recognition model, obtaining adapter parameters corresponding to the hospital user side, according to the method, adapter parameters corresponding to all hospital user sides are aggregated by using a federated average algorithm through a cloud server to obtain global model parameters, a trained medical image recognition model corresponding to the hospital user sides is determined according to the global model parameters and residual model parameters corresponding to the hospital user sides, and then medical image recognition is performed. According to the application, a federated learning mode is adopted, training can be carried out in a distributed environment, model training is carried out on the basis of protecting data privacy, a data barrier is broken, and the data value of each hospital user side is fully exerted.

Description

Technical Field

[0001] The present application relates to the field of medical image recognition technology, and in particular to a medical image-assisted diagnosis and treatment method, device, equipment and medium. Background Art

[0002] Image classification is a basic and key component of computer vision, and is closely related to machine learning, neural networks, and other aspects. With the advancement of methods over time, people are gradually expanding the methods of image classification and continuously achieving more satisfactory results. Today, image classification has been widely used in all walks of life, playing a positive role in medical treatment, industrial production, agricultural development, and other aspects. In 2012, the world witnessed the rebirth of convolutional neural networks (CNNs). CNNs have become a powerful tool for artificial intelligence to process images because of their ability to learn the robustness and high-level feature representation of images. They have also injected unlimited vitality into the field of computer vision, allowing people to continuously achieve satisfactory results in image recognition and other downstream tasks of computer vision. Among them, image recognition plays a key role in the field of computer vision and is an important basic part of various other downstream tasks.

[0003] With the development of computer technology, the computing power required for various downstream tasks has gradually increased with the computing power of computers. Machine learning, as a technology for analyzing and processing massive data, has been widely used in human society, but traditional machine learning and deep learning are based on training after all data are centralized, which also brings many limitations, the most obvious of which are the following three points: 1. Due to commercial factors or industry privacy between enterprises, there are data barriers between different industries and departments, resulting in the formation of data islands, and data cannot be shared safely. For example, in the field of medical health, for the sake of protecting patient privacy, relevant medical imaging data cannot be transmitted and shared, thus creating data barriers; 2. There is a risk of privacy leakage or exposure to malicious attackers when uploading data to the cloud for training, which is more fatal at a time when privacy concepts are becoming increasingly important; 3. Centralized training data models pose huge challenges to the computing and storage capabilities of enterprise hospital users, and the problem cannot be solved without equipment that meets the computing power requirements. Summary of the invention

[0004] The purpose of this application is to provide a medical image-assisted diagnosis and treatment method, device, equipment and medium, which can adopt a federated learning approach, can be trained in a distributed environment, perform model training on the basis of protecting data privacy, break down data barriers, and give full play to the data value of each hospital user end.

[0005] To achieve the above objectives, this application provides the following solutions:

[0006] In a first aspect, the present application provides a medical image-assisted diagnosis and treatment method, comprising:

[0007] Send the medical image recognition model to the user terminals of each hospital through the cloud server;

[0008] For each of the hospital user terminals, the local data set of the hospital user terminal is used to train the medical image recognition model, and local model parameters corresponding to the hospital user terminal are obtained; the local model parameters include adapter parameters and remaining model parameters; the remaining model parameters are model parameters other than the adapter parameters in the local model parameters corresponding to the hospital user terminal; the local data set includes a number of sample input information and a label corresponding to each of the sample input information; the sample input information includes a sample medical image and sample text information corresponding to the sample medical image; the medical image recognition model is an improved CLIP model; the improved CLIP model adds a visual adapter after the image encoder;

[0009] Aggregating the adapter parameters corresponding to all the hospital user terminals by using the federated average algorithm through the cloud server to obtain global model parameters;

[0010] For each hospital user terminal, the trained medical image recognition model corresponding to the hospital user terminal is determined through the hospital user terminal based on the global model parameters and the remaining model parameters corresponding to the hospital user terminal; the trained medical image recognition model is used to identify the target medical image of the hospital user terminal to obtain medical auxiliary information.

[0011] In a second aspect, the present application provides a medical image-assisted diagnosis and treatment device, including a hospital user terminal and a cloud server;

[0012] The cloud server is used to send the medical image recognition model to the user terminals of each hospital;

[0013] The hospital user terminal is used to: use the local data set of the hospital user terminal to train the medical image recognition model to obtain the local model parameters corresponding to the hospital user terminal; the local model parameters include adapter parameters and remaining model parameters; the remaining model parameters are the model parameters other than the adapter parameters in the local model parameters corresponding to the hospital user terminal; the local data set includes a number of sample input information and a label corresponding to each of the sample input information; the sample input information includes a sample medical image and sample text information corresponding to the sample medical image; the medical image recognition model is an improved CLIP model; the improved CLIP model adds a visual adapter after the image encoder;

[0014] The cloud server is also used to: aggregate the adapter parameters corresponding to all the hospital user terminals using a federated average algorithm to obtain global model parameters;

[0015] The hospital user terminal is also used to: determine the trained medical image recognition model corresponding to the hospital user terminal based on the global model parameters and the remaining model parameters corresponding to the hospital user terminal; the trained medical image recognition model is used to identify the target medical image of the hospital user terminal to obtain medical auxiliary information.

[0016] In a third aspect, the present application provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-mentioned medical image-assisted diagnosis and treatment method.

[0017] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which implements the above-mentioned medical image-assisted diagnosis and treatment method when executed by a processor.

[0018] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0019] The present application provides a method, device, equipment and medium for medical image-assisted diagnosis and treatment. The local data set of the hospital user end is used to train the medical image recognition model to obtain the adapter parameters corresponding to the hospital user end. The cloud server uses the federated average algorithm to aggregate the adapter parameters corresponding to all hospital user ends to obtain the global model parameters. According to the global model parameters and the remaining model parameters corresponding to the hospital user end, the trained medical image recognition model corresponding to the hospital user end is determined, and then medical image recognition is performed. The present application adopts the method of federated learning, which can be trained in a distributed environment, and the model training is performed on the basis of protecting data privacy, breaking down data barriers, and giving full play to the data value of each hospital user end. In addition, the medical image recognition task is performed by fine-tuning the CLIP model, giving full play to the powerful ability of the CLIP model to match images and texts, and can achieve satisfactory results. The improved CLIP model obtained by fine-tuning the adapter paradigm has good robustness and generalization ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 This is an application environment diagram of a medical image-assisted diagnosis and treatment method in one embodiment of the present application;

[0022] Figure 2 A schematic diagram of a process flow of a medical image-assisted diagnosis and treatment method provided in one embodiment of the present application;

[0023] Figure 3 A schematic diagram of the CLIP model structure provided in one embodiment of the present application;

[0024] Figure 4 A schematic diagram of the CLIP model reasoning process provided in one embodiment of the present application;

[0025] Figure 5 A schematic diagram of the structure of an improved CLIP model provided in one embodiment of the present application;

[0026] Figure 6 A schematic diagram of a federated learning training process provided for an embodiment of the present application;

[0027] Figure 7 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0029] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0030] Large models have achieved remarkable results in computer vision, natural language processing and other fields. With the continuous increase in the number of model parameters and the amount of training data, the text modeling perplexity of the language model is reduced in a predictable manner, and the performance on various natural language processing tasks continues to improve. The multimodal contrastive language-image pre-training large model (Contrastive Language-Image Pre-Training, CLIP) is a classic in the field of large models. The model contains an image encoder and a text encoder, and uses a large number of pictures and texts on the Internet for training using contrastive learning. Since the CLIP model can process image information and text information at the same time, many methods based on CLIP have been generated to solve multimodal tasks. The pre-trained large model contains sufficient training knowledge, can provide a rich tool set, and has strong generalization capabilities. On this basis, the modules in the pre-trained large model can be used to optimize image classification.

[0031] The core idea of ​​Federated Averaging (FedAvg) is to build a multi-party learning platform to coordinate learning between the user end and the hospital user end, and use a method similar to distributed learning to perform parallel model training on the local data of multiple user ends. After each user completes local training, the model parameters are transmitted to the cloud server, and the model parameters trained by each user end are aggregated in the cloud to obtain unified model parameters, and then the model parameters are sent to each user end for a new round of training and updating. Federated learning allows users to obtain more optimized models without leaving the local data, breaking data silos while protecting data privacy and giving full play to the value of data.

[0032] The medical image-assisted diagnosis and treatment method provided in the embodiments of the present application can be applied to Figure 1In the application environment shown. Among them, the cloud server communicates with the hospital user terminal through the network. The data storage system can store the data that the hospital user terminal and the cloud server need to process. The data storage system can be set up separately, integrated on the hospital user terminal, or placed on the cloud server or other hospital user terminals. The cloud server can send the medical image recognition model to the hospital user terminal. After the hospital user terminal receives the medical image recognition model, the hospital user terminal uses the local data set of the hospital user terminal to train the medical image recognition model to obtain the adapter parameters corresponding to the hospital user terminal. The hospital user terminal can feedback the obtained adapter parameters corresponding to the hospital user terminal for the medical image recognition model to the cloud server. The cloud server uses the federal average algorithm to aggregate the adapter parameters corresponding to all hospital user terminals to obtain the global model parameters. According to the global model parameters and the remaining model parameters corresponding to the hospital user terminal, the trained medical image recognition model corresponding to the hospital user terminal is determined, and then each hospital user terminal can use its own trained medical image recognition model to perform medical image recognition. In addition,

[0033] The cloud server may be, but is not limited to, various desktop computers, laptop computers, smart phones, and tablet computers. The hospital user terminal may be implemented as an independent hospital user terminal or a hospital user terminal cluster composed of multiple hospital user terminals.

[0034] In an exemplary embodiment, Figure 2 As shown, a medical image-assisted diagnosis and treatment method is provided. The method is executed by a computer device, and can be specifically executed by a cloud server and a hospital user terminal. In the embodiment of the present application, the method is applied to Figure 1 Taking the cloud server and hospital user end in FIG. 2 as an example, the method includes the following steps 201 to 204. Among them:

[0035] Step 201: Send the medical image recognition model to each hospital user terminal through the cloud server.

[0036] Step 202: For each of the hospital user terminals, the medical image recognition model is trained using the local data set of the hospital user terminal to obtain local model parameters corresponding to the hospital user terminal; the local model parameters include adapter parameters and remaining model parameters; the remaining model parameters are model parameters other than the adapter parameters in the local model parameters corresponding to the hospital user terminal; the local data set includes a number of sample input information and a label corresponding to each of the sample input information; the sample input information includes a sample medical image and sample text information corresponding to the sample medical image; the medical image recognition model is an improved CLIP model; the improved CLIP model adds a visual adapter after the image encoder.

[0037] Step 203: The cloud server aggregates the adapter parameters corresponding to all the hospital user terminals using a federated average algorithm to obtain global model parameters.

[0038] Step 204: For each hospital user terminal, a trained medical image recognition model corresponding to the hospital user terminal is determined through the hospital user terminal according to the global model parameters and the remaining model parameters corresponding to the hospital user terminal; the trained medical image recognition model is used to identify the target medical image of the hospital user terminal to obtain medical auxiliary information.

[0039] By implementing the above-mentioned steps 201 to 204 and adopting the method of federated learning, training can be performed in a distributed environment, model training can be performed on the basis of protecting data privacy, data barriers can be broken, and the data value of each hospital user end can be fully utilized. In addition, the medical image recognition task is performed by fine-tuning the CLIP model, which fully utilizes the powerful ability of the CLIP model to match images and texts, and can achieve satisfactory results. The improved CLIP model obtained by fine-tuning the adapter paradigm has good robustness and generalization ability. In addition, the present application can also only train the adapter parameters and freeze the CLIP model parameters, which can effectively reduce the training cost and communication overhead compared to the traditional method of fine-tuning global parameters. The present application adopts the paradigm of federated fine-tuning to design a CLIP multi-modal large model medical image assisted diagnosis and treatment method and device, which can be applied to medical imaging image recognition tasks and has important development significance.

[0040] The CLIP model is a large multimodal model for contrastive learning between vision and language proposed by OpenAI. It can understand images and texts at the same time and establish connections between the two. The core idea of ​​the CLIP model is to train the model through contrastive learning so that it can compare the similarities in the image and text embedding space. This training method enables the CLIP model to achieve amazing performance in multiple vision and language tasks, including image classification, image retrieval, zero-shot learning, etc. The CLIP model structure is as follows Figure 3 As shown in the figure, it consists of a text encoder (TextEncoder), an image encoder (ImageEncoder) and a contrastive learning (Contrastive Learning) module.

[0041] In the pre-training process, for a batch block containing n image-text pairs, the positive sample is n sample medical images and the text information corresponding to each sample medical image, and the negative sample is the pairwise combination of unpaired sample medical images and text information, with a total of (n 2-n). CLIP will first pass the sample medical image and text information through the text encoder and image encoder to obtain the text encoding vector and image encoding vector of the same dimension, each n 2 Then the cosine similarity between the text encoding vector and the image encoding vector is calculated. The objective function in the pre-training process is to make the positive sample (i.e. Figure 3 The blue part in the figure has a high similarity value, while the negative sample has a low similarity value. This contrastive learning method design enables the model to correctly align images and texts in the embedding space. Since the pre-training process learns at least a wide range of knowledge from a large number of image-text pairs, this powerful alignment capability can significantly improve the performance of the model when the CLIP model is deployed in downstream tasks.

[0042] When using the CLIP model for inference, all categories corresponding to the data set are passed through the text encoder to obtain the corresponding category encoding. This process does not directly take the category word as input, but borrows the idea of ​​prompt engineering and uses the prompt template to convert the category into a text sentence and then input it into the text encoder. The most commonly used prompt template is "a picture of {category}". For more sophisticated data sets, "a picture of {category}, this is a flower" or similar templates will also be used to ensure the fine-grained classification. When there are n categories, n corresponding encoding vectors will naturally be obtained. The image to be classified is then passed through the image encoder to obtain the corresponding image encoding vector, and the similarity between the vector and each text encoding vector is calculated. The category with the highest similarity is the final output category. Using the same prompt template, the output will be "a picture of a cat". The overall reasoning process is as follows: Figure 4 shown.

[0043] In another exemplary embodiment of the present application, in order to achieve the classification of medical images on the basis of protecting the privacy of medical information, reduce the cost of data transmission and storage, and give full play to the value of data, such as Figure 5 As shown, the present application uses an adapter method to fine-tune the CLIP model and deploys it to a distributed system for federated fine-tuning to obtain an improved CLIP model, which is named a medical image recognition model. The medical image recognition model includes a text encoder, an image encoder, a visual adapter, a first addition module, a second addition module and a recognition module.

[0044] Wherein, the image encoder is used to: encode the sample medical image to obtain image encoding features;

[0045] The text encoder is used to: encode the text information corresponding to the sample medical image to obtain text encoding features;

[0046] The visual adapter is used to: extract features from the image coding features to obtain dynamic capture features;

[0047] The first adding module is used to: add the dynamic capture feature and the image coding feature by using a residual connection method to obtain a first added feature;

[0048] The second addition module is used to: add the first addition feature and the text encoding feature by using a residual connection method to obtain a second addition feature;

[0049] The recognition module is used to: determine a category probability vector according to the second added feature; the category probability vector includes the category probability corresponding to each image category; and determining the final category according to the category probability vector specifically includes: determining the image category corresponding to the category probability with the largest value as the final category.

[0050] In one example of the present application, the Transformer model can be used as the structure of the text encoder, and the image encoder can use multiple versions of ResNet and ViT as the main structure, such as ResNet-50, ResNet-101, ViT-B / 16 and ViT-B / 32.

[0051] The adapter paradigm is used to fine-tune the CLIP model and apply it to the medical image classification task. While keeping the original CLIP backbone frozen during few-shot fine-tuning, only a small number of additional learnable linear bottleneck layers are added to the language and image branches of CLIP. The linear bottleneck layer refers to Figure 5 In the VisualAdapter, the visual adapter is composed of a multi-layer perceptron, which can be a two-layer perceptron in this application, and is not limited here. The visual adapter maps the output of the CLIP encoder once, adds the visual adapter to the CLIP model through a residual connection, and residually connects the visual adapter to the image encoder, thereby dynamically fusing the fine-tuned knowledge (dynamic capture features) with the knowledge from the CLIP backbone (image encoding features), effectively solving the overfitting problem caused by ordinary fine-tuning that only adds additional linear layers, and significantly improving the robustness of the CLIP adapter.

[0052] When the visual adapter consists of two layers of perceptrons, the visual adapter performs the following transformation on the output of the image encoder, i.e., the image encoding feature f:

[0053]

[0054] Among them, A v (·) represents the visual adapter, A v(f) represents the dynamic capture feature output by the visual adapter, ReLU represents the nonlinear activation function, T represents the matrix transpose, and the adapter parameters include the weight parameters corresponding to each layer of the perceptron in the multi-layer perceptron. and are the weight parameters of the two layers of perceptrons in the visual adapter.

[0055] The new knowledge captured by fine-tuning (dynamically captured features) is added to the image encoded features through residual connections:

[0056] f * =αA v (f) T +(1-α)f (2);

[0057] Among them, f * is the first added feature, α is the configuration parameter for adding the image encoding feature to the dynamic capture feature, that is, the configuration parameter for adding the original knowledge to the adapter knowledge, which is used as a “residual ratio” to help adjust the degree of retaining the original knowledge to obtain better performance.

[0058] After obtaining the first added feature f * After that, the first added feature f * The second added feature is added to the text encoding feature, and then the category probability vector is calculated according to the second added feature using the following formula (3), and the category of the image is predicted by selecting the class with the highest probability.

[0059]

[0060] In the formula, P represents the category probability vector, p i represents the category probability of the i-th image category, i = 1, 2, 3...K, K is the total number of image categories.

[0061] The above medical image recognition model is deployed in a distributed federated environment, that is, it is sent to the user end of each hospital. The adapter parameters are aggregated using the federated average algorithm, and the parameters are aggregated according to the data volume of each local user. The specific algorithm process is shown in Table 1.

[0062] Table 1 Federated learning algorithm process

[0063]

[0064] Where K is the total number of clients; k is the user index symbol, k = 1, 2, 3...k...K; w is the adapter parameter, the subscript represents the training round, w t is the adapter parameter cluster of round t, is the adapter parameter of the kth user in the t+1th round, w t+1is the adapter parameter cluster of the t+1th round, t means that each user will perform local training once in each round; ClientUpdate means client update; is the local user training set, is the batch size, b is the batch index symbol, That is, the samples selected from the training set for training according to the number of samples in few-shot learning; η is the learning rate of gradient descent training; represents the gradient, It represents the proportion of the training data volume of the kth user to the total data volume, that is, the weight of the user's parameters in each round of aggregation parameters.

[0065] In step 203, the adapter parameters corresponding to all the hospital user terminals are aggregated using a federated average algorithm to obtain global model parameters, which specifically include:

[0066] Initialize the adapter parameter cluster; the adapter parameter cluster includes the adapter parameters corresponding to all hospital user ends;

[0067] Local user training set According to the batch sample size The training set is divided into several batches, each batch of which includes several sample input information and labels corresponding to each of the sample input information; the sample input information includes sample medical images and sample text information corresponding to the sample medical images;

[0068] The hospital user terminal uses any use Update the adapter parameter cluster to obtain the adapter parameters corresponding to all the hospital user terminals; wherein, i means that each user needs to repeat the training sample multiple times when performing a local training.

[0069] Transmitting the adapter parameters corresponding to all the hospital user terminals to the cloud server;

[0070] The cloud server aggregates the adapter parameters corresponding to all the hospital user terminals, and transmits the aggregated global model parameters to each hospital user terminal. The aggregation formula is as follows:

[0071]

[0072] like Figure 6As shown in the figure, local model fine-tuning and cloud parameter aggregation are deployed on the camera and edge server respectively. The hospital client uses local medical imaging data to train local adapter parameters, saves the trained model parameters and uploads them to the cloud server. The cloud server aggregates the received adapter parameters and sends the aggregated global model parameters to each hospital client.

[0073] When training a medical image recognition model, the input is a sample medical image and the text information corresponding to the sample medical image, and the label is the classified image category. It should be noted that different text information will also have different image effects. For example, when training on X-ray related medical data sets, the performance of the text information "an X-rayphoto ofa{}." is significantly better than "aphoto ofa{}.".

[0074] The present application also provides an application scenario, which applies the above-mentioned medical image-assisted diagnosis and treatment method. Specifically: The medical image-assisted diagnosis and treatment method provided in this embodiment can be applied in the medical image-assisted diagnosis and treatment scenario. The medical image-assisted diagnosis and treatment scenario includes a model construction link, a model training link and a medical image recognition link; the medical image recognition model enters the model training link from the model construction link, obtains the corresponding trained medical image recognition model through human-machine collaboration, and enters the downstream medical image recognition link. The medical image-assisted diagnosis and treatment method provided in this embodiment belongs to the model training link. Specifically, in the model training link process for the video, the hospital user end uses the local data set of the hospital user end to train the medical image recognition model, and obtains the adapter parameters corresponding to the hospital user end. The hospital user end can feedback the obtained adapter parameters corresponding to the hospital user end for the medical image recognition model to the cloud server, and the cloud server uses the federated average algorithm to aggregate the adapter parameters corresponding to all hospital user ends to obtain the global model parameters, and determines the trained medical image recognition model corresponding to the hospital user end according to the global model parameters and the remaining model parameters corresponding to the hospital user end, and then each hospital user end can use its own trained medical image recognition model for medical image recognition.

[0075] Based on the same inventive concept, the embodiment of the present application also provides a medical image-assisted diagnosis and treatment device for implementing the medical image-assisted diagnosis and treatment method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more medical image-assisted diagnosis and treatment device embodiments provided below can refer to the limitations of the medical image-assisted diagnosis and treatment method above, and will not be repeated here.

[0076] In an exemplary embodiment, a medical image-assisted diagnosis and treatment device is provided, including a hospital user terminal and a cloud server.

[0077] Among them, the cloud server is used to send the medical image recognition model to the user terminals of each hospital;

[0078] The hospital user terminal is used to: use the local data set of the hospital user terminal to train the medical image recognition model to obtain the local model parameters corresponding to the hospital user terminal; the local model parameters include adapter parameters and remaining model parameters; the remaining model parameters are the model parameters other than the adapter parameters in the local model parameters corresponding to the hospital user terminal; the local data set includes a number of sample input information and a label corresponding to each of the sample input information; the sample input information includes a sample medical image and sample text information corresponding to the sample medical image; the medical image recognition model is an improved CLIP model; the improved CLIP model adds a visual adapter after the image encoder;

[0079] The cloud server is also used to: aggregate the adapter parameters corresponding to all the hospital user terminals using a federated average algorithm to obtain global model parameters;

[0080] The hospital user terminal is also used to: determine the trained medical image recognition model corresponding to the hospital user terminal based on the global model parameters and the remaining model parameters corresponding to the hospital user terminal; the trained medical image recognition model is used to identify the target medical image of the hospital user terminal to obtain medical auxiliary information.

[0081] In an exemplary embodiment, a computer device is provided. The computer device may be a hospital user terminal or a cloud server. The internal structure diagram thereof may be as follows: Figure 7 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store medical image processing data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external cloud server through a network connection. When the computer program is executed by the processor, a medical image-assisted diagnosis and treatment method is implemented.

[0082] Those skilled in the art will understand that Figure 7The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0083] In an exemplary embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0084] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0085] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0086] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0087] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0088] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0089] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A medical image-assisted diagnosis and treatment method, characterized in that: The medical image-assisted diagnosis and treatment method comprises: Send the medical image recognition model to the user terminals of each hospital through the cloud server; For each of the hospital user terminals, the local data set of the hospital user terminal is used to train the medical image recognition model, and local model parameters corresponding to the hospital user terminal are obtained; the local model parameters include adapter parameters and remaining model parameters; the remaining model parameters are model parameters other than the adapter parameters in the local model parameters corresponding to the hospital user terminal; the local data set includes a number of sample input information and a label corresponding to each of the sample input information; the sample input information includes a sample medical image and sample text information corresponding to the sample medical image; the medical image recognition model is an improved CLIP model; the improved CLIP model adds a visual adapter after the image encoder; Aggregating the adapter parameters corresponding to all the hospital user terminals by using the federated average algorithm through the cloud server to obtain global model parameters; For each hospital user terminal, the trained medical image recognition model corresponding to the hospital user terminal is determined through the hospital user terminal based on the global model parameters and the remaining model parameters corresponding to the hospital user terminal; the trained medical image recognition model is used to identify the target medical image of the hospital user terminal to obtain medical auxiliary information.

2. The medical image-assisted diagnosis and treatment method according to claim 1, characterized in that: The medical image recognition model includes a text encoder, an image encoder, a visual adapter, a first addition module, a second addition module and a recognition module; Wherein, the image encoder is used to: encode the sample medical image to obtain image encoding features; The text encoder is used to: encode the text information corresponding to the sample medical image to obtain text encoding features; The visual adapter is used to: extract features from the image coding features to obtain dynamic capture features; The first adding module is used to: add the dynamic capture feature and the image coding feature by using a residual connection method to obtain a first added feature; The second addition module is used to: add the first addition feature and the text encoding feature by using a residual connection method to obtain a second addition feature; The recognition module is used to: determine a category probability vector according to the second added feature; the category probability vector includes a category probability corresponding to each image category; and determine a final category according to the category probability vector.

3. The medical image-assisted diagnosis and treatment method according to claim 2, characterized in that: Determining the final category according to the category probability vector specifically includes: The image category corresponding to the category probability with the largest value is determined as the final category.

4. The medical image-assisted diagnosis and treatment method according to claim 1, characterized in that: The visual adapter is composed of a multi-layer perceptron.

5. The medical image-assisted diagnosis and treatment method according to claim 4, characterized in that: The adapter parameters include weight parameters corresponding to each layer of the perceptron in the multi-layer perceptron.

6. A medical image-assisted diagnosis and treatment device, characterized in that: The medical image-assisted diagnosis and treatment device includes a hospital user terminal and a cloud server; The cloud server is used to send the medical image recognition model to the user terminals of each hospital; The hospital user terminal is used to: use the local data set of the hospital user terminal to train the medical image recognition model to obtain the local model parameters corresponding to the hospital user terminal; the local model parameters include adapter parameters and remaining model parameters; the remaining model parameters are the model parameters other than the adapter parameters in the local model parameters corresponding to the hospital user terminal; the local data set includes a number of sample input information and a label corresponding to each of the sample input information; the sample input information includes a sample medical image and sample text information corresponding to the sample medical image; the medical image recognition model is an improved CLIP model; the improved CLIP model adds a visual adapter after the image encoder; The cloud server is also used to: aggregate the adapter parameters corresponding to all the hospital user terminals using a federated average algorithm to obtain global model parameters; The hospital user terminal is also used to: determine the trained medical image recognition model corresponding to the hospital user terminal based on the global model parameters and the remaining model parameters corresponding to the hospital user terminal; the trained medical image recognition model is used to identify the target medical image of the hospital user terminal to obtain medical auxiliary information.

7. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the medical image-assisted diagnosis and treatment method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the medical image-assisted diagnosis and treatment method described in any one of claims 1 to 5 is implemented.

Images