Visual language large model-based livestock and poultry parasitic disease diagnosis and treatment method

Abstract

The invention discloses a livestock intestinal parasitic disease diagnosis and treatment method based on a visual language large model, which comprises the following steps of: training the visual language large model by utilizing a parasite image-text feature alignment data set, so that the large language model has visual capability on parasites; then, constructing a livestock and poultry parasitic disease diagnosis and treatment dialogue data set, and training the LoRA fine-tuning visual language large model by using the diagnosis and treatment dialogue data set to obtain a diagnosis and treatment dialogue model; utilizing the diagnosis and treatment dialogue model to realize livestock parasitic disease diagnosis and treatment. The diagnosis and treatment efficiency of parasitic diseases of livestock and poultry can be improved, the misdiagnosis rate is reduced, and the diagnosis and treatment process is optimized.

Description

Technical Field

[0001] The invention relates to the field of visual language large models and livestock and poultry breeding technology, and in particular to a method for diagnosing and treating livestock and poultry parasitic diseases based on the visual language large model. Background Art

[0002] Intestinal parasites are common infectious diseases in livestock and poultry farming. Most of them are latent infections and are easily overlooked during the farming process. Intestinal parasitic diseases in livestock and poultry are serious and harmful diseases that can cause blood vessel rupture, intestinal infection, and digestive dysfunction. Clinical symptoms include anemia, weight loss, dysentery, mental depression, etc., which pose a threat to the healthy development of the livestock and poultry farming industry.

[0003] In clinical practice, parasitic diseases are mainly diagnosed by manually judging whether pathological sections contain parasites. Microscopic images of pathological sections of livestock and poultry intestinal parasites have the following characteristics:

[0004] 1) High resolution;

[0005] 2) low contrast between foreground and background in the parasite region;

[0006] 3) In addition to parasites, there is liquid contamination in the image.

[0007] The current diagnosis and treatment method is manual testing by professional medical personnel, which has many disadvantages such as high professional requirements for personnel and narrow application. This seriously affects the effectiveness of the diagnosis and treatment of intestinal parasitic diseases in livestock and poultry, and is not conducive to the early and accurate prevention and control of epidemics in large-scale livestock and poultry breeding enterprises.

[0008] With the birth of Transformer, the network architecture in the field of natural language processing has undergone fundamental changes. The attention mechanism in Transformer has gradually become standard. In recent years, the application of Transformer in the field of computer vision has also seen explosive growth. For example, Vision Transformer uses the same Transformer architecture as in the field of NLP, creatively dividing the image into multiple parts and inputting them in the form of tokens. Through comprehensive pre-training on a large amount of data, ViT has demonstrated superior performance that exceeds ResNet. The introduction of Transformer has made CV and NLP more consistent in structure, laying the foundation for the design and implementation of multimodal models.

[0009] Given the success of ChatGPT and open source instruction-adjusted large language models in general fields, researchers have designed and implemented multiple chatbots focused on the biomedical field, such as ChatDoctor, Med-Alpaca, PMC-LLaMA, Clinical Camel, DoctorGLM, and Hua Tuo. These biomedical LLM chatbots are pre-trained based on open source language large models, and then fine-tuned using specific biomedical instruction-following datasets. Through this process, these large language models have shown extraordinary potential to provide effective assistance in a wide range of biomedical scenarios.

[0010] In summary, existing parasite detection methods can classify parasites, but cannot conduct multiple rounds of dialogue and detailed description of parasite types, shapes, colors, morphological characteristics, and diagnosis and treatment methods. Existing large visual language models perform poorly in the diagnosis and treatment of livestock and poultry parasites: (1) The accuracy of livestock and poultry parasite image classification is low; (2) It is difficult to evaluate parasitic diseases based on symptoms. Summary of the invention

[0011] In order to solve the deficiencies in the prior art, the present invention provides a method for diagnosing and treating intestinal parasitic diseases of livestock and poultry based on a large visual language model, aiming to improve the efficiency of diagnosis and treatment of intestinal parasites of livestock and poultry and overcome the problem that the existing large visual language model cannot adapt well to the data of livestock and poultry parasitic diseases.

[0012] The technical solution adopted by the present invention is as follows:

[0013] A method for diagnosing and treating intestinal parasitic diseases of livestock and poultry based on a large visual language model comprises the following steps:

[0014] S1, collect images containing parasites, write parasite description documents and questions based on the parasite information in the images, and build a parasite image-text feature alignment dataset based on parasite images, questions and answers;

[0015] S2, constructing a large visual language model, the network structure of which includes a parasite image fine-grained perception module, an image-text alignment module and a large language model; the input of the parasite image fine-grained perception module is the original parasite image, and the output is a fine-grained enhanced feature map; the image-text alignment module performs a linear transformation on the feature map output by the parasite image fine-grained perception module, and outputs a token with the same length as the text vector; the output of the image-text alignment module and the question are input into the large language model, and the large language model outputs the answer;

[0016] S3, uses the parasite image-text feature alignment dataset obtained in S1 to train the visual language model constructed in S2, so that the large language model has visual capabilities for parasites;

[0017] S4, constructing a dialogue dataset for diagnosis and treatment of intestinal parasitic diseases in livestock and poultry, wherein the dialogue dataset includes parasite description information, symptoms of parasitic diseases, and diagnosis and treatment plans for parasitic diseases;

[0018] S5, based on the visual language model trained in S3, adds LoRA layers to the attention mechanism Query, Value and fully connected weight matrix of the Transformer structure in the visual language model to form a LoRA fine-tuned visual language model; the LoRA fine-tuned visual language model is trained using the livestock and poultry intestinal parasitic disease diagnosis and treatment dialogue dataset constructed in S4 to obtain a diagnosis and treatment dialogue model;

[0019] S6, taking the parasite image and question as the input of the diagnosis and treatment dialogue model, using the diagnosis and treatment dialogue model to output the corresponding answer, to achieve the diagnosis and treatment of livestock and poultry parasitic diseases.

[0020] Furthermore, the working method of the parasite image fine-grained perception module is as follows:

[0021] Vision Transformer is used to extract features from the original parasite image. The extracted features generate Key and Value through the Linear layer. The heat map generated by the parasite classification model on the original parasite image is used for Embedding. The Embedding generates Query through the Linear layer. Finally, the extracted features and the generated Query, Key, and Value are input into the cross-attention layer to achieve fine-grained feature enhancement.

[0022] Furthermore, the image-text alignment module uses a two-layer MLP.

[0023] Furthermore, the large language model adopts glm-4-9b-chat, Qwen2-7B or Baichuan2-7B-Chat.

[0024] Furthermore, the LoRA layer includes two low-rank matrices A and B; the matrix A is used for dimensionality reduction, and the matrix B is used for dimensionality increase. During the forward propagation process, the original weight matrix W of the visual language large model is added to the update matrix BA obtained by multiplying the two low-rank matrices to form a new weight matrix. During back propagation, only the A and B matrices will be updated, freezing the pre-trained parameters of the visual language large model, that is, the original W remains unchanged.

[0025] Further, the diagnosis and treatment process of S6 is as follows:

[0026] S6.1, using SegmentAnything to segment the parasite image to be diagnosed and the given point or range box form Prompt to generate a parasite map;

[0027] S6.2, the parasite diagram and question are input into the diagnosis and treatment dialogue model, and the diagnosis and treatment dialogue model outputs the corresponding answer.

[0028] Furthermore, the segmentation model SegmentAnything in S1 segments the parasite image therefrom.

[0029] Furthermore, the parasite description document includes the type, shape, color, and morphological characteristics of the parasite, and is stored in a vector database in Excel format.

[0030] Furthermore, the method for writing questions in S1 is to manually write questions about the content of the parasite description document, and use a large language model to generate questions with similar meanings but different expressions.

[0031] Furthermore, the method for generating answers in S1 is as follows: results are retrieved from the vector database according to the question, and the results and the question are combined into a new prompt; and an answer is generated according to the prompt using a large language model.

[0032] Beneficial effects of the present invention:

[0033] (1) The method for diagnosing and treating intestinal parasitic diseases of livestock and poultry based on a visual language large model proposed in the present invention can quickly and accurately identify the types of parasites. The visual language large model can identify the types of parasites by analyzing the parasite images under a microscope, which is very important for the early diagnosis and treatment of the disease. It can reduce the workload of veterinary laboratories and speed up the diagnosis process, especially in remote areas or with limited resources, and improve the diagnosis efficiency through remote diagnosis support.

[0034] (2) Symptom-based disease classification: By understanding textual descriptions of livestock and poultry health conditions, the model can associate specific symptoms with known parasitic diseases, assisting veterinarians in making preliminary judgments.

[0035] (3) Disease treatment plan recommendations: Based on the parasite species and disease symptoms, the model can recommend effective treatment plans, including drug selection, administration method, and treatment course guidance. Provide personalized treatment recommendations, taking into account factors such as the animal's age, weight, and health status to optimize the treatment effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a flow chart of a method for diagnosing and treating livestock and poultry parasitic diseases based on a large visual language model according to the present invention.

[0037] Figure 2 This is a flow chart of the method for constructing parasite image-text feature alignment question-answering data of the present invention.

[0038] Figure 3 This is a schematic diagram of the visual language large model network of the present invention.

[0039] Figure 4 Schematic diagram of the image fine-grained perception module of the present invention.

[0040] Figure 5 This is a schematic diagram of the weight calculation of the LoRA training large language model of the present invention. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0042] like Figure 1 As shown, this embodiment provides a method for diagnosing and treating livestock and poultry parasitic diseases based on a large visual language model, which specifically includes the following steps:

[0043] S1, collect images containing parasites, write parasite description documents and questions based on the parasite information in the images, and build a parasite image-text feature alignment dataset based on parasite images, questions and answers; the specific steps include:

[0044] S1.1, obtain relevant images containing parasites, use the segmentation model SegmentAnything to segment the parasite images from them, and store them separately;

[0045] S1.2, labeling the segmented parasite images according to the parasite types;

[0046] S1.3, for the parasite image, manually write a parasite description document, wherein the parasite description document includes the type, shape, color, and morphological characteristics of the parasite, and stores it in a vector database in Excel format;

[0047] S1.4, manually write questions about the content of the parasite description document, and use the large language model to generate questions with similar meanings but different expressions;

[0048] S1.5, retrieve the results according to the question in the vector database, and combine the results and the question into a new prompt; use the large language model to generate an answer according to the prompt; the flow chart is as follows Figure 2 shown.

[0049] S1.6, randomly organize images, questions and answers into a parasite image-text feature alignment dataset according to the parasite species.

[0050] In S1.4 of the present invention, the manually written question is as follows:

[0051] a. What type of parasite is this?

[0052] b. What unique morphological characteristics does this parasite have?

[0053] c. Can you describe in detail the color and texture of the parasite?

[0054] d. Describe the specific appearance of this parasite under a microscope.

[0055] e. Can you describe the appearance of this parasite in one sentence?

[0056] f. Which morphological details of this parasite are most recognizable?

[0057] g. What morphological features does this parasite have?

[0058] h. How can morphological characteristics be used to distinguish this parasite from other types?

[0059] But it is not limited to the above 8 questions.

[0060] S2, constructed as Figure 3 The visual language large model shown in the figure has a network structure including a parasite image fine-grained perception module, an image-text alignment module and a large language model; wherein the parasite image fine-grained perception module receives as input the original parasite image and outputs a fine-grained enhanced feature map; the image-text alignment module performs a linear transformation on the feature map output by the parasite image fine-grained perception module and outputs a token with the same length as the text vector; the output of the image-text alignment module and the question are input into the large language model, and the large language model outputs the answer.

[0061] The structure of each module in the visual language large model network is as follows:

[0062] Figure 4 Schematic diagram of the parasite image fine-grained perception module. The working method of the parasite image fine-grained perception module is as follows:

[0063] First, the Vision Transformer with initialization parameters from CLIP-ViT-bigG is used to extract features from the original parasite image. The extracted features generate Key and Value through the Linear layer. The heat map generated by the parasite classification model on the original parasite image is used for Embedding. The Embedding generates Query through the Linear layer. Finally, the extracted features and the generated Query, Key, and Value are input into the cross-attention layer to achieve feature enhancement at a finer granularity.

[0064] In the present invention, the parasite classification model adopts a convolutional neural network, referring to the research on intestinal parasite egg recognition model based on convolutional neural network.

[0065] In the present invention, the image-text alignment module uses a two-layer MLP.

[0066] In the present invention, the large language model adopts glm-4-9b-chat, Qwen2-7B or Baichuan2-7B-Chat.

[0067] S3, using the parasite image-text feature alignment dataset obtained in S1 to train the visual language model constructed in S2, obtains the parasite image-text feature aligned visual language model, so that the large language model has visual ability for parasites. The specific training process is as follows:

[0068] The parasite image-text feature alignment dataset obtained by S1 is input into the visual language large model network constructed by S2 to train the visual language large model of parasite image-text feature alignment.

[0069] During the training process, the large language model parameters are frozen, and the fine-grained perception module and image-text alignment module are fully trained.

[0070] S4, build a dialogue dataset for the diagnosis and treatment of intestinal parasitic diseases in livestock and poultry. The dialogue dataset contains parasite description information, symptoms of parasitic diseases, and diagnosis and treatment plans for parasitic diseases. The construction process is as follows:

[0071] First, collect texts containing parasite species identification, descriptions of parasite shape, color, and morphological characteristics, livestock and poultry disease characteristics, and parasitic disease prevention and treatment methods; second, use Prompt to guide the large language model to generate conversation data in Json format; finally, randomly select half of the conversation data and fuse it with the parasite image-text feature alignment dataset constructed by S1, so that a conversation contains the image input by the user.

[0072] In the present invention, the prompt used consists of parasite-related text context and dialogue requirements, for example:

[0073] The description of parasitic diseases is as follows: "The prevention and control of coccidiosis mainly relies on drug treatment. There are a large number of effective drugs on the market, including aminopropyl, nicarbazine, coccidioidomycin, clomiphene citrate, chlorpheniramine, chlorpheniramine, chlorpheniramine, monensin, lasalomycin, salinomycin, narasin and meduramycin, etc. In addition, there are a variety of compound drug preparations. The chicken coccidiosis vaccines currently available on the market are all active vaccines, which are divided into two types: strong strain vaccines and weak strain vaccines. Since both vaccines contain live worm components, it is important to follow the correct operating procedures and instructions during use. As for genetically engineered vaccines, they are still in the laboratory research and development stage and have not yet been commercialized." Please design a dialogue based on the description of parasitic diseases provided. The dialogue requirements are as follows:

[0074] (1) Only generate questions with clear answers;

[0075] (2) The responses to the questions are based only on the context provided;

[0076] (3) The generated dialogue is given in JSON format;

[0077] The following is the format of the generated JSON conversation:

[0078] [{"id":"0",

[0079] "img":”,

[0080] "conyersations":[

[0081] {"from":"human",

[0082] "value":"What other means are there to prevent chicken coccidiosis besides drug treatment?"},

[0083] {"from":"llm",

[0084] "value":"Currently, in addition to the use of drugs, chicken coccidiosis can also be prevented through vaccination. The vaccines on the market are all active vaccines, including two types: strong strains and weak strains."},

[0085] {"from":"human",

[0086] "value":"What types of chicken coccidiosis vaccines are available on the market?"},

[0087] {"from":"llm",

[0088] "value":"All chicken coccidiosis vaccines available on the market are active vaccines, which are mainly divided into two categories: strong strain vaccines and weak strain vaccines."}

[0089] }]

[0090] S5, based on the visual language model of parasite image-text feature alignment trained in S3, adds LoRA layer to the attention mechanism Query, Value and fully connected weight matrix of the Transformer structure in the visual language model to form a LoRA fine-tuned visual language model. The LoRA fine-tuned visual language model is trained using the livestock and poultry intestinal parasitic disease diagnosis and treatment dialogue dataset obtained in S4 to obtain a diagnosis and treatment dialogue model.

[0091] More specifically, the LoRA layer includes two low-rank matrices, namely matrix A and matrix B; matrix A is used for dimensionality reduction, and matrix B is used for dimensionality increase. During the forward propagation process, the original weight matrix W of the visual language model is added to the update matrix BA obtained by multiplying these two low-rank matrices to form a new weight matrix. During the back propagation, only the A and B matrices will be updated, and the pre-trained parameters of the visual language model will be frozen, that is, the original W remains unchanged, the VisionTransformer parameters in the fine-grained perception module are frozen, and the parameters of the image-text alignment module are fully trained. The weight calculation method during the training of the large language model in LoRA is as follows: Figure 5 shown.

[0092] S6, taking the parasite image and question as the input of the diagnosis and treatment dialogue model, using the diagnosis and treatment dialogue model to output the corresponding answer, and realizing the diagnosis and treatment of livestock and poultry parasitic diseases. The specific diagnosis and treatment process is as follows:

[0093] S6.1, using SegmentAnything to segment the parasite image to be diagnosed and the given point or range box form Prompt to generate a parasite map;

[0094] S6.2, the parasite diagram and question are input into the diagnosis and treatment dialogue model, and the diagnosis and treatment dialogue model outputs the corresponding answer.

[0095] More preferably, when the livestock and poultry parasitic disease information changes or is updated, the diagnosis and treatment dialogue model needs to be updated to ensure the comprehensiveness of the diagnosis and treatment dialogue model; therefore, the new livestock and poultry parasitic disease information is used to update the model according to steps S1-5, and the updated model can refer to S6 to implement the diagnosis and treatment of livestock and poultry parasitic diseases.

[0096] More preferably, the above-mentioned retraining of the diagnosis and treatment dialogue model with new livestock and poultry parasitic disease information requires a long time; especially when the livestock and poultry parasitic disease information is updated quickly, the frequency of training and updating the model will also increase, which will cause a lot of time waste. In view of this problem, in order to reduce the frequency and time of subsequent updates and training of the diagnosis and treatment dialogue model, the present invention constructs an external knowledge base, inputs the new livestock and poultry parasitic disease information (refer to S1.3) into the external knowledge base (which is essentially a vector database), and in the new round of diagnosis and treatment process, the external knowledge base is searched according to the question to obtain the result, and the result and the question are combined into a new Prompt; the Prompt is input into the diagnosis and treatment dialogue model to generate an answer.

[0097] The above embodiments are only used to illustrate the design ideas and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, any equivalent changes or modifications made based on the principles and design ideas disclosed by the present invention are within the protection scope of the present invention.

Claims

1. A method for diagnosing and treating intestinal parasitic diseases of livestock and poultry based on a large visual language model, characterized in that: The following steps are involved: S1, collect images containing parasites, write parasite description documents and questions based on the parasite information in the images, and build a parasite image-text feature alignment dataset based on parasite images, questions and answers; S2, constructing a large visual language model, wherein the network structure of the large visual language model includes a parasite image fine-grained perception module, an image-text alignment module and a large language model; the input of the parasite image fine-grained perception module is the original parasite image, and the output is a fine-grained enhanced feature map; the image-text alignment module performs a linear transformation on the feature map output by the parasite image fine-grained perception module, and outputs a token with the same length as the text vector; The output of the image-text alignment module and the question are input into the large language model, and the large language model outputs the answer; S3, uses the parasite image-text feature alignment dataset obtained in S1 to train the visual language model constructed in S2, so that the large language model has visual capabilities for parasites; S4, constructing a dialogue dataset for diagnosis and treatment of intestinal parasitic diseases in livestock and poultry, wherein the dialogue dataset includes parasite description information, symptoms of parasitic diseases, and diagnosis and treatment plans for parasitic diseases; S5, based on the visual language model trained in S3, adds LoRA layers to the attention mechanism Query, Value and fully connected weight matrix of the Transformer structure in the visual language model to form a LoRA fine-tuned visual language model; the LoRA fine-tuned visual language model is trained using the livestock and poultry intestinal parasitic disease diagnosis and treatment dialogue dataset constructed in S4 to obtain a diagnosis and treatment dialogue model; S6, taking the parasite image and question as the input of the diagnosis and treatment dialogue model, using the diagnosis and treatment dialogue model to output the corresponding answer, to achieve the diagnosis and treatment of livestock and poultry parasitic diseases.

2. According to claim 1, a method for diagnosing and treating intestinal parasitic diseases of livestock and poultry based on a large visual language model is characterized in that: The working method of the parasite image fine-grained perception module is as follows: Vision Transformer is used to extract features from the original parasite image. The extracted features generate Key and Value through the Linear layer. The heat map generated by the parasite classification model on the original parasite image is used for Embedding. The Embedding generates Query through the Linear layer. Finally, the extracted features and the generated Query, Key, and Value are input into the cross-attention layer to achieve fine-grained feature enhancement.

3. The method for diagnosing and treating intestinal parasitic diseases of livestock and poultry based on a large visual language model according to claim 1 is characterized in that: The image-text alignment module uses a two-layer MLP.

4. The method for diagnosing and treating intestinal parasitic diseases of livestock and poultry based on a large visual language model according to claim 1, characterized in that: The large language model adopts glm-4-9b-chat, Qwen2-7B or Baichuan2-7B-Chat.

5. The method for diagnosing and treating intestinal parasitic diseases of livestock and poultry based on a large visual language model according to claim 1, characterized in that: The LoRA layer includes two low-rank matrices A and B; the matrix A is used for dimensionality reduction, and the matrix B is used for dimensionality increase. During the forward propagation process, the original weight matrix W of the visual language large model is added to the update matrix BA obtained by multiplying these two low-rank matrices to form a new weight matrix. During back propagation, only the A and B matrices will be updated, freezing the pre-trained parameters of the visual language large model, that is, the original W remains unchanged.

6. The method for diagnosing and treating intestinal parasitic diseases of livestock and poultry based on a large visual language model according to claim 1, characterized in that: The diagnosis and treatment process of S6 is as follows: S6.1, using SegmentAnything to segment the parasite image to be diagnosed and the given point or range box form Prompt to generate a parasite map; S6.2, the parasite diagram and question are input into the diagnosis and treatment dialogue model, and the diagnosis and treatment dialogue model outputs the corresponding answer.

7. The method for diagnosing and treating intestinal parasitic diseases of livestock and poultry based on a large visual language model according to claim 1, characterized in that: The segmentation model SegmentAnything in S1 segments the parasite image from it.

8. The method for diagnosing and treating intestinal parasitic diseases of livestock and poultry based on a large visual language model according to claim 1, characterized in that: The parasite description document includes the type, shape, color, and morphological characteristics of the parasite, and is stored in a vector database in Excel format.

9. The method for diagnosing and treating intestinal parasitic diseases of livestock and poultry based on a large visual language model according to claim 1, characterized in that: The method of writing questions in S1: manually write questions about the content of the parasite description document, and use the large language model to generate questions with similar meanings but different expressions.

10. The method for diagnosing and treating intestinal parasitic diseases of livestock and poultry based on a large visual language model according to claim 1, characterized in that: The method for generating answers in S1 is as follows: retrieve results from the vector database based on the question, and combine the results and the question into a new prompt; use the large language model to generate answers based on the prompt.

Images