Method, device and electronic equipment for identifying images of livestock and poultry diseases

Abstract

The present invention provides a method, device and electronic device for identifying images of livestock and poultry diseases, which are applied to the field of image recognition, wherein the method comprises: obtaining a target livestock and poultry image and a text description, wherein the text description is used to indicate the disease category of the target livestock and poultry image; segmenting and encoding the target livestock and poultry image through an image encoder of a multimodal contrastive language image pre-training model to obtain at least one image feature representation; reconstructing and encoding the text description through a text encoder of the multimodal contrastive language image pre-training model to obtain at least one text description feature; aligning the image feature representation with the text description feature based on the similarity of the multimodal contrastive language image pre-training model to obtain a candidate disease corresponding to the target part; the present invention can efficiently identify livestock and poultry disease images with a small amount of labeled data or zero samples.

Description

Technical Field

[0001] The present invention relates to the field of image recognition technology, and in particular to a method, device and electronic equipment for recognizing livestock and poultry disease images. Background Art

[0002] At present, in livestock and poultry farms, individual testing and disease diagnosis of livestock and poultry are mainly carried out by professional inspectors in the breeding houses. However, this method has disadvantages such as large workload, strong subjectivity, poor environment, and risk of cross-infection.

[0003] In related technologies, by analyzing the image data of livestock and poultry, artificial intelligence systems can assist in early warning of epidemics; however, since the types of epidemics are complex and diverse and the initial symptoms of livestock and poultry diseases are not obvious, it is difficult to comprehensively collect a large amount of data, making abnormal image samples of livestock and poultry diseases very scarce. Using visual information alone cannot well obtain disease characteristics and cannot meet the needs of accurately identifying livestock and poultry diseases in actual scenarios.

[0004] It can be seen that the image recognition method of livestock and poultry diseases in the related art has a technical problem of low recognition accuracy due to the lack of abnormal image samples of livestock and poultry diseases. Summary of the invention

[0005] The present invention provides a method, device and electronic device for recognizing images of livestock and poultry diseases, so as to solve the defect of low recognition accuracy of the image recognition method of livestock and poultry diseases in the prior art, and realize efficient recognition of livestock and poultry disease images with a small amount of labeled data or zero samples.

[0006] The present invention provides a method for identifying images of livestock and poultry epidemic diseases, comprising the following steps.

[0007] Obtain a target livestock image and a text description, wherein the text description is used to represent the disease category of the target livestock image; input the target livestock image and the text description into a multimodal contrastive language image pre-training model to obtain candidate diseases suffered by the target part of the livestock in the target livestock image output by the multimodal contrastive language image pre-training model, wherein the method includes: segmenting and encoding the target livestock image through an image encoder of the multimodal contrastive language image pre-training model to obtain at least one image feature representation, wherein the image feature representation is used to represent the properties of the target part of the target livestock image; reconstructing and encoding the text description through a text encoder of the multimodal contrastive language image pre-training model to obtain at least one text description feature, wherein the text description feature is used to represent the candidate diseases in the disease category; aligning the image feature representation with the text description feature based on the similarity of the multimodal contrastive language image pre-training model to obtain the candidate disease corresponding to the target part.

[0008] According to a method for identifying livestock and poultry disease images provided by the present invention, the target livestock and poultry images are segmented and encoded to obtain at least one image feature representation, including: instance segmentation of the target livestock and poultry images to obtain a set of part area images; mapping each part area image in the set of part area images to an embedding space to obtain at least one image vector representation; normalizing each of the image vector representations to obtain image feature representations corresponding to each of the part area images.

[0009] According to a method for identifying livestock and poultry disease images provided by the present invention, after performing instance segmentation on the target livestock and poultry image to obtain at least one part area image, the method further includes: identifying the species and part based on the part area image to determine the segmented part of the part area image; assigning preset labels to the segmented part to obtain a target part label of the part area image; and determining the range of candidate diseases of the target livestock and poultry image based on the target part label of the part area image.

[0010] According to a livestock and poultry disease image recognition method provided by the present invention, the text description is reconstructed and encoded to obtain at least one text description feature, including: determining at least one target disease category; reconstructing the text description based on a preset disease type feature corresponding to the target disease category to obtain a descriptor set corresponding to the target disease category; mapping each descriptor of the descriptor set to an embedding space to obtain at least one text vector representation; normalizing each of the text vector representations to obtain text description features corresponding to each of the descriptors.

[0011] According to a method for recognizing livestock and poultry disease images provided by the present invention, the alignment based on the similarity between the image feature representation and the text description feature to obtain the candidate disease corresponding to the target part includes: determining the matching score between the image feature representation and the target disease category based on a target scoring function, wherein the target scoring function includes: a similarity mean function, a similarity weighting function, and a grade assignment function; taking the target disease category with a matching score greater than a preset score threshold as the candidate disease of the target part corresponding to the image feature representation; wherein, when the target scoring function is the similarity mean function, determining the matching score between the image feature representation and the target disease category based on the target scoring function includes: determining the similarity between the text description features corresponding to each descriptor in a descriptor set corresponding to the image feature representation and at least one target disease category; and taking the average value of the similarities as the matching score between the image feature representation and the target disease category.

[0012] According to a method for identifying livestock and poultry disease images provided by the present invention, before obtaining the target livestock and poultry images and text descriptions, the method also includes: obtaining a set of livestock and poultry image label samples; when the number of samples in the livestock and poultry image label sample set is less than a preset sample number threshold, constructing a visual cache of the livestock and poultry image label sample set, which includes: extracting the visual embedding of each livestock and poultry image label sample in the livestock and poultry image label sample set through a visual encoder of a multimodal contrastive language image pre-training model; performing horizontal splicing based on the visual embedding to obtain an image information matrix, and using the image information matrix as the key part of the visual cache; determining the one-hot encoding of each livestock and poultry image label sample in the livestock and poultry image label sample set, and the classification label of each livestock and poultry image label sample in the livestock and poultry image label sample set; performing vertical connection based on the one-hot encoding, and performing horizontal linking based on the classification label to obtain the value part of the visual cache.

[0013] The present invention also provides an image recognition device for livestock and poultry diseases, comprising the following units: an acquisition unit, used to acquire a target livestock and poultry image and a text description, wherein the text description is used to indicate the disease category of the target livestock and poultry image; an identification unit, used to input the target livestock and poultry image and the text description into a multimodal contrast language image pre-training model, and obtain candidate diseases suffered by the target part of the livestock and poultry in the target livestock and poultry image output by the multimodal contrast language image pre-training model, wherein the device comprises: a segmentation module, used to segment the target livestock and poultry image through an image encoder of the multimodal contrast language image pre-training model; The multimodal contrastive language image pre-training model is used to reconstruct and encode the text description through the text encoder to obtain at least one text description feature, wherein the text description feature is used to represent the candidate disease in the disease category; the multimodal contrastive language image pre-training model is used to align the image feature representation with the text description feature based on the similarity to obtain the candidate disease corresponding to the target part.

[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for recognizing images of livestock and poultry epidemic diseases as described above is implemented.

[0015] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method for identifying images of livestock and poultry epidemic diseases as described in any one of the above is implemented.

[0016] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned methods for identifying images of livestock and poultry epidemic diseases.

[0017] The livestock and poultry disease image recognition method, device and electronic device provided by the present invention obtain target livestock and poultry images and text descriptions used to represent the disease categories of the target livestock and poultry images; segment and encode the target livestock and poultry images through an image encoder of a multimodal contrast language image pre-training model to obtain an image feature representation used to represent the target part of the target livestock and poultry image, thereby being able to segment each part area with diagnostic value from the image to be identified, and use the image in the area for encoding to obtain an image feature representation of the target part; because the types of livestock and poultry diseases belong to special concepts in a professional field, the corresponding samples are often in the pre-training data of the multimodal contrast language image pre-training model. This will affect the performance of zero-sample disease image recognition. The present invention uses the general text description of the disease category as auxiliary information to reconstruct the text features of each disease category, thereby providing the basic ability of zero-sample disease category image recognition and being able to recognize common concepts without training samples; through a multimodal comparative language image pre-training model, alignment is performed based on the similarity between the image feature representation and the text description feature, thereby constructing the feature association between the image modality and the text modality, providing the ability of zero-sample disease category image recognition; thereby solving the technical problem of low recognition accuracy due to the lack of abnormal image samples of livestock and poultry diseases in the image recognition method of the related art. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced one by one below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 It is a schematic flow chart of the livestock and poultry epidemic disease image recognition method provided by the present invention.

[0020] Figure 2 It is an overall flow chart of the livestock and poultry disease image recognition method provided by the present invention.

[0021] Figure 3 It is a schematic diagram of the framework of the livestock and poultry epidemic disease image recognition model provided by the present invention.

[0022] Figure 4 It is a structural schematic diagram of the livestock and poultry epidemic disease image recognition device provided by the present invention.

[0023] Figure 5 It is a schematic diagram of the physical structure of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] Accurate diagnosis of livestock and poultry diseases is one of the important tasks of modern animal husbandry. It is crucial to ensure the healthy development of animal husbandry, improve the quality and safety of livestock and poultry products, and protect the economic interests of farmers. It requires analyzing the appearance characteristics of livestock and poultry by processing the image or video data of livestock and poultry. Timely and accurate identification and diagnosis of livestock and poultry diseases is the key to taking effective prevention and control measures, reducing the risk of disease transmission, and protecting the healthy development of livestock and poultry farming. At present, in livestock and poultry farms, individual detection and disease diagnosis of livestock and poultry are mainly carried out by professional inspectors in the breeding house, but this method has the disadvantages of large workload, strong subjectivity, harsh environment, and risk of cross infection. With the continuous advancement of artificial intelligence technology, the use of deep learning algorithms to automatically learn features from images has become the mainstream, and has promoted the application of image recognition technology in the agricultural field, providing a new solution for the early detection and diagnosis of livestock and poultry diseases. By analyzing the image data of livestock and poultry, the artificial intelligence system can assist in early warning of diseases and improve the accuracy and efficiency of diagnosis.

[0026] Traditional image recognition methods rely on a large amount of labeled data and complex feature engineering. However, due to the complexity and diversity of diseases and the lack of obvious symptoms in the early stages of livestock and poultry diseases, it is difficult to collect a large amount of data in a comprehensive manner, making image samples of abnormal livestock and poultry diseases very rare. It is not possible to obtain disease features well using only visual information, which further increases the difficulty of training image-based disease recognition models. In recent years, the image-text pre-training model has attracted much attention due to its outstanding performance in the field of image recognition. Through large-scale image-text pair comparison learning, it has achieved cross-modal knowledge alignment and has a strong zero-sample classification ability, which solves the problem of scarce samples to a certain extent. The cross-modal image-text pre-training model has made many advances in the field of human medicine. However, since the names of livestock and poultry diseases belong to sub-fields, the model rarely sees relevant text and cannot align it with existing images. Therefore, the ordinary image-text pre-training model still cannot meet the needs of accurately identifying livestock and poultry diseases in actual scenarios.

[0027] In view of the above problems, the present invention proposes a method for identifying images of livestock and poultry diseases based on a pre-trained image-text model to introduce disease text descriptions. Closely related research directions include pre-trained image-text models, zero-shot learning, and few-shot learning. Among them, the pre-trained image-text model has become a key technology in the field of computer vision and artificial intelligence, especially showing significant performance in image recognition and classification tasks. As an advanced representative in this field, the multimodal contrastive language-image pre-training model (CLIP, Contrastive Language-Image Pre-training) model achieves effective alignment between images and texts through contrastive learning, thereby supporting advanced tasks such as zero-shot classification. The model is trained using large-scale image-text pairs, and the visual and text modalities are encoded through the Transformer architecture to achieve cross-modal knowledge fusion. In order to further improve the performance of the model in few-shot learning tasks, the present invention supports classification tasks by constructing a visual cache. The visual cache consists of visual features and category labels of a small number of labeled samples, and can quickly integrate sample knowledge of new tasks without additional training.

[0028] In addition, in order to optimize the text cache and improve the model's ability to recognize new categories, the introduction of category descriptors provides the model with richer text information. Descriptors are generated through automated methods, which can capture specific detail features in the image and improve the accuracy and interpretability of classification.

[0029] The present invention proposes a livestock and poultry disease image recognition method based on a graphic and text pre-training model. Aiming at the actual needs of disease monitoring in the livestock and poultry breeding industry, the method constructs a cross-modal recognition framework, which can utilize text descriptions of disease categories to achieve efficient recognition of livestock and poultry disease images with a small amount of labeled data or zero samples.

[0030] Optionally, the method for recognizing images of livestock and poultry disease symptoms of the embodiment of the present application can be executed by a server, or by a terminal device, or jointly by a server and a terminal device, taking the example of the method for recognizing images of livestock and poultry disease symptoms of the embodiment being executed by a server.

[0031] Figure 1 FIG. 1 is a flow chart of the method for identifying livestock and poultry disease images provided by the present invention, such as Figure 1 As shown, the method includes the following steps.

[0032] Step 101, obtaining a target livestock and poultry image and a text description, wherein the text description is used to indicate the disease category of the target livestock and poultry image;

[0033] In the embodiment of the present invention, the target livestock and poultry image may be a target image that needs to be identified with livestock and poultry diseases. Here, the target livestock and poultry image includes at least one target livestock and poultry.

[0034] For example, the image content of the target livestock and poultry image is "a rooster". At this time, the target livestock and poultry of the target livestock and poultry image is the "rooster" in the image content, and the target parts (segmented parts) of the target livestock and poultry can be the chicken head, lower limbs, wings, etc. The characteristics of these parts are the main basis for disease identification.

[0035] It should be noted that the target livestock and poultry image may be a plurality of images, and each target livestock and poultry image may include at least one target livestock and poultry.

[0036] In the embodiment of the present invention, the text description used to represent the disease category of the target livestock and poultry image may be input by the user according to the target livestock and poultry image, or may be a pre-set description text for the livestock and poultry disease category.

[0037] Step 102, input the target livestock image and text description into the multimodal comparative language image pre-training model, and obtain the candidate epidemic diseases suffered by the target part of the livestock in the target livestock image output by the multimodal comparative language image pre-training model.

[0038] It should be noted that the target livestock and poultry images and text descriptions can be input into the multimodal contrastive language image pre-training model at the same time, or can be input into the multimodal contrastive language image pre-training model one after another according to the actual application scenario.

[0039] For example, the text description can be pre-input into the multimodal comparative language image pre-training model so that the multimodal comparative language image pre-training model can determine multiple text description features corresponding to different disease categories, and in the actual application stage, only the target livestock and poultry images are input into the multimodal comparative language image pre-training model to obtain candidate diseases corresponding to the target parts in the target livestock and poultry images.

[0040] In an embodiment of the present invention, the recognition model framework is constructed based on the multimodal contrastive language-image pre-training model CLIP (Contrastive Language-Image Pre-training), and CLIP is used to construct feature associations between image modality and text modality, providing the ability of zero-sample disease image recognition.

[0041] Here, the multimodal contrastive language image pre-training model is used to jointly pre-train natural language tasks and image understanding tasks. CLIP learns the similarity between images and texts, enabling the model to automatically extract discriminative feature representations. During training, CLIP receives an image and a related text fragment as positive sample pairs, and also receives unrelated images and text fragments as negative sample pairs. The goal of the model is to maximize the similarity of positive sample pairs and minimize the similarity of negative sample pairs.

[0042] The CLIP model typically consists of two separate encoders: a convolutional neural network encoder for processing images and a Transformer encoder for processing text.

[0043] Through the embodiments of the present application, based on the multimodal contrast language image pre-training model, it is possible to use the text description of the disease to realize the image recognition of livestock and poultry disease symptoms, input the livestock and poultry images to be identified, and the model outputs the probability of each individual in the image suffering from various possible infectious diseases.

[0044] The above step 102 includes the following specific steps:

[0045] Step 1021, segmenting and encoding the target livestock and poultry image through the image encoder of the multimodal contrastive language image pre-training model to obtain at least one image feature representation.

[0046] The image feature representation is used to represent the properties of the target part of the target livestock and poultry image.

[0047] In an embodiment of the present invention, the image encoder part of the CLIP model is retained and used to extract visual features of the input image, and each part region with diagnostic value is segmented from the target livestock and poultry image using instance segmentation technology to obtain a part region image set, and each part region image in the part region image set is encoded to obtain an image feature representation for representing the characteristics of the target part of the target livestock and poultry image.

[0048] For example, a pre-trained instance segmentation model is used to segment each area with diagnostic value from the image to be identified, and the image within the area is used to identify the disease to eliminate the interference of complex and changeable background. At the same time, the species and part names of the area are identified to determine the range of candidate disease types.

[0049] Step 1022, reconstruct and encode the text description through the text encoder of the multimodal contrast language image pre-training model to obtain at least one text description feature.

[0050] Among them, the text description feature is used to represent the candidate diseases in the disease category.

[0051] In an embodiment of the present invention, the text encoder of CLIP is reconstructed, and the text description features of epidemic diseases are used as auxiliary information to reconstruct the text features of each disease, thereby solving the problem of missing professional vocabulary in CLIP pre-training data.

[0052] Since epidemic diseases belong to special concepts in professional fields, corresponding samples are often missing in the pre-training data of CLIP, which will affect the performance of zero-sample epidemic disease image recognition. The present invention uses the general text description features of epidemic disease symptoms as auxiliary information to reconstruct the text features of each type of symptom, thereby reconstructing the image-text matching relationship of epidemic disease symptom images and improving the performance of zero-sample recognition.

[0053] In an embodiment of the present invention, in the text encoder part of the multimodal contrastive language image pre-training model, a category descriptor is introduced to optimize the text end, and the text description features of the epidemic category are reconstructed and encoded by using the text description of the epidemic symptoms as auxiliary information.

[0054] Step 1023, through the multimodal contrast language image pre-training model, alignment is performed based on the similarity between the image feature representation and the text description feature to obtain the candidate epidemic disease corresponding to the target part.

[0055] In an embodiment of the present invention, by calculating the similarity between the image features of the part to be identified and the text description features of the candidate disease categories of the corresponding species and part, the probability that the livestock and poultry individuals in the image suffer from various candidate diseases is obtained.

[0056] For each area to be identified (i.e., target area), the image feature representation is first extracted. Based on the results of species identification and area identification in the area, the range of candidate diseases that can be diagnosed by the image feature representation is determined. Then, the similarity between the image feature representation of the area and the (reconstructed) text description features of the corresponding candidate diseases is calculated. Based on the similarity, the probability of the individual suffering from various candidate diseases is obtained and output, and the candidate diseases corresponding to the target area are obtained.

[0057] In some embodiments, information from multiple images is integrated to achieve more accurate disease identification based on feature fusion.

[0058] Through the embodiment of the present invention, CLIP is used to construct feature associations between image modalities and text modalities, providing the capability of zero-sample disease image recognition.

[0059] Through the above steps of the embodiment of the present invention, the target livestock and poultry image and text description are obtained, wherein the text description is used to represent the disease category of the target livestock and poultry image; the target livestock and poultry image and text description are input into the multimodal contrast language image pre-training model to obtain the candidate diseases suffered by the target part of the livestock and poultry in the target livestock and poultry image output by the multimodal contrast language image pre-training model, wherein the method includes: segmenting and encoding the target livestock and poultry image through the image encoder of the multimodal contrast language image pre-training model to obtain at least one image feature representation, wherein the image feature representation is used to represent the properties of the target part of the target livestock and poultry image; reconstructing and encoding the text description through the text encoder of the multimodal contrast language image pre-training model to obtain at least one text description feature, wherein the text description feature is used to represent the candidate diseases in the disease category; aligning the image feature representation and the text description feature based on the similarity of the multimodal contrast language image pre-training model to obtain the candidate diseases corresponding to the target part. Thus, the image recognition method of livestock and poultry diseases in the related art is solved, and there is a technical problem of low recognition accuracy due to the lack of abnormal image samples of livestock and poultry diseases.

[0060] According to a livestock and poultry disease image recognition method provided by the present invention, target livestock and poultry images are segmented and encoded to obtain at least one image feature representation, including:

[0061] Perform instance segmentation on the target livestock and poultry images to obtain a set of part area images;

[0062] Mapping each part region image in the part region image set to an embedding space to obtain at least one image vector representation;

[0063] Each image vector representation is normalized to obtain image feature representations corresponding to each part region image.

[0064] In the embodiment of the present invention, the instance segmentation technology is used to segment each part region with diagnostic value from the image to be identified, and obtain a part region image set.

[0065] Among them, instance segmentation technology is a further refinement based on object detection, which is used to separate the foreground and background of an object and achieve pixel-level object separation. In medical image processing, this technology can be applied to segment tumors, identify organs, locate lesions and other areas with diagnostic value.

[0066] In the embodiment of the present invention, the image encoder of the retained CLIP can adopt a ResNet or VisionTransformer structure. For example, the ViT-B / 32 network is used as the backbone network to divide the image into multiple patches to obtain a set of part region images, which are processed by the Transformer model to obtain a high-dimensional feature representation (i.e., image feature representation) of each part region image.

[0067] For example, the input part region image is mapped to the D-dimensional embedding space through the CLIP image encoder to obtain the image vector representation , and then normalized by L2 to obtain the unit vector (i.e., image feature representation), refer to the following formula (1):

[0068] (1)

[0069] in, Represents the unit vector obtained after normalization, that is, the image feature representation, represents the image vector representation, represents the L2 norm.

[0070] According to a method for identifying livestock and poultry disease images provided by the present invention, after performing instance segmentation on the target livestock and poultry image to obtain at least one part area image, the method further includes:

[0071] Identify the species and the part based on the part region image, and determine the segmented part of the part region image;

[0072] Assigning preset labels to the segmented parts to obtain target part labels of the part area image;

[0073] Based on the target part labels of the part area images, the range of candidate diseases of the target livestock and poultry images is determined.

[0074] In an embodiment of the present invention, instance segmentation technology is used to segment each area with diagnostic value from the image to be identified for disease identification. Select an instance segmentation model such as Mask R-CNN or YOLOv5, and pre-train the selected model on an existing data set to ensure that the model can identify key parts of livestock and poultry. After segmenting each independent object from the image to be identified through instance segmentation, identify the specific parts of each segmented object (such as chicken head, lower limbs, wings, etc.), and assign a predefined label to each segmented part. Based on these labels, the model will determine the range of candidate disease types and match them with the corresponding subsequent disease types and disease text descriptions.

[0075] In some embodiments, image features are first extracted for each area to be identified, and the range of candidate diseases that can be diagnosed is determined based on the results of species identification and site identification in the area. Then, the similarity between the regional image features and the corresponding reconstructed text description features of each candidate disease is calculated, and the probability of the individual suffering from various candidate diseases is obtained and output based on the similarity.

[0076] Through the embodiments of the present application, instance segmentation technology is used to segment each area with diagnostic value from the image to be identified for disease identification. At the same time, the species and part name of the area are identified to determine the range of candidate disease types.

[0077] According to a livestock and poultry disease image recognition method provided by the present invention, the text description is reconstructed and encoded to obtain at least one text description feature, including:

[0078] Identify at least one target disease category;

[0079] Based on the preset disease type features corresponding to the target disease category, the text description is reconstructed to obtain a descriptor set corresponding to the target disease category;

[0080] Mapping each descriptor of the descriptor set to an embedding space to obtain at least one text vector representation;

[0081] Each text vector representation is normalized to obtain the text description features corresponding to each descriptor.

[0082] In an embodiment of the present invention, based on the descriptor of the target disease category, text description information is introduced to assist in reconstructing the text description features. The text encoder of the CLIP model is an encoder based on the Transformer architecture similar to BERT. Its function is to convert the description text into a high-dimensional vector representation that can be compared with the image features, that is, a point in the embedding space aligned with the image embedding vector (that is, the image feature representation above), so that the model can recognize and classify images in zero-sample learning scenarios. Based on the text encoder of CLIP, the present invention introduces a category descriptor to optimize the representation of the text end, and defines a scoring function so that the model can more accurately recognize and classify images in zero-sample learning scenarios. The model can not only understand the text labels of each category, but also capture the association between richer semantic information and visual features.

[0083] Specifically, for each target disease category Define (i.e. reconstruct) a set of descriptors These descriptors are manually edited by domain experts based on category features. For example, the descriptors for healthy chickens may include "red comb", "upright comb", "yellow beak", etc. The text encoder is used to generate the descriptor set Each descriptor in Generate corresponding text embeddings to obtain the target disease category The text vector representation of .

[0084] Each text vector representation is normalized to obtain the text description features corresponding to each descriptor. For details, please refer to the following formula (2):

[0085] (2)

[0086] in, Represents the text description feature, represents the text vector representation, represents the L2 norm.

[0087] In an embodiment of the present invention, since the types of epidemic diseases belong to special concepts in a professional field, the corresponding samples are often missing in the pre-training data of CLIP, which will affect the performance of zero-sample epidemic disease image recognition. The present invention uses the general text description features of epidemic disease symptoms as auxiliary information to reconstruct the text features of each type of symptom, thereby reconstructing the image-text matching relationship of epidemic disease symptom images and improving the performance of zero-sample recognition.

[0088] Through the embodiments of the present application, the CLIP text encoder is reconstructed, and the text description features of epidemic diseases are used as auxiliary information to reconstruct the text features of each disease, thereby solving the problem of missing professional vocabulary in the CLIP pre-training data.

[0089] According to a method for recognizing livestock and poultry disease images provided by the present invention, candidate diseases corresponding to target parts are obtained by aligning image feature representations with text description features based on similarity, including:

[0090] Determine the matching score between the image feature representation and the target disease category based on the target scoring function, wherein the target scoring function includes: a similarity mean function, a similarity weighting function, and a grade assignment function;

[0091] The target disease category with a matching score greater than a preset score threshold is regarded as a candidate disease of the target part corresponding to the image feature representation;

[0092] Wherein, when the target scoring function is a similarity mean function, the matching score between the image feature representation and the target disease category is determined based on the target scoring function, including:

[0093] Determining the similarity between the text description features corresponding to each descriptor in the set of descriptors corresponding to the image feature representation and at least one target disease category;

[0094] The average value of the similarities is taken as the matching score between the image feature representation and the target disease category.

[0095] In getting After the text embedding (i.e., text description feature) of each descriptor in , each descriptor is calculated Image feature representation The probability of , the similarity between the image feature representation and each text embedding can be measured by cosine similarity using the same calculation method as in CLIP.

[0096] In order to further realize disease classification, a scoring function can be defined , used to calculate the input image feature representation Target disease category A set of descriptors The scoring function can be designed in many ways. The simplest method is to take the average of the similarities corresponding to all text descriptions of each type of epidemic; you can also use training to learn the weights of different feature descriptions and weight the similarities; or use graded scoring. Take the average definition The following formula (3):

[0097] (3)

[0098] in, represents the scoring function, Indicates the target disease category, represents image feature representation, Indicates the target disease category The number of descriptors, Represents a descriptor, Represents each descriptor Image feature representation The probability of .

[0099] Through the embodiments of the present invention, even if the features corresponding to certain descriptors are missing in the image, the model can still perform classification, and by normalizing the number of descriptors, different categories are allowed to have different numbers of descriptors. Finally, in order to make the decision-making process of the model transparent, the descriptors with the highest scores can be output, so that the user can understand why the model chooses a specific category, thereby enhancing interpretability.

[0100] According to a method for identifying livestock and poultry disease images provided by the present invention, before obtaining the target livestock and poultry image and text description, the method further includes:

[0101] Obtain a collection of labeled samples of livestock and poultry images;

[0102] When the number of samples in the livestock and poultry image labeled sample set is less than a preset sample number threshold, a visual cache of the livestock and poultry image labeled sample set is constructed, which includes:

[0103] Extracting the visual embedding of each livestock and poultry image labeled sample in the livestock and poultry image labeled sample set through the visual encoder of the multimodal contrastive language image pre-trained model;

[0104] Perform horizontal splicing based on visual embedding to obtain an image information matrix, and use the image information matrix as the key part of the visual cache;

[0105] Determine the unique hot encoding of each livestock and poultry image labeled sample in the livestock and poultry image labeled sample set, and the classification label of each livestock and poultry image labeled sample in the livestock and poultry image labeled sample set;

[0106] Vertical connections are made based on one-hot encoding, and horizontal connections are made based on classification labels to obtain the value part of the visual cache.

[0107] In the embodiment of the present invention, for each new image recognition task, the target distribution A small number of labeled training examples, where represents the training data set, Represents the training data set input data points, Indicates The labels or outputs of the input data points, represents the index of the data point in the training dataset, Represents the number of categories that need to be judged. A small number of examples representing each category are used to build a visual cache, which is a collection of key-value pairs containing key visual features and corresponding category labels (i.e., classification labels).

[0108] The visual encoder in the CLIP model is used to extract the visual embedding of each training sample, and these visual embeddings are horizontally concatenated into a matrix to form the key part of the visual cache, which can be referred to in the following formula (4):

[0109] (4)

[0110] in, represents the image information matrix, represents an element in a matrix, represents the index of the number of examples for each category, Represents the category number index, represents a real matrix, Represents dimension.

[0111] Next, we construct the corresponding cache value (i.e., visual cache), which is represented as a one-hot encoding , by vertically concatenating the one-hot encodings of K samples and then horizontally concatenating N categories by category label to form the value part of the visual cache.

[0112] In order to effectively integrate the knowledge of these small samples into the CLIP model, an affinity matrix is ​​introduced , which is achieved by calculating the similarity between the test image and the samples stored in the visual cache, which can be referred to in the following formula (5):

[0113] (5)

[0114] in, represents the affinity matrix, represents the pointwise exponential function, represents the parameter for adjusting the sharpness of similarity, represents the image information matrix, represents the transposed matrix, represents the test image (i.e. the target livestock image), represents a real matrix, Indicates the total number of categories (i.e. target disease categories), Represents the total number of samples (i.e., labeled livestock and poultry image samples).

[0115] Finally, the final classification log probability is output ,in, represents the classification log odds, Represents a scalar coefficient used to adjust The size of represents one-hot encoding ( ), represents the transposed matrix, The index of the target disease category. Represents the affinity matrix. This combination not only utilizes the rich prior knowledge encoded in the pre-trained model, but also incorporates the new information contained in a small number of training samples, thereby significantly improving the classification performance of the model under few-sample conditions.

[0116] The following describes an example of a livestock and poultry disease image recognition method provided by the present invention in a practical application scenario.

[0117] refer to Figure 2 , Figure 2 It is an overall flow chart of the livestock and poultry disease image recognition method provided by the present invention.

[0118] The present invention proposes a method for recognizing livestock and poultry disease images based on a pre-trained image-text model. Aiming at the actual needs of disease monitoring in the livestock and poultry breeding industry, the method constructs a cross-modal recognition framework, which can use the text description of disease symptoms to achieve efficient recognition of livestock and poultry disease images with a small amount of labeled data or zero samples. The method specifically includes the following steps:

[0119] Step 201, using the image-text pre-training model CLIP as a basis, retaining the structure of its image encoder, reconstructing the text encoder for the zero-sample recognition task, and building a visual cache for the few-sample recognition task.

[0120] In the present embodiment, the image encoder part of the CLIP model is retained and used to extract visual features of the input image. and text As input, they are mapped to the D-dimensional embedding space through independent encoders to obtain vector representation and , and then normalized by L2 to obtain the unit vector and , please refer to the following formula (6):

[0121] (6)

[0122] in, represents the image unit vector (i.e., image feature representation), represents the text unit vector (i.e., text description feature), represents the L2 norm, represents the image vector representation, Represents a text vector representation.

[0123] The training goal of CLIP is to align two embedding spaces learned for images and texts respectively. Given a batch of image-text pairs, CLIP can maximize the similarity of matching pairs while minimizing the similarity of all other unmatched pairs. The CLIP model uses cosine similarity to measure the degree of alignment between image and text embeddings. For details, please refer to the following formula (7):

[0124] (7)

[0125] in, Represents similarity calculation, represents the image unit vector, represents the text unit vector, represents the transposed matrix, represents the modulus of the image unit vector, Represents the magnitude of the text unit vector.

[0126] This similarity measure allows the model to be used in zero-shot learning scenarios by using a pre-computed weight matrix (text cache) The weight matrix is ​​composed of the text embedding concatenation of all classification labels and can be pre-computed and stored. In practical applications, given a new input image, the CLIP model determines the image category by calculating its similarity with each category embedding in the weight matrix.

[0127] In this method, the image encoder of the retained CLIP can adopt the ResNet or Vision Transformer structure. For example, the ViT-B / 32 network is used as the backbone network to divide the image into multiple small patches (patches) and process them through the Transformer model to obtain the high-dimensional feature representation of the image. The text encoder part of CLIP is reconstructed for zero-sample recognition tasks, and a visual cache is built in the few-sample recognition task to store and process the visual features of a small number of labeled samples, further improving the model's recognition ability.

[0128] Step 202, using instance segmentation technology to segment each area with diagnostic value from the image to be identified for disease identification.

[0129] In an embodiment of the present invention, instance segmentation technology is used to segment each area with diagnostic value from the image to be identified for disease identification. Select an instance segmentation model such as Mask R-CNN or YOLOv5, and pre-train the selected model on an existing data set to ensure that the model can identify key parts of livestock and poultry. After segmenting each independent object from the image to be identified through instance segmentation, identify the specific parts of each segmented object (such as chicken head, lower limbs, wings, etc.), and assign a predefined label to each segmented part. Based on these labels, the model will determine the range of candidate disease types and match them with the corresponding subsequent disease types and disease text descriptions.

[0130] Step 203, in the text encoder part, introduce category descriptors to optimize the text end, and reconstruct the text features of each disease category by using the text description features of the disease category as auxiliary information.

[0131] See also Figure 3 , Figure 3 It is a schematic diagram of the framework of the livestock and poultry epidemic disease image recognition model provided by the present invention.

[0132] refer to Figure 3, input the target livestock image (i.e., the image content is "rooster") and the disease category label (i.e., text description); perform instance segmentation on the target livestock image to obtain multiple part area images; input the multiple part area images into the image encoder to obtain visual embedding (i.e., image feature representation) ; Determine the similarity between the visual cache and the visual embedding.

[0133] Among them, a small number of sample images are input into the image encoder to obtain the key part of the visual cache; a small number of sample labels are one-hot encoded to obtain the value part of the visual cache.

[0134] Match the part name of the part area image with the disease category label to obtain the attribute description text; input the attribute description text into the text encoder to obtain the text cache, and determine the similarity between the text cache and the visual embedding;

[0135] The classification (logits) results of the target livestock and poultry images are determined based on the similarity between the visual cache and the visual embedding, and the similarity between the text cache and the visual embedding.

[0136] In an embodiment of the present invention, text description information is introduced as an auxiliary task to be completed by the category descriptor. The text encoder of the CLIP model is an encoder based on the Transformer architecture similar to BERT. Its function is to convert text into a high-dimensional vector representation that can be compared with image features, that is, a point in the embedding space aligned with the image embedding vector, so that the model can recognize and classify images in zero-sample learning scenarios. Based on the original CLIP text encoder, the present invention introduces category descriptors to optimize the representation of the text end, and defines a scoring function so that the model can more accurately recognize and classify images in zero-sample learning scenarios. The model can not only understand the text labels of each category, but also capture the association between richer semantic information and visual features. The modified text encoder is as follows: Figure 3 Shown in blue.

[0137] Specifically, this method is used for each target disease category Define (i.e. reconstruct) a set of descriptors These descriptors are manually edited by domain experts based on category features. For example, the descriptors for healthy chickens may include "red comb", "upright comb", "yellow beak", etc. The text encoder is used to generate the descriptor set Each descriptor in Generate the corresponding text embedding and get the category Optimized text representation of .

[0138] In getting After the text embedding (i.e., text description feature) of each descriptor in , each descriptor is calculated With image The probability of , the similarity between the image feature representation and each text embedding can be measured by cosine similarity using the same calculation method as in CLIP.

[0139] In order to further realize disease classification, a scoring function can be defined , used to calculate the input image feature representation Target disease category A set of descriptors The scoring function can be designed in many ways. The simplest method is to take the average of the similarities corresponding to all text descriptions of each type of epidemic; you can also use training to learn the weights of different feature descriptions and weight the similarities; or use graded scoring. Take the average definition The following formula (8):

[0140] (8)

[0141] in, represents the scoring function, Indicates the target disease category, represents image feature representation, Indicates the target disease category The number of descriptors, Represents a descriptor, Represents each descriptor Image feature representation The probability of .

[0142] Even if the image lacks features corresponding to some descriptors, the model can still perform classification, and by normalizing the number of descriptors, different categories can have different numbers of descriptors. In the end, in order to make the model's decision-making process transparent, the highest-scoring descriptors can be output, allowing users to understand why the model chooses a specific category, enhancing interpretability.

[0143] Continue to refer Figure 2 In step 204, in the case of few-sample recognition, a visual cache based on a small number of labeled samples is constructed to enhance the model's ability to extract visual features.

[0144] The present invention adopts the method proposed in Tip-adapter to build a visual cache based on a small number of labeled samples, which integrates key visual features and category labels. The method introduces affinity matrix To calculate the similarity between the test image and the samples in the visual cache, so as to update the classification log probability of the model ; Finally, the prior knowledge of the pre-trained model and a small amount of sample information are combined to significantly improve the classification performance of the model under the condition of few samples. In this way, the model can more effectively learn from limited samples and generalize to new image categories to achieve accurate image recognition. This method enhances the model's ability to extract visual features in the case of few-sample recognition by integrating the visual information of a small number of labeled samples.

[0145] Specifically for each new image recognition task, collect A small number of labeled training examples, where represents the training data set, Represents the training data set input data points, Indicates The labels or outputs of the input data points, represents the index of the data point in the training dataset, Represents the number of categories that need to be judged. Represents a small number of examples for each category. These samples are used to build a visual cache, which is a set of key-value pairs containing key visual features and corresponding category labels. The visual encoder in the CLIP model is used to extract the visual embedding of each training sample, and these visual embeddings are horizontally concatenated into a matrix to form the key part of the visual cache, which can be referred to in the following formula (9):

[0146] (9)

[0147] in, represents the image information matrix, represents an element in a matrix, represents the index of the number of examples for each category, Represents the category number index, represents a real matrix, Represents dimension.

[0148] Next, construct the corresponding cache value, representing it as a one-hot encoding , by vertically concatenating the one-hot encodings of K samples and then horizontally concatenating N categories by category label to form the value part of the visual cache.

[0149] In order to effectively integrate the knowledge of these small samples into the CLIP model, an affinity matrix is ​​introduced , which is achieved by calculating the similarity between the test image and the samples stored in the visual cache, which can be referred to in the following formula (10):

[0150] (10)

[0151] in, represents the affinity matrix, represents the pointwise exponential function, represents the parameter for adjusting the sharpness of similarity, represents the image information matrix, represents the transposed matrix, represents the test image (i.e. the target livestock image), represents a real matrix, Indicates the total number of categories (i.e. target disease categories), Represents the total number of samples (i.e., labeled livestock and poultry image samples).

[0152] Finally, the final classification log probability is output ,in, represents the classification log odds, Represents a scalar coefficient used to adjust The size of represents one-hot encoding ( ), represents the transposed matrix, represents the visual cache, Represents the affinity matrix. This combination not only utilizes the rich prior knowledge encoded in the pre-trained model, but also incorporates the new information contained in a small number of training samples, thereby significantly improving the classification performance of the model under few-sample conditions.

[0153] Here, Used to adjust the weight of visual cache and text cache.

[0154] Step 205, integrating information from multiple images to achieve more accurate disease identification based on feature fusion.

[0155] In an embodiment of the present invention, information from multiple images is integrated to achieve more accurate identification of epidemic diseases based on feature fusion. In actual scenarios, it may be necessary to input multiple images of an individual to comprehensively give an epidemic diagnosis. For the case of inputting multiple images, this method extracts features from each image, obtains a visual feature vector, and then uses feature fusion to integrate information from multiple images. Fusion methods include, but are not limited to: simple averaging, that is, directly averaging the visual feature vectors of all images to obtain a comprehensive feature vector; weighted averaging, that is, assigning different weights to the feature vectors of each image, and these weights can be based on the quality, relevance or other criteria of the image, and then calculating the weighted average feature vector according to the weights; attention mechanism, that is, using an attention network to learn the importance of each type of image features, and weighting the fusion features according to the attention score; deep learning network, that is, designing a neural network, taking the feature vectors of multiple images as input, and learning the optimal fusion strategy through the network.

[0156] The fused feature vector may contain a lot of redundant or irrelevant information, and feature selection technology can be used to identify and retain the most useful features. This method can make full use of the information in multiple images and improve the accuracy and reliability of livestock and poultry disease identification.

[0157] The present invention constructs a method for recognizing images of livestock and poultry diseases based on a multimodal pre-training model and zero-sample recognition technology, and uses the text description of disease-related symptoms to realize the recognition of disease-related symptom images under the condition of missing training data. The present invention uses the instance segmentation technology to segment each part area with diagnostic value from the image to be identified for disease identification, and at the same time identifies the species and part name of the area to determine the range of candidate disease types. The recognition model framework of the present invention is constructed based on the multimodal contrastive language image pre-training model (CLIP), and uses CLIP to construct the feature association between the image modality and the text modality, providing the ability of zero-sample symptom image recognition. The present invention reconstructs the CLIP text encoder, and uses the text description features of disease-related symptoms as auxiliary information to reconstruct the text features of various symptoms, thereby solving the problem of missing professional vocabulary in the CLIP pre-training data. The present invention obtains the probability of livestock and poultry individuals in the image suffering from various candidate diseases by calculating the similarity between the image features of the part to be identified and the candidate disease-related symptom features of the corresponding species and part. By building a visual cache based on a small number of labeled samples, the present invention can fine-tune the model with a small number of labeled samples to obtain more accurate disease features. The present invention can integrate multiple image information and achieve more accurate disease classification based on feature fusion.

[0158] The following is a description of the livestock and poultry disease-related symptom image recognition device provided by the present invention. The livestock and poultry disease-related symptom image recognition device described below and the livestock and poultry disease-related symptom image recognition method described above can be referenced to each other.

[0159] refer to Figure 4 , Figure 4 It is a structural schematic diagram of the livestock and poultry disease image recognition device provided by the present invention, which includes: an acquisition unit 401, a recognition unit 402, wherein the recognition unit 402 includes a segmentation module 4021, a reconstruction module 4022 and an alignment module 4023.

[0160] The acquisition unit 401 is used to acquire a target livestock and poultry image and a text description, wherein the text description is used to indicate the disease category of the target livestock and poultry image;

[0161] The recognition unit 402 is used to input the target livestock and poultry image and the text description into the multimodal contrast language image pre-training model to obtain the candidate diseases suffered by the target part of the livestock and poultry in the target livestock and poultry image output by the multimodal contrast language image pre-training model, which includes:

[0162] The segmentation module 4021 is used to segment and encode the target livestock image through the image encoder of the multimodal contrast language image pre-training model to obtain at least one image feature representation, wherein the image feature representation is used to represent the properties of the target part of the target livestock image;

[0163] A reconstruction module 4022 is used to reconstruct and encode the text description through the text encoder of the multimodal contrastive language image pre-training model to obtain at least one text description feature, wherein the text description feature is used to represent the candidate disease in the disease category;

[0164] The alignment module 4023 is used to align the image feature representation and the text description feature based on the similarity through the multimodal contrast language image pre-training model to obtain the candidate disease corresponding to the target site.

[0165] Specifically, the above-mentioned livestock and poultry disease image recognition device provided by the present invention can implement all the method steps implemented by the above-mentioned livestock and poultry disease image recognition method embodiment, and can achieve the same technical effects. The parts and beneficial effects of this embodiment that are the same as the method embodiment will not be described in detail here.

[0166] Figure 5 is a schematic diagram of the physical structure of the electronic device provided by the present invention, such as Figure 5As shown, the electronic device may include: a processor (processor) 510, a communication interface (Communications Interface) 520, a memory (memory) 530 and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other through the communication bus 540. The processor 510 can call the logic instructions in the memory 530 to execute the livestock and poultry disease image recognition method, which includes: obtaining a target livestock and poultry image and a text description, wherein the text description is used to represent the disease category of the target livestock and poultry image; inputting the target livestock and poultry image and the text description into the multimodal comparative language image pre-training model to obtain the candidate diseases suffered by the target part of the livestock and poultry in the target livestock and poultry image output by the multimodal comparative language image pre-training model, wherein the method includes: segmenting and encoding the target livestock and poultry image through the image encoder of the multimodal comparative language image pre-training model to obtain at least one image feature representation, wherein the image feature representation is used to represent the characteristics of the target part of the target livestock and poultry image; reconstructing and encoding the text description through the text encoder of the multimodal comparative language image pre-training model to obtain at least one text description feature, wherein the text description feature is used to represent the candidate diseases in the disease category; aligning the image feature representation and the text description feature based on the similarity of the multimodal comparative language image pre-training model to obtain the candidate diseases corresponding to the target part.

[0167] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0168] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the livestock and poultry disease classification image recognition method provided by the above methods. The method includes: obtaining a target livestock and poultry image and a text description, wherein the text description is used to indicate the disease category of the target livestock and poultry image; inputting the target livestock and poultry image and the text description into a multimodal contrast language image pre-training model to obtain the disease category of the target part of the livestock and poultry in the target livestock and poultry image output by the multimodal contrast language image pre-training model. Disease selection, including: segmenting and encoding the target livestock and poultry image through an image encoder of a multimodal contrastive language image pre-training model to obtain at least one image feature representation, wherein the image feature representation is used to represent the properties of a target part of the target livestock and poultry image; reconstructing and encoding the text description through a text encoder of a multimodal contrastive language image pre-training model to obtain at least one text description feature, wherein the text description feature is used to represent a candidate disease in the disease category; aligning the image feature representation with the text description feature based on the similarity of the multimodal contrastive language image pre-training model to obtain a candidate disease corresponding to the target part.

[0169] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the livestock and poultry disease classification image recognition method provided by the above-mentioned methods, the method comprising: obtaining a target livestock and poultry image and a text description, wherein the text description is used to represent the disease category of the target livestock and poultry image; inputting the target livestock and poultry image and the text description into a multimodal comparative language image pre-training model, and obtaining a candidate disease suffered by the target part of the livestock and poultry in the target livestock and poultry image output by the multimodal comparative language image pre-training model, wherein the method comprises: using a multimodal comparative language image pre-training model to identify the disease of the target part of the livestock and poultry; The image encoder of the contrastive language image pre-training model is used to segment and encode the target livestock and poultry image to obtain at least one image feature representation, wherein the image feature representation is used to represent the characteristics of the target part of the target livestock and poultry image; the text encoder of the multimodal contrastive language image pre-training model is used to reconstruct and encode the text description to obtain at least one text description feature, wherein the text description feature is used to represent the candidate disease in the disease category; the multimodal contrastive language image pre-training model is used to align the image feature representation with the text description feature based on the similarity to obtain the candidate disease corresponding to the target part.

[0170] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0171] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0172] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for identifying images of livestock and poultry diseases, characterized in that: include: Acquire a target livestock and poultry image and a text description, wherein the text description is used to indicate the disease category of the target livestock and poultry image, and the text description includes a pre-set description text for livestock and poultry disease categories; The target livestock image and the text description are input into a multimodal contrastive language image pre-training model to obtain candidate diseases suffered by the target part of the livestock in the target livestock image output by the multimodal contrastive language image pre-training model, including: Segmenting and encoding the target livestock and poultry image through the image encoder of the multimodal contrastive language image pre-training model to obtain at least one image feature representation, wherein the image feature representation is used to represent the properties of the target part of the target livestock and poultry image; Reconstructing and encoding the text description through the text encoder of the multimodal contrastive language image pre-training model to obtain at least one text description feature, wherein the text description feature is used to represent the candidate disease in the disease category; By using the multimodal contrastive language image pre-training model, the candidate disease corresponding to the target site is obtained by aligning the image feature representation based on the similarity between the image feature representation and the text description feature; The reconstructing and encoding the text description to obtain at least one text description feature includes: Identify at least one target disease category; Reconstructing the text description based on preset disease type features corresponding to the target disease category to obtain a descriptor set corresponding to the target disease category; Mapping each descriptor of the descriptor set to an embedding space to obtain at least one text vector representation; Each of the text vector representations is normalized to obtain text description features corresponding to each of the descriptors.

2. The method for identifying livestock and poultry disease images according to claim 1, characterized in that: The step of segmenting and encoding the target livestock and poultry image to obtain at least one image feature representation includes: Performing instance segmentation on the target livestock and poultry image to obtain a set of part area images; Mapping each part region image in the part region image set to an embedding space to obtain at least one image vector representation; Each of the image vector representations is normalized to obtain image feature representations corresponding to each of the part region images.

3. The method for identifying livestock and poultry disease images according to claim 2, characterized in that: After performing instance segmentation on the target livestock and poultry image to obtain at least one part area image, the method further includes: Identify the species and the part of the part region image based on the species and the part of the part region image, and determine the segmented part of the part region image; Assigning preset labels to the segmented parts to obtain target part labels of the part region image; Based on the target part label of the part area image, the type range of candidate diseases of the target livestock and poultry image is determined.

4. The method for identifying livestock and poultry disease images according to claim 1, characterized in that: The aligning based on the similarity between the image feature representation and the text description feature to obtain the candidate disease corresponding to the target part includes: Determining a matching score between the image feature representation and the target disease category based on a target scoring function, wherein the target scoring function includes: a similarity mean function, a similarity weighting function, and a grade assignment function; The target disease category with a matching score greater than a preset score threshold is used as a candidate disease of the target part corresponding to the image feature representation; Wherein, when the target scoring function is the similarity mean function, determining the matching score between the image feature representation and the target disease category based on the target scoring function includes: Determine the similarity between the image feature representation and the text description feature corresponding to each descriptor in the descriptor set corresponding to at least one of the target disease categories; The average value of the similarities is used as the matching score between the image feature representation and the target disease category.

5. The method for identifying livestock and poultry disease images according to claim 1, characterized in that: Before acquiring the target livestock and poultry image and text description, the method further includes: Acquire a livestock and poultry image labeled sample set; when the number of samples in the livestock and poultry image labeled sample set is less than a preset sample number threshold, construct a visual cache of the livestock and poultry image labeled sample set, which includes: Extracting the visual embedding of each livestock and poultry image labeled sample in the livestock and poultry image labeled sample set through the visual encoder of the multimodal contrastive language image pre-trained model; Perform horizontal splicing based on the visual embedding to obtain an image information matrix, and use the image information matrix as a key part of the visual cache; Determine a unique hot encoding of each livestock and poultry image labeled sample in the livestock and poultry image labeled sample set, and a classification label of each livestock and poultry image labeled sample in the livestock and poultry image labeled sample set; A vertical connection is performed based on the one-hot encoding, and a horizontal connection is performed based on the classification label to obtain a value part of the visual cache.

6. A device for identifying images of livestock and poultry diseases, characterized in that: include: An acquisition unit, used for acquiring a target livestock and poultry image and a text description, wherein the text description is used to indicate the disease category of the target livestock and poultry image, and the text description includes a pre-set description text for livestock and poultry disease categories; The recognition unit is used to input the target livestock and poultry image and the text description into the multimodal contrast language image pre-training model to obtain the candidate diseases suffered by the target part of the livestock and poultry in the target livestock and poultry image output by the multimodal contrast language image pre-training model, which includes: A segmentation module, used to segment and encode the target livestock image through the image encoder of the multimodal contrast language image pre-training model to obtain at least one image feature representation, wherein the image feature representation is used to represent the properties of the target part of the target livestock image; A reconstruction module, used to reconstruct and encode the text description through the text encoder of the multimodal contrastive language image pre-training model to obtain at least one text description feature, wherein the text description feature is used to represent the candidate disease in the disease category; the reconstructing and encoding the text description to obtain at least one text description feature includes: determining at least one target disease category; reconstructing the text description based on the preset disease type feature corresponding to the target disease category to obtain a descriptor set corresponding to the target disease category; mapping each descriptor of the descriptor set to an embedding space to obtain at least one text vector representation; normalizing each of the text vector representations to obtain text description features corresponding to each of the descriptors; An alignment module is used to align the image feature representation and the text description feature based on the similarity through the multimodal contrast language image pre-training model to obtain the candidate disease corresponding to the target site.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method for identifying images of livestock and poultry epidemic diseases as described in any one of claims 1 to 5 is implemented.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for identifying images of livestock and poultry epidemic diseases as described in any one of claims 1 to 5 is implemented.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for identifying images of livestock and poultry epidemic diseases as described in any one of claims 1 to 5 is implemented.

Images