Livestock state detection method and device, electronic equipment and storage medium

Abstract

The invention provides a livestock state detection method and device, electronic equipment and a storage medium, and the method comprises the steps: determining livestock data of at least two modes of to-be-detected livestock, the at least two modes comprising at least two of an audio mode, an image mode and a text mode; performing livestock state detection based on the livestock data of the at least two modes to obtain a single detection result of each mode and a joint detection result of the at least two modes; the livestock state detection result of the to-be-detected livestock is determined based on the single detection result and the joint detection result, and detection is performed through the multi-modal livestock data, so that the problem of low reliability of the detection result obtained based on the single-modal data in the traditional scheme is well overcome, the accuracy and credibility of the livestock state detection result are improved, and the detection efficiency is improved. And effective early warning can be carried out in the early stage of disease outbreak according to the livestock state detection result, related personnel are reminded to carry out prevention and treatment, the livestock is timely treated, and the loss of a farm is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a livestock status detection method, device, electronic equipment and storage medium. Background Art

[0002] With the development of the livestock industry, people pay more and more attention to the monitoring of livestock growth and development and the prevention of their diseases during livestock breeding. Since the status of livestock is closely related to their disease prevention and growth status, livestock status detection has become a crucial step in the livestock breeding process.

[0003] At present, the process of livestock status detection mostly relies on manual work, that is, it depends on breeders, veterinarians and other personnel to judge the status of livestock by their symptoms during daily inspections, and then carry out corresponding treatment. However, there are a large number of livestock in large-scale farming, and it is difficult for manual inspections to cover all periods and all aspects. Moreover, when the inspection finds that the livestock show obvious symptoms, the livestock are often in the middle and late stages of the disease, and it is extremely difficult to treat them at this time. In addition, there are a few schemes that use the cough data of livestock for detection and judgment, but the detection effect of such schemes is often poor. The reason is that the judgment process of livestock illness is very complicated, and the credibility of the results obtained by detecting a single cough data is not high. Summary of the invention

[0004] The present invention provides a livestock status detection method, device, electronic device and storage medium, which are used to solve the defect of poor detection effect based on single audio data in the prior art, and achieve the dual improvement of the accuracy and credibility of livestock status detection results, as well as effective early warning of livestock diseases in large-scale breeding.

[0005] The present invention provides a method for detecting livestock status, comprising: Determining livestock data of at least two modalities of the livestock to be detected, the at least two modalities comprising at least two of an audio modality, an image modality, and a text modality; Performing livestock status detection based on the livestock data of the at least two modalities to obtain a single detection result of each modality and a joint detection result of the at least two modalities; Based on the single detection result and the combined detection result, a livestock status detection result of the livestock to be detected is determined.

[0006] According to a livestock status detection method provided by the present invention, when the livestock data of at least two modalities are livestock description text and livestock images, livestock status detection is performed based on the livestock data of at least two modalities to obtain a single detection result of each modality and a joint detection result of the at least two modalities, including: Extracting features from the livestock description text and the livestock image respectively to obtain livestock text features and livestock image features; Performing livestock status detection based on the livestock text features and the livestock image features respectively, and obtaining single detection results corresponding to the text modality and the image modality respectively; Perform feature vectorization based on the correlation between the livestock text features and the livestock image features to obtain a livestock text vector and a livestock image vector; Livestock status detection is performed based on the livestock text vector and the livestock image vector to obtain a joint detection result corresponding to the text modality and the image modality.

[0007] According to a livestock status detection method provided by the present invention, when the livestock data of at least two modes are livestock description text and livestock audio, the livestock status detection is performed based on the livestock data of at least two modes to obtain a single detection result of each mode and a joint detection result of the at least two modes, including: Extracting features from the livestock description text and the livestock audio respectively to obtain livestock text features and livestock audio features; Performing livestock state detection based on the livestock text features, the livestock audio and the livestock audio features, respectively, to obtain a single detection result corresponding to the text mode and the audio mode; Perform named entity recognition based on the livestock text features to obtain entity recognition results, and perform livestock status classification based on the livestock audio features to obtain audio status classification results; Based on the entity recognition result and the audio state classification result, a joint detection result corresponding to the text modality and the audio modality is determined.

[0008] According to a livestock status detection method provided by the present invention, when the livestock data of at least two modalities are livestock images and livestock audio, the livestock status detection is performed based on the livestock data of at least two modalities to obtain a single detection result of each modality and a joint detection result of the at least two modalities, including: Extracting features from the livestock image and the livestock audio respectively to obtain livestock image features and livestock audio features; Performing livestock state detection based on the livestock image features, the livestock audio and the livestock audio features, respectively, to obtain a single detection result corresponding to each of the image modality and the audio modality; Performing feature vectorization on the livestock image features to obtain a livestock image vector, and performing livestock state classification based on the livestock image vector and the livestock audio features to obtain an image state classification result and an audio state classification result; Based on the image state classification result and the audio state classification result, a joint detection result corresponding to the text modality and the audio modality is determined.

[0009] According to a livestock status detection method provided by the present invention, the livestock status detection result of the livestock to be detected is determined based on the single detection result and the combined detection result, including: Determine a state detection result of any modality based on a single detection result of any modality and a joint detection result of the at least two modalities; Based on the state detection results of the at least two modalities, the livestock state detection result of the livestock to be detected is determined.

[0010] According to a livestock status detection method provided by the present invention, the status detection result includes the livestock status and its corresponding probability; The step of determining the livestock status detection result of the livestock to be detected based on the status detection results of the at least two modes includes: In the case where the at least two modalities are any two of the audio modality, the image modality and the text modality, and the same livestock state exists in the state detection results of the two modalities, determining the livestock state detection result based on the same livestock states and their corresponding probabilities in the state detection results of the two modalities; In the case where the at least two modalities are any two of the audio modality, the image modality and the text modality, but the same livestock state does not exist in the state detection results of the two modalities, determining the livestock state detection result based on each livestock state in the state detection results of the two modalities and its corresponding probability; In a case where the at least two modalities are an audio modality, an image modality, and a text modality, and the same livestock state exists in the state detection results of the three modalities, determining the livestock state detection result based on the same livestock states in the state detection results of the three modalities and their corresponding probabilities; When the at least two modalities are audio modality, image modality and text modality, but the same livestock state does not exist in the state detection results of the three modalities, the livestock state detection result is determined based on the livestock state in the state detection results of the three modalities.

[0011] According to a livestock status detection method provided by the present invention, the step of determining livestock data of at least two modes of the livestock to be detected includes: Obtain user input corresponding to the livestock to be detected; Generate a response based on the user input to obtain a response to the question in the current round; Obtaining supplementary input corresponding to the answer to the question, and using the supplementary input as user input for the next round of the current round; Constructing the dialogue data corresponding to the livestock to be detected based on multiple rounds of user input and question responses; Multimodal data extraction is performed based on the conversation data to obtain livestock data of at least two modalities of the livestock to be detected.

[0012] The present invention also provides a livestock status detection device, comprising: A data determination unit, configured to determine livestock data of at least two modalities of the livestock to be detected, wherein the at least two modalities include at least two of an audio modality, an image modality, and a text modality; A state detection unit, configured to perform livestock state detection based on the livestock data of the at least two modalities, and obtain a single detection result of each modality and a joint detection result of the at least two modalities; A decision summarizing unit is used to determine the livestock status detection result of the livestock to be detected based on the single detection result and the combined detection result.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the livestock status detection methods described above when executing the computer program.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the livestock status detection method as described in any one of the above is implemented.

[0015] The livestock status detection method, device, electronic device and storage medium provided by the present invention perform livestock status detection through livestock data of at least two modes of the livestock to be detected, obtain a single detection result of a single mode, and a joint detection result of at least two modes, and make a decision fusion for the two to obtain the livestock status detection result of the livestock to be detected. The detection is performed by multi-modal livestock data, which well overcomes the problem of low reliability of the detection results obtained based on single-modal data in traditional solutions, and achieves a dual improvement in the accuracy and credibility of the livestock status detection results. In addition, the livestock status detection results can be used to effectively warn in the early stage of the disease outbreak, remind relevant personnel to carry out prevention and control, and rescue livestock in time to reduce the losses of the farm. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0017] Figure 1 It is a schematic diagram of the flow of the livestock status detection method provided by the present invention; Figure 2 It is an overall flow chart of the livestock status detection process provided by the present invention; Figure 3 is a schematic diagram of the text detection process provided by the present invention; Figure 4 is a schematic diagram of the image detection process provided by the present invention; Figure 5 It is a schematic diagram of the image and text detection process provided by the present invention; Figure 6 is a schematic diagram of the audio detection process provided by the present invention; Figure 7 is a schematic diagram of the text and sound detection process provided by the present invention; Figure 8 It is a schematic diagram of the image and sound detection process provided by the present invention; Fig. 9 It is a structural schematic diagram of the livestock status detection device provided by the present invention; Fig.10 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] At present, livestock status detection is mostly done manually, that is, breeders, veterinarians, etc. judge the status of livestock by their symptoms during daily inspections, such as frequent coughing and decreased food intake, and then carry out corresponding treatment. However, in large-scale farming, the number of livestock is large, and manual inspections are difficult to cover all time and all directions.

[0020] Moreover, when breeders, veterinarians, etc. discover obvious symptoms in livestock through patrolling the pens, the livestock are often already in the middle or late stages of the disease. At this time, the morbidity and infectiousness are extremely strong, making it extremely difficult to treat the livestock, which can easily lead to serious losses to the breeding farm.

[0021] In addition, there are a few schemes that use livestock coughing data for detection and judgment, but the detection effect of such schemes is often poor. The reason is that the process of judging whether livestock is sick is very complicated, and the credibility of the results obtained through single coughing data detection is not high.

[0022] Furthermore, with the development of deep learning technology, there are still a few solutions for livestock disease detection and judgment through deep learning models, that is, using cough sample data and its corresponding disease type labels to pre-train a livestock disease detection model, which records the mapping relationship between livestock cough data and disease types. Therefore, when the cough data is input into the livestock disease recognition model, the disease type corresponding to the cough data can be output accordingly.

[0023] However, although such solutions can detect and judge livestock diseases, their detection results are often poor. The reason is that the cough data used for detection are mostly mixed with other sounds, such as the eating sounds and humming sounds of pigs, and the noise will directly affect the detection results; more importantly, the process of judging livestock diseases is very complicated, and the accuracy and credibility of the results obtained through single cough data detection are not high.

[0024] In this regard, the present invention provides a livestock status detection method, which aims to perform livestock status detection on livestock to be detected through livestock data of at least two modalities, obtain a single detection result of a single modality, and a joint detection result of at least two modalities, and make a decision fusion between the two to obtain the livestock status detection result of the livestock to be detected. The detection through multi-modal livestock data solves the problem of low reliability of detection results obtained based on single modality data in traditional solutions, realizes high-accuracy and automated livestock status detection, reduces the workload of patrol personnel, and can provide effective early warning in the early stage of disease outbreak.

[0025] Figure 1 Schematic diagram of the livestock status detection method provided by the present invention, such as Figure 1 As shown, the method includes: Step 110, determining livestock data of at least two modalities of the livestock to be detected, wherein the at least two modalities include at least two of an audio modality, an image modality, and a text modality; Step 120, performing livestock status detection based on livestock data of at least two modalities, obtaining a single detection result of each modality and a joint detection result of at least two modalities; Step 130 , determining a livestock status detection result of the livestock to be detected based on the single detection result and the combined detection result.

[0026] Specifically, before performing livestock status detection, it is first necessary to determine the detection object, that is, the livestock to be detected. The livestock to be detected here can be pigs in a pig house, or cows and sheep in a cowshed or sheepfold, etc. The embodiment of the present invention does not make specific limitations on this.

[0027] After determining the livestock to be detected, it is necessary to obtain the data required for detecting it. Considering the problem that the accuracy and reliability of the results obtained by detecting based on a single audio data in the traditional scheme are not high, in the embodiment of the present invention, it is proposed to use multi-modal livestock data for detection to avoid the limitations of detection based on single-modal data.

[0028] Based on this, in an embodiment of the present invention, multimodal livestock data corresponding to the livestock to be detected can also be obtained, that is, livestock data of at least two modes of the livestock to be detected are determined, and the at least two modes here can include at least two of text image, image mode, and audio mode. In other words, the livestock data of the livestock to be detected obtained here can be any two of the livestock data in text mode, such as livestock description text (livestock status description text, livestock appearance description text, livestock posture description text, etc.), livestock data in image mode, such as livestock images (livestock lying images, livestock eating images, etc.), and livestock data in audio mode, such as livestock audio (livestock sleeping audio, livestock eating audio, etc.), and can also include the above three at the same time.

[0029] Among them, the livestock data in text mode is provided by breeders, veterinarians, etc. For example, breeders, veterinarians, etc. can determine the livestock description text of the livestock to be detected based on the posture and condition of the livestock to be detected observed during the inspection. The livestock data in image mode and audio mode can be obtained by collecting image acquisition equipment and sound pickup components installed in the livestock pen, or can be obtained by breeders, veterinarians, etc. manually through image acquisition. For example, the data in the livestock pen to be detected or its corresponding livestock pen can be retrieved from the data in the livestock pen collected by the image acquisition equipment and the sound pickup component, or the data of the livestock to be detected can be actively collected through handheld devices such as mobile phones, voice recorders, etc., which are not specifically limited in the embodiments of the present invention.

[0030] Furthermore, after obtaining livestock data of at least two modes of the livestock to be detected, in an embodiment of the present invention, livestock status detection can be performed based on the livestock data of at least two modes to obtain detection results. However, considering that if detection is performed based on the data of each mode alone, although there are multiple modal detection results obtained, the detection results of each mode are independent of each other and unrelated to each other. Therefore, although the reliability of the final detection result determined is improved to a certain extent compared with the detection result obtained by traditional detection based on single modal data, it still cannot well guarantee the comprehensiveness, accuracy and credibility of the detection result. In this regard, in an embodiment of the present invention, it is proposed that in the process of livestock status detection, multimodal data can be used for joint detection to obtain the corresponding detection result, that is, the joint detection result.

[0031] Specifically, here, livestock status detection can be first performed based on livestock data of each modality, and the detection result corresponding to each modality can be obtained, that is, a single detection result of each modality, for example, a text recognition result corresponding to the text modality, an image classification result corresponding to the image modality, and an audio classification result corresponding to the audio modality. At the same time, livestock status detection can be performed based on livestock data of two modalities to obtain detection results corresponding to the two modalities, that is, a joint detection result of the two modalities, for example, a joint detection result corresponding to the text modality and the image modality, a joint detection result corresponding to the text modality and the audio modality, and a joint detection corresponding to the image modality and the audio modality.

[0032] Figure 2 is an overall flow chart of the livestock status detection process provided by the present invention, such as Figure 2 As shown, after obtaining livestock data of at least two modalities, the livestock data can be processed through a gating network first, and different expert models can be called for processing according to the different livestock data obtained to obtain the detection results output by different expert models. Finally, the detection results output by each expert model can be summarized and decided to obtain the required livestock status detection results of the livestock to be detected.

[0033] For example, when the livestock data is livestock description text and livestock audio, you can select a text expert model, an audio expert model, and a text-audio hybrid expert model. By using these three to perform livestock status detection, you can obtain single detection results for the text modality and audio modality respectively, as well as the joint detection results corresponding to the text modality and audio modality.

[0034] For another example, when the livestock data consists of livestock description text and livestock images, you can select a text expert model, an image expert model, and an image-text hybrid expert model. By using these three to perform livestock status detection, you can obtain single detection results for the text modality and image modality respectively, as well as the joint detection results corresponding to the text modality and image modality.

[0035] For another example, when the livestock data is livestock images and livestock audio, you can select an image expert model, an audio expert model, and an image-audio mixed expert model. By using these three to perform livestock status detection, you can obtain single detection results for the audio modality and image modality respectively, as well as the joint detection results corresponding to the audio modality and image modality.

[0036] In addition, when the livestock data is livestock description text, livestock images and livestock audio, all expert models can be selected, including text expert model, image expert model, audio expert model, image-text mixed expert model, image-audio mixed expert model, and text-audio mixed expert model.

[0037] The weights of the selected different models are random, but the weights and values ​​of all the selected models are 1. The joint detection result can be the detection result of the two modalities, or the detection result after the fusion of the two modalities, which is not specifically limited in the embodiment of the present invention.

[0038] After that, a decision summary can be made based on the single detection results of each modality obtained by livestock status detection, and the joint detection results of at least two modalities, so as to fuse the detection results at multiple levels and obtain accurate and reliable livestock status detection results. Specifically, here, the single detection results of each modality and the joint detection results of multiple modalities can be fused through a decision module. The decision module can select a corresponding fusion strategy for fusion according to the specific modality and type of the obtained detection results, so as to optimize the detection results through the comparison of single modality and multi-modality, and the merging of different modalities, and obtain the most accurate and reliable livestock status detection results.

[0039] Here, the livestock status detection result can be a warning state or a non-warning state. The warning state indicates that the livestock to be tested currently has a disease, is in poor condition, and has a greater risk, so a warning is required, pending further verification by the breeder, veterinarian, etc.; correspondingly, the non-warning state indicates that the livestock to be tested currently has no disease, is in good condition, and has no risk, so no warning is required.

[0040] In the embodiment of the present invention, different expert models are used to detect livestock status according to different modalities of the acquired livestock data, and single detection results of each modality and multi-modal joint detection results are obtained. Finally, the detection results output by different expert models are summarized for decision-making to obtain accurate and reliable livestock status detection results. The detection results and the corresponding livestock data can be provided to veterinarians for verification, which can greatly reduce the time and workload of patrolling the fences, and avoid problems such as insufficient patrol personnel and untimely diagnosis.

[0041] The livestock status detection method provided by the present invention performs livestock status detection through livestock data of at least two modes of livestock to be detected, obtains a single detection result of a single mode, and a joint detection result of at least two modes, and makes a decision fusion for the two to obtain the livestock status detection result of the livestock to be detected. The detection is performed through multi-modal livestock data, which well overcomes the problem of low reliability of the detection result obtained based on single-modal data in the traditional solution, and achieves a double improvement in the accuracy and credibility of the livestock status detection result. In addition, the livestock status detection result can be used to effectively warn in the early stage of the disease outbreak, remind relevant personnel to carry out prevention and treatment, and timely rescue livestock to reduce the loss of the farm; at the same time, the automated livestock status detection process avoids the subjectivity and uncertainty of manual detection judgment, and also saves the time and energy required for manual judgment.

[0042] Based on the above embodiment, in the case where the livestock data of at least two modalities are livestock description text and livestock image, step 120 includes: Extracting features from livestock description text and livestock images respectively to obtain livestock text features and livestock image features; Livestock status detection is performed based on livestock text features and livestock image features respectively, and a single detection result corresponding to the text modality and the image modality is obtained; Based on the correlation between livestock text features and livestock image features, feature vectorization is performed to obtain livestock text vectors and livestock image vectors; Livestock status detection is performed based on livestock text vectors and livestock image vectors, and a joint detection result corresponding to the text modality and the image modality is obtained.

[0043] Specifically, when the acquired livestock data is data in text mode and data in image mode, that is, livestock description text and livestock images, livestock status detection can be implemented by a text expert model, an image expert model, and an image-text hybrid expert model, specifically including: Figure 3 It is a schematic diagram of the text detection process provided by the present invention, such as Figure 3As shown, first, feature extraction can be performed on the livestock description text to obtain its corresponding text features, namely, livestock text features. Specifically, the livestock description text can be input into a language model, such as BERT (Bidirectional Encoder Representations from Transformers), to perform feature extraction to obtain livestock text features, and can be vectorized to convert the text into a vector; then, livestock status detection can be performed based on this vector to obtain a single detection result of the text modality. Specifically, this can be performed by matching this vector with a vector knowledge base in a text expert model to query and calculate the similarity value between the vector and the vector in the vector knowledge base, and determining a number of vectors with higher similarity (such as 3, 4, etc.) based on the calculation results, and outputting their corresponding diseases and states as results. In this way, a single detection result of the text modality can be obtained, also known as a text recognition result.

[0044] Before using the text expert model to detect livestock status, in the embodiment of the present invention, it is necessary to pre-build a text expert model. The construction process of the text expert model specifically includes: first, a large amount of livestock status data can be collected, that is, books and papers related to livestock diseases, status, etc., and corresponding livestock disease status description documents can be extracted from livestock disease and status archives of the farm; then, the acquired data can be structured to organize it into structured data, specifically, the description of each disease and status can be expanded through a large language model (such as the Spark Cognitive Large Model), and finally the description text corresponding to a disease and status can be expanded to about 100, and then the disease and status description text is segmented to split the description text into small segments with descriptions of key parts according to different parts of the livestock; after that, the split text can be feature extracted and vectorized to map it to a vector space of a certain dimension, realize data vectorization, and store it in a vector knowledge base, and then this vector knowledge base can be used to construct a text expert model.

[0045] Figure 4 It is a schematic diagram of the image detection process provided by the present invention, such as Figure 4As shown, first, feature extraction can be performed on livestock images to obtain corresponding image features, namely, livestock image features. Specifically, feature extraction can be performed using the yolov8 (you only look once version 8) architecture in the image expert model to obtain livestock image features. Subsequently, livestock status detection can be performed based on the livestock image features to obtain a single detection result of the image modality. Specifically, based on the livestock image features, classification can be performed using the multi-label classification head in the image expert model to obtain probability values ​​of livestock images corresponding to different disease and status categories, namely, category probability distribution. Further screening can be performed to retain diseases and states with relatively high probabilities (probability above 25%), which are output as the result. In this way, a single detection result of the image modality can be obtained, also known as an image classification result.

[0046] Among them, the image expert model is built on the basis of the overall architecture of yolov8. The model is modified by setting multiple classification heads and multi-class loss functions to enable it to perform multi-label image classification tasks. During model training, the model is trained through the constructed multi-label image dataset. A single image and multi-class labels are input to calculate the loss functions and probability values ​​of different categories. Finally, a trained multi-label image classification model can be obtained.

[0047] Figure 5 It is a schematic diagram of the image and text detection process provided by the present invention, such as Figure 5 As shown in the figure, the image-text hybrid expert model adopts a dual-stream structure, with images input on one side and text input on the other side. For the input livestock description text, feature extraction can be performed first to obtain livestock text features, and livestock images can be extracted through the CLIP (Contrastive Language–Image Pre-training) model to obtain features of different patches; then, the livestock text features and livestock image features of different image blocks can be input into the transformers corresponding to their respective branches. After multi-layer transformer processing, the vector features of the image and the vector features of the text, i.e., livestock text vectors and livestock image vectors, can be finally obtained. It is worth noting that in the transformers of the two branches, the K and L parameters of the transformer layers corresponding to the image and text will interact to fuse different feature parameters, and finally a set of livestock text vectors and livestock image vectors can be obtained. Then, livestock status detection can be performed based on the livestock text vectors and livestock image vectors to obtain the joint detection results corresponding to the text modality and the image modality, i.e., by inputting a set of images and texts, the probabilities of the disease and status categories corresponding to the images and texts, i.e., the category probability distribution, can be output.

[0048] It should be noted that before using the image-text hybrid expert model for detection, it is necessary to pre-train the image-text hybrid expert model. The specific process includes: pre-constructing a multimodal dataset, selecting the image and text pair data corresponding to the image and text, and then pre-training based on the open source VLE (Vision-Language Encoder) model. VLE adopts a dual-stream structure, with the image input on one side and the text corresponding to the image input on the other side. First, feature extraction is performed on each branch to obtain text features and features of different patches. These two are used as inputs of the transformer in each branch. After multi-layer transformer processing, the vectors corresponding to the image and text are finally obtained. The K and L parameters of the transformer layers corresponding to the image and text interact to fuse and learn different feature parameters. Finally, a set of image vectors and text vectors can be obtained, and then the MLM (Masked Language Modelling) and ITM (Interactive Inference Head) losses of these two sets of features are calculated. The model is trained based on this loss, and a trained model can be obtained.

[0049] Among them, the construction of a multimodal data set includes: since livestock will show different performances under different diseases and symptoms, in terms of sound, different diseases correspond to different sounds made by livestock, such as wheezing, dry cough, wet cough, humming, etc.; in terms of images, different diseases correspond to different livestock postures, body movements, body reactions, and pathological conditions in different parts, etc., and the conditions of livestock in multiple angles under different diseases and states can be recorded to be able to see the overall posture and key parts of the livestock; in terms of description, different diseases correspond to different descriptions of the disease, whether academic professional descriptions of the disease or daily veterinary descriptions of different diseases. Based on this, in an embodiment of the present invention, the sound, image, and disease description text of the livestock can be recorded for different diseases and states, and a multimodal data set can be constructed based on this.

[0050] In an embodiment of the present invention, common livestock diseases and conditions in farms are classified, different knowledge bases and databases are established according to different disease and condition categories, and a multimodal data set is obtained. Then, different expert models are constructed and trained using the multimodal data set, and different expert models are used according to livestock data of different modalities. Decisions are summarized for the detection results output by different expert models, thereby achieving high-accuracy and automated livestock loading detection, and obtaining accurate and reliable livestock status detection results.

[0051] Based on the above embodiment, in the case where the livestock data of at least two modalities are livestock description text and livestock audio, step 120 includes: Extracting features from the livestock description text and the livestock audio respectively to obtain livestock text features and livestock audio features; Performing livestock state detection based on livestock text features, livestock audio and livestock audio features, respectively, to obtain single detection results corresponding to the text modality and the audio modality; Perform named entity recognition based on livestock text features to obtain entity recognition results, and perform livestock status classification based on livestock audio features to obtain audio status classification results; Based on the entity recognition results and the audio state classification results, the joint detection results corresponding to the text modality and the audio modality are determined.

[0052] Specifically, when the acquired livestock data is data in text mode and data in audio mode, that is, livestock description text and livestock audio, livestock status detection can be implemented by a text expert model, an audio expert model, and a text-audio hybrid expert model, specifically including: Figure 6 is a schematic diagram of the audio detection process provided by the present invention, such as Figure 6 As shown, first, feature extraction can be performed on the livestock audio to obtain livestock audio features. Specifically, the livestock audio can be subjected to noise reduction processing first, and then frame processing is performed and audio features are extracted to obtain livestock audio features. The livestock audio features here can be PCEN (Per-channel energy normalization), log mel features (logmelspectrogram features), MFCC (Mel Frequency Cepstral Coefficients), etc. Afterwards, the extracted livestock audio features and livestock audio can be input into the audio expert model for livestock status detection to obtain a single detection result of the audio modality; specifically, the input livestock audio and livestock audio features can be firstly embedded, and then the embedding results are input into the encoding network for encoding, and finally classified through the classification head to obtain the probability of livestock audio corresponding to different diseases and status categories, that is, the category probability distribution, and further screening can be performed to retain several categories with higher probabilities (the first 2, 3, etc.). If the probabilities of these categories are all greater than preset values ​​(such as 40%, 45%, etc.), these categories are retained, otherwise, only the category with the largest probability is retained and output as the result. In this way, a single detection result of the audio modality can be obtained, also known as the audio classification result.

[0053] Before using the audio expert model to detect livestock status, in an embodiment of the present invention, it is necessary to pre-train the audio expert model. The specific process includes: constructing an initial model for audio detection based on a convolutional neural network, and then using the audio corresponding to different diseases and statuses in a multimodal data set to train the initial model, and finally obtaining a trained model, namely, an audio expert model.

[0054] The process of performing livestock status detection based on livestock description text to obtain a single detection result of text modality has been described in detail above and will not be repeated here.

[0055] Figure 7 Schematic diagram of the text and speech detection process provided by the present invention, such as Figure 7 As shown in the figure, the text-audio hybrid expert model adopts a dual-stream structure, with text input on one side and audio input on the other side. For the input livestock description text, feature extraction can be performed first to obtain livestock text features. Specifically, it is encoded into feature vector tokens through BERT to obtain livestock text features. Then, named entity recognition can be performed based on this livestock text feature to identify the categories and corresponding probabilities of livestock diseases and states described in the livestock description text, and it can be further processed to screen out categories with higher probabilities (probability greater than 35%, 40%, etc.) to obtain the final entity recognition result. At the same time, the livestock audio can be encoded first to obtain livestock audio features. Then, the livestock status can be classified using the transformer classification network based on this livestock audio feature to obtain the categories and probabilities of diseases and states corresponding to the livestock audio, that is, the audio status classification result. Combining this audio status classification result with the entity recognition result, the joint detection result corresponding to the text modality and the image modality can be determined.

[0056] It should be noted that before using the text-audio hybrid expert model for detection, it is necessary to pre-train the text-audio hybrid expert model. The specific process includes: selecting the audio and text corresponding to the text from the pre-built multimodal data set, and then building the initial model. The overall architecture of the model is similar to the image-text hybrid expert model, but the audio branch is different. Specifically, a piece of audio is input on one side of the branch, and it passes through the encoder and transformer classification network to obtain the category and probability of the disease and state corresponding to the audio. The text branch is first processed by BERT to obtain text features, and then named entity recognition is performed. Finally, the total loss function is calculated based on the named entity recognition results of the text and the classification results of the audio. The loss function includes the classification loss of the audio and the loss function of the named entity recognition.

[0057] Based on the above embodiment, when the livestock data of at least two modalities are livestock images and livestock audio, step 120 includes: Extracting features of livestock images and livestock audio respectively to obtain livestock image features and livestock audio features; Performing livestock state detection based on livestock image features, livestock audio and livestock audio features, respectively, to obtain single detection results corresponding to the image modality and the audio modality; Performing feature vectorization on livestock image features to obtain livestock image vectors, and performing livestock state classification based on the livestock image vectors and livestock audio features to obtain image state classification results and audio state classification results; Based on the image state classification results and the audio state classification results, a joint detection result corresponding to the text modality and the audio modality is determined.

[0058] Specifically, when the acquired livestock data is data in image mode and data in audio mode, that is, livestock images and livestock audio, livestock state detection can be implemented by an image expert model, an audio expert model, and an image-audio hybrid expert model, specifically including: The process of detecting livestock status based on livestock images and livestock status based on livestock audio to obtain single detection results corresponding to image modality and audio modality respectively has been described in detail above and will not be repeated here.

[0059] Figure 8 It is a schematic diagram of the image and sound detection process provided by the present invention, such as Figure 8 As shown in the figure, the image-audio hybrid expert model adopts a dual-stream structure, with one side inputting the image and the other side inputting the audio. For the input livestock image, feature extraction can be performed first to obtain livestock image features. Specifically, feature extraction is performed through the CLIP (Contrastive Language–Image Pre-training) model to obtain features of different patches; then, the livestock image features of different image blocks can be input into the transformer. After multi-layer transformer processing, the vector features of the image, the livestock image vector, can be finally obtained. The livestock state classification can be performed on the basis of this livestock image vector to obtain the category and probability of the disease and state corresponding to the livestock image, that is, the image state classification result. For the input livestock audio, encoding can be performed first to obtain livestock audio features. Then, the livestock state classification can be performed using the transformer classification network based on this livestock audio feature to obtain the category and probability of the disease and state corresponding to the livestock audio, that is, the audio state classification result. Combining this image state classification result and the audio state classification result, the joint detection result corresponding to the image modality and the audio modality can be determined.

[0060] It should be noted that before using the image-audio hybrid expert model for detection, it is also necessary to pre-train the image-audio hybrid expert model. The specific process includes: selecting the image and audio pair data corresponding to the image and audio from the pre-constructed multimodal dataset, and then building the initial model. The overall architecture of the model is similar to the image-text hybrid expert model. Both branches are classification networks. The image branch performs a multi-label classification task and finally outputs the probability values ​​of the image corresponding to different labels. The audio branch performs an audio classification task and finally outputs the classification result corresponding to the audio. The loss function is the sum of the loss functions of these two branches.

[0061] Based on the listed embodiment, step 130 includes: Determine a state detection result of the modality based on a single detection result of any modality and a joint detection result of at least two modalities; Based on the status detection results of at least two modalities, a livestock status detection result of the livestock to be detected is determined.

[0062] Specifically, in step 130, the process of determining the livestock status detection result of the livestock to be detected according to the single detection result and the combined detection result may specifically include: It can be understood that after obtaining the single detection result of each modality and the joint detection result of multiple modalities, in an embodiment of the present invention, each modality can be processed first, and a comparison can be made based on the single detection result of any modality and the joint detection result of at least two modalities including the modality to determine whether the same disease or condition category exists in the single detection result and the joint detection result under the same modality, and the state detection result of the modality can be determined based on the comparison results, combined with the single detection result of the modality and the joint detection result of at least two modalities including the modality.

[0063] Furthermore, after the previous step, the state detection results of each modality can be obtained. At this time, the state detection results of all modalities can be fused and summarized, and finally the livestock state detection results of the livestock to be detected can be obtained. Specifically, the state detection results of different modalities can be combined to determine whether there are the same disease and state categories, and different merging methods can be selected to merge the state detection results of all modalities according to the comparison results. For example, when there is only one category in the state detection results of all modalities, if there are the same categories, the same categories can be merged and used as the final livestock state detection result. If there are no same categories, the state detection results of all modalities can be combined and used as the final livestock state detection result.

[0064] Based on the above embodiment, the state detection result includes the livestock state and its corresponding probability; Determining a livestock state detection result of the livestock to be detected based on the state detection results of at least two modalities includes: In a case where at least two of the modalities are any two of the audio modality, the image modality, and the text modality, and the same livestock state exists in the state detection results of the two modalities, determining the livestock state detection result based on the same livestock states in the state detection results of the two modalities and their corresponding probabilities; In a case where at least two modalities are any two of an audio modality, an image modality, and a text modality, but the same livestock state does not exist in the state detection results of the two modalities, determining the livestock state detection result based on each livestock state in the state detection results of the two modalities and its corresponding probability; In a case where at least two of the modalities are an audio modality, an image modality, and a text modality, and the same livestock state exists in the state detection results of the three modalities, determining the livestock state detection result based on the same livestock states in the state detection results of the three modalities and their corresponding probabilities; When at least two modalities are audio modality, image modality and text modality, but the same livestock state does not exist in the state detection results of the three modalities, the livestock state detection result is determined based on the livestock state in the state detection results of the three modalities.

[0065] Specifically, the process of determining the livestock status detection result of the livestock to be detected based on the status detection results of at least two modes specifically includes the following situations: First, when at least two modalities are any two of the audio modality, the image modality and the text modality, and the same livestock state exists in the state detection results of the two modalities, the livestock state detection result can be determined based on all the same livestock states in the state detection results of the two modalities and their corresponding probabilities; conversely, if the same livestock state does not exist in the state detection results of the two modalities, the livestock state detection result can be determined based on all the livestock states in the state detection results of the two modalities and their corresponding probabilities.

[0066] For example, when the livestock data is livestock description text and livestock images, the output results of three expert models will be obtained. At this time, the single detection result of the text modality and the joint detection results corresponding to the image modality and the text modality can be compared first. If the same category / livestock state exists in the two, the livestock state with the highest probability among all the same livestock states can be used as the state detection result of the text modality. If there is no identical livestock state, the single detection result and the result obtained through text recognition in the joint detection result are saved as the state detection result of the text modality. Similarly, the state detection result of the image modality can be obtained in the same way. Finally, the state detection result of the text modality and the state detection result of the image modality can be compared. If there is only one livestock state in both, the two are compared to determine whether they are the same. If they are the same, this livestock state is used as the livestock state detection result. If they are different, these two different livestock states are used as the final livestock state detection result. If either the state detection result of the text modality or the state detection result of the image modality contains multiple livestock states, they are compared to determine whether there are the same livestock states in the two. If so, the livestock state with the highest probability among all the same livestock states is used as the final livestock state detection result. Otherwise, if not, the livestock state with the highest probability in the state detection result of the text modality and the state detection result of the image modality are used together as the livestock state detection result.

[0067] When the livestock data are livestock images and livestock audio, and livestock description text and livestock audio, the decision fusion process is basically the same.

[0068] Second, when at least two modalities are audio modality, image modality and text modality, and the same livestock state exists in the state detection results of the three modalities, the livestock state detection result can be determined based on all the same livestock states in the state detection results of the three modalities and their corresponding probabilities; conversely, if the same livestock state does not exist in the state detection results of the three modalities, the livestock state detection result can be determined based on all the livestock states in the state detection results of the three modalities.

[0069] For example, when livestock data includes livestock description text, livestock audio, and livestock images, the output results of 6 expert models will be obtained. At this time, they can be divided according to different modalities. First, the three types of results, text, audio, and image, are divided according to the modality, and then the results of each modality are compared respectively. In the comparison process, the same livestock state is first found for the results of the same modality. If there is the same livestock state, the livestock state with the highest probability among all the same livestock states is used as the state detection result of this modality. If not, the livestock state with the highest probability among different results is determined and used as the state detection result of this modality; for example, for text modality, if there is no same livestock state in the single detection result of text modality, the joint detection result corresponding to image modality and text modality, and the joint detection result corresponding to text modality and audio modality, the livestock state with the highest probability is selected from these three respectively, and the selected livestock state is used as the state detection result of text modality. Similarly, the state detection result of image modality and the state detection result of audio modality can be obtained.

[0070] Next, the state detection results of the text modality, the state detection results of the image modality, and the state detection results of the audio modality can be compared. If there are identical livestock states among the three, the livestock state with the highest probability among all the identical livestock states is used as the final livestock state detection result. If there are no identical livestock states, all the livestock states among the three are used as the final livestock state detection results.

[0071] Based on the above embodiment, step 110 includes: Obtain user input corresponding to the livestock to be detected; Generate responses based on user input to get responses to questions in the current round; Obtaining supplementary input corresponding to the question answer, and using the supplementary input as the user input for the next round of the current round; Based on multiple rounds of user input and question responses, the conversation data corresponding to the livestock to be detected is constructed; Multimodal data extraction is performed based on the conversation data to obtain livestock data of at least two modalities of the livestock to be detected.

[0072] Specifically, in step 110, the process of determining livestock data of at least two modes of the livestock to be detected may specifically include: It is understandable that when a breeder or veterinarian finds an abnormality during daily fence inspection and needs to perform livestock status detection, or when the livestock status detection cycle / time set by the farm is reached, the livestock status detection process will be started. At this time, the user's (breeder, veterinarian, etc.) input can be received, that is, the user's input data for the livestock to be detected, which can be text, image or audio, and the embodiment of the present invention does not specifically limit this.

[0073] However, considering that in actual interaction, the data input by the user for the first time is mostly incomplete, and does not clearly and intuitively describe and present the condition of the livestock to be detected, therefore, in order to make the livestock status detection process more accurate and comprehensive, in an embodiment of the present invention, a guided manner can be used to interact with the user to guide the user to input multimodal information about the livestock to be detected, so as to fully understand the condition of the livestock to be detected, thereby ensuring the accuracy of subsequent detection.

[0074] Specifically, in an embodiment of the present invention, after obtaining the input data corresponding to the livestock to be detected by the user, a reply may be generated according to the input data of the user, so as to guide the user to supplement more information in the form of questions, that is, a reply may be generated according to the user input to obtain the reply to the question of the current round, and the reply question may include a description of the information that needs to be supplemented by the user, for example, "The current information is not enough to accurately judge the state of the pigs. Can you provide images of pigs eating and audio of pigs sleeping to facilitate my further judgment?" After seeing the reply question, the user can supplement the information in a targeted manner and input it again, and the user's supplementary input for the reply to the question of the current round can be obtained. At this time, this supplementary input can be used as the user input of the next round of the current round. Repeat the above process to obtain a multi-round dialogue data.

[0075] After that, information extraction can be performed on the constructed multi-round dialogue data to extract multi-modal data, such as text modality, image modality data, etc. At this time, a large language model (such as the Spark Cognitive Large Model) can be called to process the multi-round dialogue data to extract the user's intentions and information in different modalities, such as livestock diseases, symptoms description texts, livestock images, livestock weight, body shape, etc., and they can be sorted to finally obtain livestock data in at least two modalities of the livestock to be tested.

[0076] The livestock state detection device provided by the present invention is described below. The livestock state detection device described below and the livestock state detection method described above can be referred to each other.

[0077] Fig. 9 Schematic diagram of the structure of the livestock status detection device provided by the present invention. Fig. 9 As shown, the device comprises: A data determination unit 910 is used to determine livestock data of at least two modalities of the livestock to be detected, wherein the at least two modalities include at least two of an audio modality, an image modality, and a text modality; A state detection unit 920, configured to perform livestock state detection based on the livestock data of the at least two modalities, and obtain a single detection result of each modality and a joint detection result of the at least two modalities; The decision summarizing unit 930 is used to determine the livestock status detection result of the livestock to be detected based on the single detection result and the combined detection result.

[0078] The livestock status detection device provided by the present invention performs livestock status detection through livestock data of at least two modes of livestock to be detected, obtains a single detection result of a single mode, and a joint detection result of at least two modes, and makes a decision fusion between the two to obtain the livestock status detection result of the livestock to be detected. The detection is performed through multi-modal livestock data, which well overcomes the problem of low reliability of the detection result obtained based on single-modal data in the traditional solution, and achieves a double improvement in the accuracy and credibility of the livestock status detection result. In addition, the livestock status detection result can be used to effectively warn in the early stage of the disease outbreak, remind relevant personnel to carry out prevention and treatment, and timely rescue livestock to reduce the loss of the farm; at the same time, the automated livestock status detection process avoids the subjectivity and uncertainty of manual detection and judgment, and also saves the time and energy required for manual judgment.

[0079] Based on the above embodiment, when the livestock data of the at least two modalities are livestock description text and livestock image, the state detection unit 920 is used to: Extracting features from the livestock description text and the livestock image respectively to obtain livestock text features and livestock image features; Performing livestock status detection based on the livestock text features and the livestock image features respectively, and obtaining single detection results corresponding to the text modality and the image modality respectively; Perform feature vectorization based on the correlation between the livestock text features and the livestock image features to obtain a livestock text vector and a livestock image vector; Livestock status detection is performed based on the livestock text vector and the livestock image vector to obtain a joint detection result corresponding to the text modality and the image modality.

[0080] Based on the above embodiment, when the livestock data of the at least two modes are livestock description text and livestock audio, the state detection unit 920 is used to: Extracting features from the livestock description text and the livestock audio respectively to obtain livestock text features and livestock audio features; Performing livestock state detection based on the livestock text features, the livestock audio and the livestock audio features, respectively, to obtain a single detection result corresponding to the text mode and the audio mode; Perform named entity recognition based on the livestock text features to obtain entity recognition results, and perform livestock status classification based on the livestock audio features to obtain audio status classification results; Based on the entity recognition result and the audio state classification result, a joint detection result corresponding to the text modality and the audio modality is determined.

[0081] Based on the above embodiment, when the livestock data of the at least two modalities are livestock images and livestock audio, the state detection unit 920 is used to: Extracting features from the livestock image and the livestock audio respectively to obtain livestock image features and livestock audio features; Performing livestock state detection based on the livestock image features, the livestock audio and the livestock audio features, respectively, to obtain a single detection result corresponding to each of the image modality and the audio modality; Performing feature vectorization on the livestock image features to obtain a livestock image vector, and performing livestock state classification based on the livestock image vector and the livestock audio features to obtain an image state classification result and an audio state classification result; Based on the image state classification result and the audio state classification result, a joint detection result corresponding to the text modality and the audio modality is determined.

[0082] Based on the above embodiment, the decision aggregation unit 920 is used to: Determine a state detection result of any modality based on a single detection result of any modality and a joint detection result of the at least two modalities; Based on the state detection results of the at least two modalities, the livestock state detection result of the livestock to be detected is determined.

[0083] Based on the above embodiment, the state detection result includes the livestock state and its corresponding probability; the decision summary unit 920 is used to: In the case where the at least two modalities are any two of the audio modality, the image modality and the text modality, and the same livestock state exists in the state detection results of the two modalities, determining the livestock state detection result based on the same livestock states and their corresponding probabilities in the state detection results of the two modalities; In the case where the at least two modalities are any two of the audio modality, the image modality and the text modality, but the same livestock state does not exist in the state detection results of the two modalities, determining the livestock state detection result based on each livestock state in the state detection results of the two modalities and its corresponding probability; In a case where the at least two modalities are an audio modality, an image modality, and a text modality, and the same livestock state exists in the state detection results of the three modalities, determining the livestock state detection result based on the same livestock states in the state detection results of the three modalities and their corresponding probabilities; When the at least two modalities are audio modality, image modality and text modality, but the same livestock state does not exist in the state detection results of the three modalities, the livestock state detection result is determined based on the livestock state in the state detection results of the three modalities.

[0084] Based on the above embodiment, the data determination unit 910 is used to: Obtain user input corresponding to the livestock to be detected; Generate a response based on the user input to obtain a response to the question in the current round; Obtaining supplementary input corresponding to the answer to the question, and using the supplementary input as user input for the next round of the current round; Constructing the dialogue data corresponding to the livestock to be detected based on multiple rounds of user input and question responses; Multimodal data extraction is performed based on the conversation data to obtain livestock data of at least two modalities of the livestock to be detected.

[0085] Fig.10 An example of a physical structure diagram of an electronic device is shown in FIG. Fig.10 As shown, the electronic device may include: a processor 1010, a communication interface 1020, a memory 1030 and a communication bus 1040, wherein the processor 1010, the communication interface 1020 and the memory 1030 communicate with each other through the communication bus 1040. The processor 1010 may call the logic instructions in the memory 1030 to execute the livestock state detection method, which includes: determining livestock data of at least two modes of livestock to be detected, wherein the at least two modes include at least two of an audio mode, an image mode and a text mode; performing livestock state detection based on the livestock data of the at least two modes, obtaining a single detection result of each mode and a joint detection result of the at least two modes; and determining the livestock state detection result of the livestock to be detected based on the single detection result and the joint detection result.

[0086] In addition, the logic instructions in the above-mentioned memory 1030 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0087] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the livestock status detection method provided by the above-mentioned methods, and the method includes: determining livestock data of at least two modalities of the livestock to be detected, and the at least two modalities include at least two of audio modality, image modality and text modality; performing livestock status detection based on the livestock data of the at least two modalities to obtain a single detection result of each modality, and a joint detection result of the at least two modalities; based on the single detection result and the joint detection result, determining the livestock status detection result of the livestock to be detected.

[0088] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the livestock status detection method provided by the above-mentioned methods, the method comprising: determining livestock data of at least two modalities of the livestock to be detected, the at least two modalities comprising at least two of an audio modality, an image modality and a text modality; performing livestock status detection based on the livestock data of the at least two modalities to obtain a single detection result of each modality, and a joint detection result of the at least two modalities; determining the livestock status detection result of the livestock to be detected based on the single detection result and the joint detection result.

[0089] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0090] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0091] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting livestock status, characterized in that: include: Determining livestock data of at least two modalities of the livestock to be detected, the at least two modalities comprising at least two of an audio modality, an image modality, and a text modality; Performing livestock status detection based on the livestock data of the at least two modalities to obtain a single detection result of each modality and a joint detection result of the at least two modalities; Based on the single detection result and the combined detection result, a livestock status detection result of the livestock to be detected is determined.

2. The livestock status detection method according to claim 1, characterized in that: In the case where the livestock data of the at least two modalities are livestock description text and livestock images, the livestock status detection based on the livestock data of the at least two modalities to obtain a single detection result of each modality and a joint detection result of the at least two modalities includes: Extracting features from the livestock description text and the livestock image respectively to obtain livestock text features and livestock image features; Performing livestock status detection based on the livestock text features and the livestock image features respectively, and obtaining single detection results corresponding to the text modality and the image modality respectively; Perform feature vectorization based on the correlation between the livestock text features and the livestock image features to obtain a livestock text vector and a livestock image vector; Livestock status detection is performed based on the livestock text vector and the livestock image vector to obtain a joint detection result corresponding to the text modality and the image modality.

3. The livestock status detection method according to claim 1, characterized in that: In the case where the livestock data of the at least two modalities are livestock description text and livestock audio, the livestock status detection is performed based on the livestock data of the at least two modalities to obtain a single detection result of each modality and a joint detection result of the at least two modalities, including: Extracting features from the livestock description text and the livestock audio respectively to obtain livestock text features and livestock audio features; Performing livestock state detection based on the livestock text features, the livestock audio and the livestock audio features, respectively, to obtain a single detection result corresponding to the text mode and the audio mode; Perform named entity recognition based on the livestock text features to obtain entity recognition results, and perform livestock status classification based on the livestock audio features to obtain audio status classification results; Based on the entity recognition result and the audio state classification result, a joint detection result corresponding to the text modality and the audio modality is determined.

4. The livestock status detection method according to claim 1, characterized in that: In the case where the livestock data of the at least two modalities are livestock images and livestock audio, the livestock state detection based on the livestock data of the at least two modalities to obtain a single detection result of each modality and a joint detection result of the at least two modalities includes: Extracting features from the livestock image and the livestock audio respectively to obtain livestock image features and livestock audio features; Performing livestock state detection based on the livestock image features, the livestock audio and the livestock audio features, respectively, to obtain a single detection result corresponding to the image modality and the audio modality; Performing feature vectorization on the livestock image features to obtain a livestock image vector, and performing livestock state classification based on the livestock image vector and the livestock audio features to obtain an image state classification result and an audio state classification result; Based on the image state classification result and the audio state classification result, a joint detection result corresponding to the text modality and the audio modality is determined.

5. The livestock status detection method according to any one of claims 1 to 4, characterized in that: The step of determining the livestock status detection result of the livestock to be detected based on the single detection result and the combined detection result includes: Determine a state detection result of any modality based on a single detection result of any modality and a joint detection result of the at least two modalities; Based on the state detection results of the at least two modalities, the livestock state detection result of the livestock to be detected is determined.

6. The livestock status detection method according to claim 5, characterized in that: The state detection result includes the livestock state and its corresponding probability; The step of determining the livestock status detection result of the livestock to be detected based on the status detection results of the at least two modes includes: In the case where the at least two modalities are any two of the audio modality, the image modality and the text modality, and the same livestock state exists in the state detection results of the two modalities, determining the livestock state detection result based on the same livestock states in the state detection results of the two modalities and their corresponding probabilities; In a case where the at least two modalities are any two of the audio modality, the image modality and the text modality, but the same livestock state does not exist in the state detection results of the two modalities, determining the livestock state detection result based on each livestock state in the state detection results of the two modalities and its corresponding probability; In a case where the at least two modalities are an audio modality, an image modality, and a text modality, and the same livestock state exists in the state detection results of the three modalities, determining the livestock state detection result based on the same livestock states in the state detection results of the three modalities and their corresponding probabilities; When the at least two modalities are audio modality, image modality and text modality, but the same livestock state does not exist in the state detection results of the three modalities, the livestock state detection result is determined based on the livestock state in the state detection results of the three modalities.

7. The livestock status detection method according to any one of claims 1 to 4, characterized in that: The step of determining livestock data of at least two modes of the livestock to be detected comprises: Obtain user input corresponding to the livestock to be detected; Generate a response based on the user input to obtain a response to the question in the current round; Obtaining supplementary input corresponding to the answer to the question, and using the supplementary input as user input for the next round of the current round; Constructing the dialogue data corresponding to the livestock to be detected based on multiple rounds of user input and question responses; Multimodal data extraction is performed based on the conversation data to obtain livestock data of at least two modalities of the livestock to be detected.

8. A livestock status detection device, characterized in that: include: A data determination unit, configured to determine livestock data of at least two modalities of the livestock to be detected, wherein the at least two modalities include at least two of an audio modality, an image modality, and a text modality; A state detection unit, configured to perform livestock state detection based on the livestock data of the at least two modalities, and obtain a single detection result of each modality and a joint detection result of the at least two modalities; A decision summarizing unit is used to determine the livestock status detection result of the livestock to be detected based on the single detection result and the combined detection result.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the livestock status detection method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the livestock status detection method according to any one of claims 1 to 7 is implemented.

Images