Pet health analysis methods, devices, electronic equipment and storage media

Abstract

This application provides a pet health analysis method, device, electronic device, and storage medium. The method includes acquiring pet toileting videos, which include video data captured from multiple perspectives within a pet toileting device; identifying pet toileting behavior categories in the videos to obtain toileting behavior category information; extracting excrement image frames from the videos and performing physiological indicator analysis on these frames to obtain physiological indicator analysis results; and using a visual language model to perform health analysis based on the toileting behavior category information and the physiological indicator analysis results to obtain pet health analysis results. This method can improve the accuracy of pet health analysis.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, specifically to a pet health analysis method, device, electronic device, and storage medium. Background Technology

[0002] In recent years, with the continuous development of technology and the accelerating pace of life, people's life pressure has been constantly increasing. The companionship of pets can greatly alleviate people's mental stress and bring them physical and mental pleasure, which has led to the rapid development of the pet supplies market in recent years.

[0003] During pet ownership, issues such as diet, weather changes, pet habits, and other unforeseen circumstances can lead to pet illnesses. If pet owners fail to detect and treat these illnesses promptly, the pet's condition may worsen.

[0004] However, in related technologies, determining whether a pet is sick still relies on human judgment, and there is a lack of efficient methods for pet health analysis. Summary of the Invention

[0005] The main objective of this application is to provide a pet health analysis method, device, electronic device, and storage medium, aiming to improve the efficiency and accuracy of pet health analysis.

[0006] To achieve the above objectives, a first aspect of this application proposes a pet health analysis method, the method comprising:

[0007] Acquire pet toileting videos, which include video data taken from multiple angles inside the pet toileting device;

[0008] The pet toileting videos are analyzed to identify pet toileting behavior categories, thus obtaining toileting behavior category information.

[0009] The excrement image frames are extracted from the pet toilet video, and physiological index analysis is performed on the excrement image frames to obtain the physiological index analysis results.

[0010] Based on the toileting behavior category information and the physiological indicator analysis results, a visual language model is invoked to perform health analysis and obtain pet health analysis results.

[0011] To achieve the above objectives, a second aspect of this application provides a pet health analysis device, the device comprising:

[0012] An acquisition unit is used to acquire pet toileting videos, which include video data taken from multiple perspectives inside the pet toileting device.

[0013] The identification unit is used to identify the pet toileting behavior category from the pet toileting video to obtain toileting behavior category information.

[0014] The extraction unit is used to extract excrement image frames from the pet toilet video and perform physiological index analysis on the excrement image frames to obtain physiological index analysis results.

[0015] The analysis unit is used to call a visual language model to perform health analysis based on the toilet behavior category information and the physiological indicator analysis results, and obtain pet health analysis results.

[0016] Optionally, in some embodiments, the analysis unit includes:

[0017] The first acquisition subunit is used to acquire pet health analysis prompt text, wherein the pet health analysis prompt text indicates the generation of pet health analysis results;

[0018] The input subunit is used to input the pet health analysis prompt text, the toilet behavior category information, and the physiological indicator analysis results into the visual language model to obtain the pet health analysis results output by the visual language model.

[0019] Optionally, in some embodiments, this application also provides a model training apparatus, including:

[0020] The pre-training subunit is used to acquire multiple first pet behavior video samples and pre-train the visual language model based on the multiple first pet behavior video samples.

[0021] The second acquisition subunit is used to acquire multiple second pet behavior video samples, which include multiple pet toileting video samples, pet health analysis prompt text samples corresponding to each pet toileting video sample, and corresponding pet health problem text tags.

[0022] The first fine-tuning subunit is used to fine-tune the visual language model based on the multiple second pet behavior video samples to obtain the trained visual language model.

[0023] Optionally, in some embodiments, the extraction unit includes:

[0024] An extraction subunit is used to extract a sequence of excrement image frames from the pet toilet video;

[0025] The analysis subunit is used to perform colorimetric analysis on the excrement image frame sequence to obtain colorimetric analysis results.

[0026] A subunit is defined for determining physiological indicator analysis results based on the colorimetric analysis results and key image frames extracted from the image frame sequence.

[0027] Optionally, in some embodiments, the pet health analysis device provided in this application further includes:

[0028] The third acquisition subunit is used to acquire the treatment plan corresponding to the health abnormality when the pet health analysis result indicates that the pet has a health abnormality.

[0029] A generation subunit is used to generate a health report based on the health abnormality and the treatment plan, and send the health report to the pet management application client corresponding to the pet.

[0030] Optionally, in some embodiments, the generating subunit includes:

[0031] The acquisition module is used to acquire the historical health analysis results of the pet within a preset time period;

[0032] The sending module is used to generate a health report based on the historical health analysis results, the health abnormalities, and the treatment plan, and send the health report to the pet management application client corresponding to the pet.

[0033] Optionally, in some embodiments, the pet health analysis device provided in this application further includes:

[0034] A receiving subunit is used to receive health feedback information regarding the health abnormality returned by the pet management application client;

[0035] The second fine-tuning subunit is used to update the sample data for fine-tuning the visual language model based on the health feedback information, and to fine-tune the visual language model based on the updated sample data.

[0036] Optionally, in some embodiments, this application also provides a video capture device, including:

[0037] The fourth acquisition subunit is used to acquire environmental information around the pet toilet equipment, and to perform pet approach detection based on the environmental information to obtain the detection result;

[0038] The acquisition subunit is used to trigger multiple camera devices in the pet toilet equipment to acquire pet toilet video when it is determined based on the detection result that a pet has entered the pet toilet equipment.

[0039] Optionally, in some embodiments, the identification unit includes:

[0040] The splitting subunit is used to split the multi-angle video data contained in the pet toilet video into frames to obtain the video frame sequence corresponding to each video data.

[0041] The alignment subunit is used to perform frame-level time alignment on multiple video frame sequences and combine the multiple video frame sequences into a target video frame sequence based on the alignment result.

[0042] The identification subunit is used to identify toilet behavior categories based on the target video frame sequence to obtain toilet behavior category information.

[0043] To achieve the above objectives, a third aspect of this application provides an electronic device including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the pet health analysis method described in the first aspect.

[0044] To achieve the above objectives, a fourth aspect of the present application provides a storage medium storing a computer program that, when executed by a processor, implements the pet health analysis method described in the first aspect.

[0045] To achieve the above objectives, a fifth aspect of this application provides a computer program product comprising a computer program that is read and executed by a processor of a computer device, causing the computer device to perform the pet health analysis method described in the first aspect.

[0046] The pet health analysis method proposed in this application involves acquiring pet toileting videos, which include video data captured from multiple perspectives within a pet toileting device; identifying pet toileting behavior categories from the videos to obtain toileting behavior category information; extracting excrement image frames from the videos and performing physiological index analysis on the excrement image frames to obtain physiological index analysis results; and using a visual language model based on the toileting behavior category information and physiological index analysis results to perform health analysis and obtain pet health analysis results.

[0047] Therefore, the pet health analysis method provided in this application collects video footage of a pet using the toilet via a camera installed in the toilet equipment. It then analyzes the pet's toileting behavior and excrement state based on the video footage, and further infers the pet's health status based on a visual language model. This method requires no human intervention and can automatically perform health analysis on pets while they are using the toilet, thus improving the efficiency of pet health analysis. Furthermore, by using a trained visual language model for pet health analysis, this method can significantly improve the accuracy of pet health analysis. Attached Figure Description

[0048] The accompanying drawings are used to provide a further understanding of the technical solutions of this application and constitute a part of the specification. They are used together with the embodiments of this application to explain the technical solutions of this application and do not constitute a limitation on the technical solutions of this application.

[0049] Figure 1 A schematic diagram of the system used in the pet health analysis method provided in this application;

[0050] Figure 2 A flowchart illustrating the pet health analysis method provided in this application;

[0051] Figure 3 A schematic diagram of the structure of the visual language model provided in this application;

[0052] Figure 4 This is a schematic diagram of the data flow during the pet health analysis process in the embodiments of this application;

[0053] Figure 5 A schematic diagram of the interface of the pet management application provided in this application;

[0054] Figure 6 A schematic diagram of the pet health analysis device provided in this application;

[0055] Figure 7 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0057] Before providing a further detailed description of the embodiments of this application, the nouns and terms used in the embodiments of this application are explained, and the nouns and terms used in the embodiments of this application shall be interpreted as follows:

[0058] Vision-Language Models (VLMs) combine visual and linguistic modalities to address cross-modal understanding issues between images and text. Their core task is to understand and generate linguistic information related to visual content, or to generate corresponding visual content from linguistic information, through cross-modal representation learning. Compared to traditional unimodal models, VLMs have the advantage of combining visual and linguistic information for multifaceted reasoning and generation. For example, in image description tasks, VLMs need to understand information such as objects, scenes, and actions in an image and generate accurate descriptions using language. Conversely, in image question answering tasks, the model needs to extract relevant information from the image based on a natural language question and provide an accurate answer.

[0059] Pet medical expenses constitute a significant portion of pet ownership costs, especially for novice pet owners who, lacking sufficient pet care knowledge, are unsure how to handle pet health issues and thus require trips to the vet. The essential examinations and treatments at vets can be quite expensive, placing a considerable financial burden on pet owners. Furthermore, novice pet owners sometimes fail to detect early signs of health problems in their pets, leading to the deterioration of health due to delayed treatment. The inventors of this application discovered in their research that most overt health problems in pets are easily detected (such as external injuries), while latent health problems (such as internal diseases) are difficult to identify. Many of these latent health problems are related to the pet's gastrointestinal tract. Therefore, analyzing toilet behavior and excrement can accurately diagnose pet health issues.

[0060] Therefore, to improve the efficiency and accuracy of pet health problem analysis, this application provides a pet health analysis method. This method involves setting up a pet toileting device equipped with a camera to capture video of the pet's toileting process, and then performing pet health analysis based on the captured video. The pet health analysis method provided in the embodiments of this application will be described below.

[0061] Reference Figure 1 This is a schematic diagram of the system used in the pet health analysis method provided in this application. As shown, the system includes at least one pet toilet device 110, at least one pet management application client 130, and a pet management application server 120. The pet toilet device 110 can be a device that provides a place for pets to defecate and collects or treats pet excrement. For example, when the pet is a cat, the pet toilet device 110 can be a litter box. The pet management application client 130 is specifically a terminal that has a pet management application installed. This terminal can be a mobile terminal such as a smartphone, or a tablet, personal computer, head-mounted device, or vehicle terminal, etc. In addition to managing the pet toilet device, the pet management application can also control other pet equipment such as pet feeding devices and pet toy devices. The pet management application server 120 can be a server that provides management, data processing, and interaction services to multiple pet management application clients. The pet management application server 120 can be a high-performance computer in a network platform, a cluster of multiple high-performance computers, a portion of a high-performance computer (e.g., a virtual machine), or a combination of portions of multiple high-performance computers (e.g., virtual machines). The pet management application server 120 can be a physical server or a cloud server.

[0062] In this embodiment, pet owners can initiate a pet health analysis command through a pet management application client 130. Upon receiving the command, the pet management application server 120 determines the corresponding pet toileting device 110 based on the pet information in the command. Then, the pet toileting device 110 performs pet proximity detection. When a pet approaches and enters the device to relieve itself, multiple cameras within the device are activated to capture video footage of the pet using the toilet. The device sends the captured video footage to the pet management application server 120. The server performs toileting behavior category recognition based on the video footage and extracts excrement image frames for physiological indicator recognition. Furthermore, the recognized toileting behavior category information and physiological indicator information are input into a visual language model for health analysis, resulting in a pet health analysis result. Finally, the pet management application server 120 sends the pet health analysis result to the corresponding pet management application client 130.

[0063] It is understood that the pet management application server 120 can connect to multiple pet toilet devices 110 and multiple pet management application clients 130. The three shown in the figure are for illustrative purposes only and do not limit the number of pet toilet devices 110 and pet management application clients 130.

[0064] Reference Figure 2 This is a flowchart illustrating the pet health analysis method provided in this application.

[0065] In some embodiments, the pet health analysis method provided in this application includes, but is not limited to, steps S201 to S204.

[0066] Step S201: Obtain video of the pet using the toilet.

[0067] Specifically, the pet health analysis method provided in this application can be applied to the pet management application server 120. Pet toileting videos can be obtained in real-time from the pet toileting device 110, or from the storage area of ​​the pet management application server 120.

[0068] In this embodiment, the pet toileting device can not only be equipped with multiple cameras, but also with various sensors such as gravity sensors, infrared sensors, sound detection devices, and odor detection devices. The multiple cameras installed in the pet toileting device can capture video in real time and upload it to the corresponding storage space in the pet management application server 120 for storage. When a pet's health analysis is needed, the video captured by the pet toileting device for that pet can be retrieved from this storage space, and the toileting video can be extracted for analysis. Alternatively, in some embodiments, the pet toileting device 110 can trigger video capture upon receiving a video capture command from the pet management application server 120. For example, when a user sends a health analysis command to the pet management application server 120 through the pet management application client 130, the pet management application server 120 sends a video capture command to the pet toileting device 110 based on the health analysis command, at which point the pet toileting device 110 triggers the cameras to capture video. This avoids the problem of wasting camera energy and server storage space by capturing video when health analysis is not required. Alternatively, in some other embodiments, after receiving the health analysis command from the pet management application client 130, the pet management application server 120 does not directly send a video capture command to the pet toilet device 110. Instead, it performs pet proximity detection based on the sensing data returned by other sensors of the pet toilet device 110. When a pet is detected approaching the pet toilet device 110, a video capture command is sent to the pet toilet device 110 to begin capturing video of the pet using the toilet. This allows for more accurate video capture and avoids the waste of energy and storage resources caused by capturing invalid video.

[0069] In some embodiments, the pet health analysis instruction may include pet information, such as specifying that a health analysis be performed on one or more pets. However, in some multi-pet households, multiple pets may share a single pet toileting device. In this case, after capturing pet toileting videos, the pets in the videos captured by the pet toileting device can be further identified through pet identity verification. Then, based on the identity verification results, the pet toileting videos corresponding to the pets whose identities match the pet information included in the pet health analysis instruction can be selected from the videos captured by multiple camera devices. This avoids interference from other pets' toileting videos with the pet health analysis results, leading to more accurate pet health analysis results.

[0070] In some embodiments, the process of collecting videos of pets using the toilet includes:

[0071] Acquire environmental information around the pet toilet equipment, and perform pet approach detection based on the environmental information to obtain the detection results;

[0072] When the detection results indicate that a pet has entered the pet toileting device, multiple cameras in the pet toileting device are triggered to capture video of the pet using the toilet.

[0073] In this embodiment, the camera device in the pet toileting device can perform pet proximity detection before collecting video of the pet using the toilet. Only when the approach of the pet is detected and it is confirmed that the pet has entered the pet toileting device will the multiple cameras installed in the device be triggered to collect video of the pet using the toilet. This avoids resource waste caused by keeping the cameras constantly on, reduces the collection of noisy video, and improves the efficiency of downstream pet health analysis tasks.

[0074] Specifically, one approach is to first obtain environmental information about the area surrounding the pet toilet facility. For example, infrared detection can be used near the facility to determine if an animal is approaching. Once an animal is detected, it can then be detected whether it has entered the facility. Alternatively, video recording can be used to capture images of the area around the facility to determine if a pet is approaching. Another option is to install a gravity sensor within the facility to detect the approach of a pet.

[0075] Step S202: Perform pet toileting behavior category recognition on the pet toileting video to obtain toileting behavior category information.

[0076] After acquiring videos of pets using the toilet, the system can identify toilet behavior categories based on these videos, thus obtaining toilet behavior category information. These categories can be one or more of several pre-defined categories related to pet health information. For example, pet behavior categories may include defecation, urination, constipation (difficulty defecating), difficulty urinating, frequent toileting, and pain symptoms. Some pet toilet behavior categories can be obtained through analysis of the pet's posture in the videos; others can be obtained through analysis of the pet's posture and facial expressions; and still others can be obtained through analysis of the pet toilet video sequence and the acquisition time of each video within the sequence.

[0077] In some embodiments, pet behavior category recognition based on pet toileting videos can be performed using a pet toileting behavior category recognition model. Specifically, this model can be a multi-classification model, with a recurrent neural network (RNN) as its base for processing image sequence data. The model can be pre-trained unsupervised using a large number of pet behavior videos, enabling it to learn the ability to extract pet behavior features. Then, a series of pet toileting video samples or sequences can be constructed, each corresponding to a single toileting behavior category label. The multi-classification model is further fine-tuned in a supervised manner using the constructed training sample data to obtain the pet toileting behavior category recognition model. Finally, the model can be deployed to a pet management application server. When pet toileting behavior category recognition based on videos is needed, the model can be invoked to identify the corresponding toileting behavior category information.

[0078] In some embodiments, pet toileting videos are used to identify pet toileting behavior categories to obtain toileting behavior category information, including:

[0079] The video data from multiple angles in the pet toilet video were split into frames to obtain the video frame sequence corresponding to each video data.

[0080] Perform frame-level time alignment on multiple video frame sequences, and combine the multiple video frame sequences into a target video frame sequence based on the alignment results;

[0081] Toilet behavior category recognition is performed based on the target video frame sequence to obtain toilet behavior category information.

[0082] In related technologies, when identifying pet toileting behavior categories based on pet toileting videos, the behavior categories are typically identified separately for videos taken from different angles. Then, the confidence levels of the multiple behavior category identification results are evaluated, and the final toileting behavior category information is determined based on the confidence evaluation results. However, this method struggles to accurately correlate pet toileting behavior videos taken from different angles, resulting in low accuracy in pet behavior category identification.

[0083] In this embodiment, after acquiring pet toilet-use videos captured from multiple angles, the video data from each angle can be split into frames to obtain a video frame sequence corresponding to each video data. Each frame in the video frame sequence corresponds to a capture time. Then, frame-level time alignment can be performed on the video frames in the multiple video frame sequences based on the capture time information corresponding to each video frame. Specifically, video frames from different video frame sequences corresponding to the same capture time are determined as a video frame group corresponding to that capture time. Then, multiple video frames in the video frame group are combined to obtain the target video frame corresponding to that capture time. Specifically, combining multiple video frames in the video frame group can involve cropping overlapping parts of the video frames and keeping only one copy, then determining the splicing direction of the multiple video frames, and then splicing these multiple video frames to obtain a complete target video frame. This target video frame contains pet behavior information captured from multiple shooting angles corresponding to that capture time. Then, for each capture time, the corresponding target video frame is spliced ​​using the above method to obtain the target video frame sequence.

[0084] Furthermore, by identifying toilet behavior categories based on the target video frame sequence, accurate toilet behavior category information can be obtained.

[0085] Since pet toilet training videos consist of multiple videos taken from various angles, before performing pet toilet training behavior category recognition based on these videos, the multiple videos can be time-aligned at the image frame level. Then, pet toilet training behavior category recognition can be performed based on the time-aligned pet toilet training videos, thereby obtaining more accurate pet behavior category recognition results.

[0086] Step S203: Extract excrement image frames from the pet toilet video and perform physiological index analysis on the excrement image frames to obtain the physiological index analysis results.

[0087] The excrement image frames extracted from the pet toilet video can include image frames of excrement taken during the pet's defecation process, or image frames of the excrement location taken after the pet toilet device has buried the excrement after the pet has finished defecation.

[0088] Extracting excrement image frames from pet toilet videos can be done by first extracting image frames containing pet excrement or excrement burial objects from the pet toilet video, and then cropping the portion of pet excrement or excrement burial objects from those image frames to obtain excrement image frames.

[0089] In some embodiments, the pet toileting device includes a color-changing excrement smother; excrement image frames are extracted from pet toileting videos; and physiological index analysis is performed on the excrement image frames to obtain physiological index analysis results, including:

[0090] Extracting image frame sequences of excrement from pet toilet videos;

[0091] Colorimetric analysis was performed on the sequence of excrement image frames to obtain the colorimetric analysis results;

[0092] Physiological indicators were determined based on the results of colorimetric analysis and key image frames extracted from the image frame sequence.

[0093] This application also provides a color-changing pet waste sump, such as color-changing cat litter. When a pet uses the toilet, the waste comes into contact with the color-changing pet waste sump, and the sump changes color according to the properties of the chemical substances in the waste. For example, the pH value of the pet waste will cause the color-changing pet waste sump to change color differently. The color-changing pet waste sump in the pet toilet device can be replaced periodically to ensure the color-changing effect.

[0094] In this way, excrement image frames can be extracted from pet toileting videos. These excrement image frames are a sequence of images captured within a certain period after the pet toileting device automatically buries the excrement. This period can be determined based on the reaction time of the colorimetric reaction, for example, set to 5 minutes. After obtaining the excrement image frame sequence, further colorimetric analysis can be performed on the excrement image frame sequence to obtain the colorimetric analysis results.

[0095] Specifically, this application provides a computer vision-based intelligent color analysis system. The system employs RGB color space analysis combined with an HSV conversion algorithm to accurately identify color changes in excrement burial materials. The system can also be equipped with supplementary lighting for the camera device to ensure accurate capture of color changes in excrement burial materials under different lighting conditions. After capturing the color changes in excrement burial materials, the pH value, occult blood index, and protein index of pet excrement can be inferred.

[0096] In some embodiments, pH value, occult blood index, and protein index are determined based on image color changes of excrement burial material, which can be achieved using a deep learning model. This deep learning model can be trained using multiple image samples of different colors labeled with pH value, occult blood index, and protein index, thereby enabling the deep learning model to learn the ability to infer pH value, occult blood index, and protein index based on image color.

[0097] In addition to colorimetric analysis based on the sequence of excrement image frames in pet toileting videos, key image frames can be extracted from the videos, and then anomaly detection can be performed on the excrement based on these key image frames. These key image frames can include images of excrement captured during the pet's toileting process, images of excrement captured before the automatic burying mechanism begins after the pet has finished using the toilet, and images of the excrement's location captured after the automatic burying mechanism begins. Then, based on the shape and color of the excrement captured in the key frames, analysis is performed to determine whether there are abnormalities such as bloody stools, soft stools, loose stools, or the presence of parasites. Based on these anomalies, along with pH values, occult blood levels, and protein levels, a complete physiological indicator analysis is determined.

[0098] The specific analysis process, which analyzes the shape and color of excrement captured in keyframes to determine if there are abnormalities such as bloody stool, soft stool, loose stool, or the presence of parasites, can utilize one or more pre-trained image recognition models. For example, a bloody stool recognition model can be used, which identifies the presence of bloody stool by detecting the color of the excrement in the keyframe; a soft stool recognition model can be used, which identifies the presence of soft or loose stool by detecting the shape of the excrement in the keyframe; and a parasite recognition model can be used, which identifies the presence of parasites in pet excrement by combining the identification of discolored points in the keyframes with inter-frame differences. Discolored points in the excrement in the keyframes have a high probability of being parasites, and since some parasites are still alive when excreted, the presence of surviving parasites can be determined by the inter-frame differences between keyframes. Surviving parasites will cause inter-frame differences through their movement.

[0099] Step S204: Based on the toilet behavior category information and physiological indicator analysis results, the visual language model is called to perform health analysis and obtain the pet health analysis results.

[0100] Based on the pet toileting video identification to classify pet toileting behaviors and the analysis of pet physiological indicators, further health analysis can be performed based on the identified toileting behavior categories and the detected physiological indicators, thereby quickly obtaining accurate health analysis results. Specifically, in this embodiment, to improve the accuracy of pet health analysis results, a trained visual language model can be used to generate the pet health analysis results.

[0101] Visual language models are multimodal neural network models capable of processing both image and text data simultaneously. Their core task is to understand and generate linguistic information related to visual content, or to generate corresponding visual content from linguistic information, through cross-modal representation learning. Compared to traditional unimodal models, visual language models have the advantage of combining visual and linguistic information for multifaceted reasoning and generation. For example, in image description tasks, visual language models need to understand information such as objects, scenes, and actions in an image and generate accurate descriptions through language. The working principle of visual language models is as follows:

[0102] Visual language models typically achieve cross-modal understanding through the following steps:

[0103] 1. Image Coding: Extracting features from images using models such as Convolutional Neural Networks (CNNs) or Vision Transformers (ViTs). These features are typically low-dimensional representations of the image.

[0104] 2. Text Encoding: Using pre-trained language models, such as BERT or GPT, the input text is converted into a high-dimensional vector representation. These text representations capture the grammatical, semantic, and contextual information of the language.

[0105] 3. Cross-modal fusion: This involves fusing image and text representations. Common methods include using two-stream neural networks and attention mechanisms. In this way, the model can learn the relationships between images and text.

[0106] 4. Reasoning and Generation: Based on the fused representation, the model performs reasoning (e.g., image question answering) or generation (e.g., image description). Different tasks will use different output strategies.

[0107] Please see Figure 3 The figure shows a schematic diagram of the structure of the visual language model provided in this application. As shown, the visual language model provided in this application includes an image encoder 310, a text encoder 320, a feature fusion layer 330, and a content generation layer 340.

[0108] In this embodiment of the application, a visual language model is invoked to perform health analysis based on toilet behavior category information and physiological indicator analysis results, resulting in pet health analysis results, including:

[0109] Get the pet health analysis prompt text, which indicates the generation of the pet's health analysis results;

[0110] The pet health analysis prompt text, toilet behavior category information, and physiological indicator analysis results are input into the visual language model to obtain the pet health analysis results output by the visual language model.

[0111] In this embodiment of the application, after identifying toilet behavior category information and analyzing the pet's physiological indicators, a prompt text can be further obtained to guide the visual language model in performing health analysis based on the toilet behavior category information and physiological indicator analysis results. Then, the pet health analysis prompt text, toilet behavior category information, and physiological indicator analysis results are input into the visual language model for analysis, resulting in the pet health analysis results output by the visual language model.

[0112] Specifically, the pet health analysis prompt text can be "Please analyze whether the pet has any health abnormalities?"; the toilet behavior category information can include behavior category text, such as "abnormal defecation posture" or "frequent urination", or it can include video clips related to the behavior category text, which can be video clips extracted from pet toilet videos based on the identified toilet behavior category; the physiological indicator analysis results can also include indicator text, such as "high pH value" or "occult blood in urine", and can also include images corresponding to the physiological indicator analysis results, such as images of discolored excrement burial material and urine extracted from pet toilet videos.

[0113] Then, the pet health analysis prompt text, behavior category text, and indicator text can be input into the text encoder 320 for text feature extraction. Video clips related to the behavior categories and images corresponding to the physiological indicator analysis results can be input into the image encoder 310 for image feature extraction. The extracted text and image features are then fused through the feature fusion layer 330 to obtain fused features. Finally, a content generation layer is used to generate content based on the fused features, thus obtaining the pet health analysis results.

[0114] In some embodiments, the training process of a visual language model includes:

[0115] Multiple first pet behavior video samples were obtained, and a visual language model was pre-trained based on these samples.

[0116] Multiple second pet behavior video samples were obtained. The second pet behavior video samples included multiple pet toileting video samples, pet health analysis prompt text samples corresponding to each pet toileting video sample, and corresponding pet health problem text tags.

[0117] The visual language model was fine-tuned based on multiple second pet behavior video samples to obtain the trained visual language model.

[0118] In this embodiment, a visual language model is trained using a combination of unsupervised pre-training and supervised fine-tuning to improve its ability to analyze pet health. Specifically, a large number of first pet behavior video samples are first acquired, and then these samples are used to pre-train the visual language model. These first pet behavior video samples can include not only various pet behavior videos, such as videos of toileting, eating, playing, and resting, but also text describing pet behavior categories and text evaluating pet health. Using these samples to pre-train the visual language model improves its feature extraction and processing capabilities.

[0119] Next, multiple second pet behavior video samples are acquired. These samples include multiple pet toileting video samples, corresponding pet health analysis prompt text samples, and corresponding pet health problem text tags for each pet toileting video sample. The visual language model is then further fine-tuned using these second pet behavior video samples to obtain the trained visual language model. When fine-tuning the visual language model using the second pet behavior video samples, data such as pet toileting behavior categories and physiological indicator analysis results can be extracted from the pet toileting video samples. The image and text data from these samples are then input into the image encoder and text encoder, respectively, for feature extraction. The extracted features are then processed through a feature fusion layer and a content generation layer to obtain pet health problem prediction results. Furthermore, a loss function value is calculated based on the pet health problem prediction results and pet health problem text tags. The model parameters of the visual language model are then iteratively updated according to the loss function value until convergence, thus completing the fine-tuning of the visual language model.

[0120] Please see Figure 4This figure illustrates the data flow during pet health analysis in this embodiment. As shown, in this embodiment, the pet toilet equipment is equipped with various sensors and cameras to collect multimodal data. After collecting multimodal data on pet toileting, toileting behavior category data can be obtained by identifying the toileting behavior category in the video stream of the multimodal data, and colorimetric analysis can be performed on the colorimetric images in the multimodal data to obtain the pet's physiological indicators. Then, the behavior category data, physiological indicators, and auxiliary data in the multimodal data can be input into a visual language model for health analysis to obtain health analysis results. Auxiliary data includes the pet's weight data, age data, historical health data, and some expert knowledge on pet health. Specifically, text data of the pet's real-time weight and age data can be obtained from the pet's basic attribute information, as well as text data corresponding to the pet's medical records and corresponding medical data from the past year. In addition, text data corresponding to expert knowledge related to pet disease treatment and pet health management can also be obtained. Then, these text data, along with pet health analysis prompts, toilet behavior categories, and physiological indicators, are combined into the text input for the visual language model. This input is then used to perform pet health analysis on the visual language model, resulting in the output health analysis results.

[0121] In some embodiments, after calling a visual language model to perform health analysis based on toilet behavior category information and physiological indicator analysis results, and obtaining the pet health analysis results, the method further includes:

[0122] When the pet health analysis results indicate that the pet has a health abnormality, obtain the corresponding treatment plan for the health abnormality;

[0123] A health report is generated based on the health abnormalities and treatment plan, and then sent to the pet's corresponding pet management application client.

[0124] In this embodiment, after the pet management application server obtains the pet's health analysis results based on the acquired pet toileting video, it can determine whether the pet has any health abnormalities. If the pet has no health abnormalities, the server can generate and save a health analysis log for the pet owner to view later through the pet management application. If the pet has health abnormalities, it can further obtain the corresponding treatment plan, then generate a corresponding health report based on the health abnormality and the treatment plan, and send the health report to the pet's corresponding pet management application client.

[0125] The specific process of obtaining health abnormalities and corresponding treatment plans can be as follows: First, an abnormal description text of the health abnormality can be generated, and the video stream or key image frame corresponding to the health abnormality can be extracted from the pet toilet video. Then, the abnormal description text of the health abnormality and the corresponding video stream or key image frame can be input into the pet medical big language model for analysis to obtain the treatment plan output by the pet medical big language model.

[0126] Pet medical expenses constitute a significant portion of pet ownership costs, and veterinary fees are currently exorbitant. If pet owners lack basic knowledge and experience in pet medical care, every instance of health abnormality requires a trip to the vet, leading to high pet ownership costs. The pet health analysis method provided in this application, after quickly and accurately analyzing a pet's health based on pet toileting video data, can also provide timely and effective treatment suggestions when abnormalities are detected. This ensures that pet health problems are detected promptly and that pets receive timely treatment and temporary intervention, preventing minor health issues from escalating into major problems. It also reduces pet medical costs and improves the pet ownership experience.

[0127] Please see Figure 5 This is a schematic diagram of the interface of the pet management application provided in this application. As shown, after the pet management application client receives a health report sent by the pet management application server, it can display a health report display interface 500 on the application interface. The health report display interface 500 can display one or more health abnormality tags 510, which can include health abnormality information and corresponding treatment plans. Pet owners can further click on the health abnormality tag 510 to switch to the health treatment interface. The health treatment interface can display detailed data on the health abnormality, including video or image data related to the health abnormality, data on the basis for judging the health abnormality, and detailed recommended treatment information. Thus, if treatment at a veterinary hospital is required, this video or image data related to the health abnormality can be displayed to provide detailed pathological information so that the doctor can make an accurate diagnosis. The detailed recommended treatment information can also include recommended veterinary hospitals, recommended medications, and purchase links.

[0128] In some embodiments, a health report is generated based on health abnormalities and treatment plans, and the health report is sent to the pet management application client corresponding to the pet, including:

[0129] Obtain historical health analysis results for pets within a preset time period;

[0130] A health report is generated based on historical health analysis results, health abnormalities, and treatment plans, and then sent to the pet's corresponding pet management application client.

[0131] In this embodiment of the application, in order to further improve the accuracy of the generated health report, after generating the pet's health analysis results based on the pet's toilet video and determining the treatment plan corresponding to the pet's health analysis results, it is also possible to further obtain the pet's historical health analysis results within a certain period of time, and then generate a health report based on the historical health analysis results, health abnormalities and treatment plans.

[0132] Specifically, for example, by obtaining the pet's historical health analysis results for the past week, if the historical health analysis results indicate that the pet has no health abnormalities, then the health abnormality is determined to be a newly emerging health abnormality, and a treatment plan can be determined based on this health abnormality. If the historical health analysis results indicate that the pet already had health abnormalities, then the pet's health status (e.g., recovering, deteriorating, etc.) can be judged based on the historical health abnormalities and the currently identified health abnormality. The treatment plan can then be updated based on the health abnormality and the health status, and a health report can be generated based on the updated treatment plan and the health status data.

[0133] In some embodiments, after generating a health report based on health abnormalities and treatment plans, and sending the health report to the pet's corresponding pet management application client, the method further includes:

[0134] Receive health feedback information regarding health abnormalities returned by the pet management application client;

[0135] The visual language model is fine-tuned based on updated sample data of health feedback information, and the visual language model is fine-tuned based on the updated sample data.

[0136] In this embodiment, after a health report is displayed in the pet management application, a health feedback control can be further displayed on the health report display interface 500. Pet owners can touch the health feedback control to display the health feedback interface, where they can then fill in health feedback information. This health feedback information can be an affirmation of the displayed health report or a correction to it. For example, if a pet owner has undergone a health check by a professional veterinarian and makes an accurate diagnosis, but finds that the veterinarian's diagnosis differs from the health analysis results recognized by the pet management application server, they can upload the veterinarian's diagnosis to correct the health analysis results recognized by the pet management application server.

[0137] After correcting the health analysis results based on health feedback information, the sample data used to train the visual language model can be updated based on health feedback information and pet toilet videos. The visual language model can then be regularly updated and trained based on the updated sample data, thereby ensuring that the visual language model can be continuously optimized and further improving the accuracy of pet health analysis.

[0138] In summary, the pet health analysis method proposed in this application involves acquiring pet toileting videos, which include video data captured from multiple perspectives within the pet toileting device; identifying pet toileting behavior categories from the videos to obtain toileting behavior category information; extracting excrement image frames from the videos and performing physiological index analysis on the excrement image frames to obtain physiological index analysis results; and using a visual language model based on the toileting behavior category information and physiological index analysis results to perform health analysis, thereby obtaining the pet health analysis result.

[0139] Therefore, the pet health analysis method provided in this application collects toilet-use video footage from a camera installed in the toilet equipment, analyzes the pet's toilet-use behavior and excrement state based on the video footage, and then infers the pet's health status based on a visual language model. This method requires no human intervention and can automatically perform real-time health analysis on the pet based on its toilet-use behavior and excrement state, thus improving the efficiency of pet health analysis. Furthermore, by using a trained visual language model for pet health analysis, this method can significantly improve the accuracy of pet health analysis.

[0140] In some embodiments, when training the visual language model, pet toileting videos and pet health analysis prompt text can be directly used as input data, and pet health tags can be used as training labels to fine-tune the visual language model. Thus, when using the trained visual language model for pet health analysis, the acquired pet toileting videos can be directly input into the image encoder of the visual language model for feature extraction, and the pet health analysis prompt text can be input into the text encoder for feature extraction. The extracted features are then fused and used to generate the pet health analysis result. In some embodiments, historical pet toileting videos and health analysis data can also be acquired and input together with the currently acquired pet toileting videos into the visual language model for pet health analysis, resulting in a more accurate health analysis result.

[0141] Reference Figure 6 In some embodiments, this application also provides a pet health analysis device 600, which includes:

[0142] The acquisition unit 610 is used to acquire pet toileting videos, which include video data taken from multiple angles inside the pet toileting device.

[0143] The recognition unit 620 is used to identify the pet toileting behavior category from pet toileting videos and obtain toileting behavior category information.

[0144] Extraction unit 630 is used to extract excrement image frames from pet toilet videos and perform physiological index analysis on the excrement image frames to obtain physiological index analysis results.

[0145] Analysis unit 640 is used to call the visual language model to perform health analysis based on toilet behavior category information and physiological indicator analysis results, and obtain pet health analysis results.

[0146] Optionally, in some embodiments, the analysis unit includes:

[0147] The first acquisition subunit is used to acquire the pet health analysis prompt text, which indicates the generation of the pet's health analysis results.

[0148] The input subunit is used to input pet health analysis prompts, toilet behavior category information, and physiological indicator analysis results into the visual language model to obtain the pet health analysis results output by the visual language model.

[0149] Optionally, in some embodiments, this application also provides a model training apparatus, including:

[0150] The pre-training subunit is used to acquire multiple first pet behavior video samples and pre-train the visual language model based on these multiple first pet behavior video samples.

[0151] The second acquisition subunit is used to acquire multiple second pet behavior video samples. The second pet behavior video samples include multiple pet toileting video samples, pet health analysis prompt text samples corresponding to each pet toileting video sample, and corresponding pet health problem text tags.

[0152] The first fine-tuning subunit is used to fine-tune the visual language model based on multiple second pet behavior video samples to obtain the trained visual language model.

[0153] Optionally, in some embodiments, the extraction unit includes:

[0154] Extraction subunit, used to extract the sequence of excrement image frames from pet toilet videos;

[0155] The analysis subunit is used to perform colorimetric analysis on the sequence of excrement image frames to obtain colorimetric analysis results.

[0156] The sub-unit is determined to identify physiological indicator analysis results based on colorimetric analysis results and key image frames extracted from the image frame sequence.

[0157] Optionally, in some embodiments, the pet health analysis device provided in this application further includes:

[0158] The third acquisition subunit is used to acquire the corresponding treatment plan when the pet health analysis results indicate that the pet has a health abnormality;

[0159] The generation sub-unit is used to generate a health report based on health abnormalities and treatment plans, and send the health report to the pet's corresponding pet management application client.

[0160] Optionally, in some embodiments, generating sub-units includes:

[0161] The acquisition module is used to obtain the historical health analysis results of pets within a preset time period;

[0162] The sending module is used to generate a health report based on historical health analysis results, health abnormalities, and treatment plans, and then send the health report to the pet management application client corresponding to the pet.

[0163] Optionally, in some embodiments, the pet health analysis device provided in this application further includes:

[0164] The receiving subunit is used to receive health feedback information regarding health abnormalities returned by the pet management application client;

[0165] The second fine-tuning subunit is used to update the sample data for fine-tuning the visual language model based on health feedback information, and to fine-tune the visual language model based on the updated sample data.

[0166] Optionally, in some embodiments, this application also provides a video capture device, including:

[0167] The fourth acquisition subunit is used to acquire environmental information around the pet toilet equipment and perform pet approach detection based on the environmental information to obtain the detection results;

[0168] The acquisition subunit is used to trigger multiple camera devices in the pet toilet equipment to acquire pet toilet video when it is determined from the detection results that a pet has entered the pet toilet equipment.

[0169] Optionally, in some embodiments, the identification unit includes:

[0170] The splitting sub-unit is used to split the multi-angle video data contained in the pet toilet video into frames, and obtain the video frame sequence corresponding to each video data.

[0171] The alignment subunit is used to perform frame-level temporal alignment on multiple video frame sequences and combine the multiple video frame sequences into a target video frame sequence based on the alignment result.

[0172] The identification subunit is used to identify toilet behavior categories based on the target video frame sequence and obtain toilet behavior category information.

[0173] Reference Figure 7 , Figure 7 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0174] The processor 701 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0175] The memory 702 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 702 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 702 and is called and executed by the processor 701 using the pet health analysis method of the embodiments of this application.

[0176] The input / output interface 703 is used to implement information input and output;

[0177] The communication interface 704 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0178] Bus 705 transmits information between various components of the device (e.g., processor 701, memory 702, input / output interface 703, and communication interface 704);

[0179] The processor 701, memory 702, input / output interface 703, and communication interface 704 are connected to each other within the device via bus 705.

[0180] This application also provides a storage medium storing a computer program, which, when executed by a processor, implements the pet health analysis method provided in this application.

[0181] This application also provides a computer program product, which includes a computer program. The processor of a computer device reads and executes the computer program, causing the computer device to perform the aforementioned pet health analysis method.

[0182] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in this disclosure and the foregoing drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “including,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatuses.

[0183] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0184] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0185] It should also be understood that the various implementation methods provided in this application can be combined arbitrarily to achieve different technical effects.

[0186] The above is a detailed description of the embodiments of this disclosure. However, this disclosure is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this disclosure. All such equivalent modifications or substitutions are included within the scope defined by the claims of this disclosure.

Claims

1. A method of pet health analysis, characterized by, The method comprises: acquiring a pet toilet video, the pet toilet video comprising video data captured from multiple perspectives within a pet toilet device; performing pet toilet behavior category identification on the pet toilet video to obtain toilet behavior category information, wherein the toilet behavior category information comprises defecation behavior, urination behavior, and excretion difficulty; extracting excrement image frames from the pet toilet video and performing physiological indicator analysis on the excrement image frames to obtain physiological indicator analysis results; calling a visual language model based on the toilet behavior category information and the physiological indicator analysis results to perform health analysis and obtain pet health analysis results; The calling of the visual language model based on the toilet behavior category information and the physiological indicator analysis results to perform health analysis and obtain pet health analysis results comprises: acquiring pet health analysis prompt text indicating the generation of pet health analysis results; inputting the pet health analysis prompt text, the toilet behavior category information, and the physiological indicator analysis results into the visual language model to obtain pet health analysis results output by the visual language model.

2. The method of claim 1, wherein, The training process of the visual language model comprises: acquiring a plurality of first pet behavior video samples, pre-training the visual language model based on the plurality of first pet behavior video samples; acquiring a plurality of second pet behavior video samples, the second pet behavior video samples comprising a plurality of pet toilet video samples, pet health analysis prompt text samples corresponding to each pet toilet video sample, and corresponding pet health problem text labels; fine-tuning the visual language model based on the plurality of second pet behavior video samples to obtain a trained visual language model.

3. The method of claim 1, wherein, The pet toilet device comprises a variable-color excrement burying object, and the extraction of excrement image frames from the pet toilet video and the physiological indicator analysis of the excrement image frames to obtain physiological indicator analysis results comprise: extracting an excrement image frame sequence from the pet toilet video; performing color development analysis on the excrement image frame sequence to obtain color development analysis results; determining physiological indicator analysis results based on the color development analysis results and key image frames extracted from the image frame sequence.

4. The method of claim 1, wherein, After the calling of the visual language model based on the toilet behavior category information and the physiological indicator analysis results to perform health analysis and obtain pet health analysis results, the method further comprises: when the pet health analysis results indicate that the pet has a health abnormality, acquiring a treatment scheme corresponding to the health abnormality; generating a health report based on the health abnormality and the treatment scheme and sending the health report to a pet management application client corresponding to the pet.

5. The method of claim 4, wherein, The generation of a health report based on the health abnormality and the treatment scheme and the sending of the health report to a pet management application client corresponding to the pet comprise: acquiring historical pet health analysis results of the pet within a preset time period; generate a health report based on the historical health analysis result, the health abnormality, and the treatment scheme, and send the health report to a pet management application client corresponding to the pet.

6. The method of claim 4, wherein, After the health report is generated based on the health abnormality and the treatment scheme and sent to the pet management application client corresponding to the pet, the method further includes: receiving health feedback information returned by the pet management application client for the health abnormality; updating sample data for fine-tuning of the visual language model based on the health feedback information, and fine-tuning the visual language model based on the updated sample data.

7. The method of claim 1, wherein, The process of collecting the pet toilet video includes: obtaining environmental information around the pet toilet device, and performing pet proximity detection based on the environmental information to obtain a detection result; when it is determined based on the detection result that the pet enters the pet toilet device, triggering multiple cameras in the pet toilet device to collect a pet toilet video.

8. The method of claim 1, wherein, The process of performing pet toilet behavior category identification on the pet toilet video to obtain toilet behavior category information includes: frame splitting multiple-angle video data included in the pet toilet video respectively to obtain video frame sequences corresponding to each video data; time aligning the multiple video frame sequences at a frame level, and combining the multiple video frame sequences into a target video frame sequence according to the alignment result; performing toilet behavior category identification based on the target video frame sequence to obtain toilet behavior category information.

9. A pet health analysis device, characterized by, The pet health analysis device includes: an obtaining unit configured to obtain a pet toilet video, the pet toilet video including video data captured from multiple perspectives in a pet toilet device; an identification unit configured to perform pet toilet behavior category identification on the pet toilet video to obtain toilet behavior category information, wherein the toilet behavior category information includes defecation behavior, urination behavior, and excretion difficulty; an extraction unit configured to extract excrement image frames from the pet toilet video, and perform physiological indicator analysis on the excrement image frames to obtain physiological indicator analysis results; an analysis unit configured to call a visual language model to perform health analysis based on the toilet behavior category information and the physiological indicator analysis results, and obtain pet health analysis results. The process of calling the visual language model to perform health analysis based on the toilet behavior category information and the physiological indicator analysis results to obtain pet health analysis results includes: obtaining pet health analysis prompt text, the pet health analysis prompt text indicating generation of pet health analysis results; inputting the pet health analysis prompt text, the toilet behavior category information, and the physiological indicator analysis results into the visual language model to obtain pet health analysis results output by the visual language model.

10. An electronic device comprising a memory and a processor, the memory storing a computer program, characterized in that, The processor executes the computer program to implement the pet health analysis method in any one of claims 1 to 8.

11. A storage medium storing a computer program, characterized in that, The computer program is executed by the processor to implement the pet health analysis method in any one of claims 1 to 8.

12. A computer program product comprising a computer program, characterized in that, The computer program is read and executed by a processor of a computer device, so that the computer device performs the pet health analysis method of any one of claims 1 to 8.

Images