System and method for automatically interpreting mood of dog using machine learning

Abstract

A computer-implemented method for detecting one or more emotions of one or more pets is disclosed. The method includes: receiving, by one or more processors, image data from at least one user device, wherein the image data includes one or more frames; detecting, by the one or more processors, at least one pet profile comprising at least one pet in the one or more frames; detecting, by the one or more processors, one or more emotions of the at least one pet based on the at least one pet profile; and displaying, by the one or more processors, the one or more emotions on at least one user interface of the user device.

Description

[0001] Cross Reference to Related Applications

[0002] This patent application is a continuation of and claims the priority benefit of U.S. Application No. 63 / 516,586, filed July 31, 2023, the entirety of which is incorporated by reference herein. TECHNICAL FIELD

[0003] Various embodiments of the present disclosure generally relate to detecting one or more emotions of one or more pets. In some embodiments, the present disclosure relates to systems and methods for detecting one or more emotions of one or more pets using a machine learning model based on one or more videos. BACKGROUND

[0004] Pets express their emotions differently than humans. A pet owner can desire to understand how a pet is feeling in order to determine whether lifestyle adjustments are needed to change the pet’s emotions. Additionally, a pet owner can desire to understand how their pet expresses emotions through body language in order to interpret how their pet is feeling in a particular situation. For example, a pet owner can desire to understand when their pet is feeling scared in order to help the pet feel safer and prevent it from harming others. Additionally, a pet owner can desire to understand the emotional state of their pet in order to determine when to seek veterinary support for a pet’s illness or injury. A pet owner can find it challenging to assess a pet’s physical features to determine the pet’s emotions because the pet owner can not know which physical features need to be assessed. Additionally, a pet owner can not be familiar with behaviors and indicators that reflect a pet’s emotions, resulting in the pet owner being unable to properly assess the pet’s feelings.

[0005] Traditional methods can include a pet owner assessing features of a pet and determining the pet’s emotions based on such features. However, such traditional methods can be challenging for a pet owner because the pet owner can not know which pet features need to be assessed. Traditional methods can not consider a pet’s breed when analyzing features of the pet. Additionally, many pet owners also lack experience, pet behavior expertise, and breed-specific knowledge when analyzing features of a pet to determine the pet’s emotions.

[0006] The present disclosure aims to address the aforementioned challenges. The background description provided herein is intended to be a summary of information and is not intended to be a comprehensive discussion of the background of the disclosure. The discussion of background information in this section is intended to provide context for the present disclosure. Unless otherwise stated in this document, no aspect of the material described in this section is prior art to the claims of the present disclosure, and all aspects of the material described in this section are included in the disclosure only as background information. SUMMARY

[0007] According to certain aspects of the present disclosure, methods and systems for detecting one or more emotions of one or more pets are disclosed.

[0008] In one aspect, exemplary embodiments of a method for detecting the emotions of one or more pets are disclosed. The method may include receiving image data from at least one user device by one or more processors, wherein the image data includes one or more frames. The method may include detecting at least one pet outline including at least one pet in the one or more frames by one or more processors. The method may include detecting one or more emotions of at least one pet based on the at least one pet outline by one or more processors. The method may include displaying one or more emotions on at least one user interface of the user device by one or more processors.

[0009] In another aspect, an exemplary embodiment of a computer system for detecting the emotions of one or more pets is disclosed. The computer system includes at least one memory storing instructions and at least one processor configured to execute the instructions to perform operations. The operations may include receiving image data from at least one user device, wherein the image data includes one or more frames. The operations may include detecting at least one pet silhouette including at least one pet in the one or more frames. The operations may include detecting one or more emotions of at least one pet based on the at least one pet silhouette. The operations may include displaying one or more emotions on at least one user interface of the user device.

[0010] In another aspect, a non-transitory computer-readable medium storing instructions, which, when executed by a processor, cause the processor to perform operations for detecting the emotions of one or more pets. The operations may include receiving image data from at least one user device, wherein the image data comprises one or more frames. The operations may include detecting at least one pet silhouette comprising at least one pet in the one or more frames. The operations may include detecting one or more emotions of at least one pet based on the at least one pet silhouette. The operations may include displaying one or more emotions on at least one user interface of the user device.

[0011] It should be understood that the foregoing general description and the following detailed description are exemplary and illustrative only, and do not limit the claimed disclosed embodiments.

[0012] Brief description of the attached figures

[0013] The accompanying drawings (which are included in and constitute a part of this specification) illustrate various exemplary embodiments and, together with the specification, serve to explain the principles of the disclosed embodiments.

[0014] FIG. 1 A flowchart illustrating an exemplary embodiment for detecting the emotions of one or more pets, according to one or more embodiments, is depicted.

[0015] FIGS. 2A-2C An example interface for analyzing pet image data to determine emotion is depicted in accordance with one or more embodiments.

[0016] FIG. 3 An example environment that can be used with the techniques described herein is depicted in accordance with one or more embodiments.

[0017] FIG. 4 An example of a computing device that can perform the techniques described herein is depicted in accordance with one or more embodiments. DETAILED DESCRIPTION

[0018] According to certain aspects of the present disclosure, methods and systems for detecting the emotion of one or more pets are disclosed. Conventional techniques can not be applicable because conventional techniques can rely on pet owners to assess the features of a pet and determine the emotion of the pet based on such features. Moreover, conventional techniques can not take into account the breed type and conformation of the pet when analyzing the features of the pet. Accordingly, there is a need for improved techniques related to detecting the emotion of a pet.

[0019] In order to improve the emotional well-being of their pets, pet owners can need to first be able to accurately identify the emotional state of the pet. However, pet owners can not be able to accurately identify different pet behaviors or can interpret behaviors differently based on their own past experiences or relationships with individuals, pets, or other pets. While owners can be able to identify more extreme emotional reactions, owners can not be able to successfully identify more subtle reactions, such as mild fear and stress. Moreover, different experiences (e.g., receiving pet behavior training, engaging in different pet-related professions, growing up with a pet, or growing up in a culture with different views on pets) have been shown to influence people’s assessment of pet emotion. Pet caregivers’ inability to accurately interpret a range of extreme and subtle pet emotional states can result in compromised pet welfare and increase the risk of negative human-pet interactions. These events can compromise the human-pet emotional bond and can result in pets being relinquished. In order to optimize pet welfare and enhance the quality of human-animal interactions, it is necessary to provide services that enable pet owners to learn to accurately assess the feelings of their pets and equip them to respond appropriately. To achieve this goal, a model that can automatically identify pet emotional states from image data (e.g., video clips) would be beneficial. This would allow pet owners to gain additional information about their pet’s feelings in different situations or interventions, providing information for decisions on whether to continue or intervene with appropriate actions, as well as assessing how the pet’s way of reacting changes over time. This can include positive situations that the pet owner or caregiver would like to know about the pet’s level of enjoyment (e.g., giving different food / treats), as well as potential negative situations that can cause the pet to become tense (e.g., grooming or interacting with strangers).

[0020] Automatically identifying pet emotional states from video clips can be beneficial in many other scenarios. Traditionally, the assessment of pet behavior and emotions for research purposes is performed by manually coding behaviors in video clips. Manually coding behaviors in videos requires a large amount of human resources and is at risk of data quality degradation due to coder under-training or coder fatigue. Automation of emotion recognition can allow large-scale monitoring and / or monitoring of pet emotional well-being without human observers. For example, in conjunction with the installation of video monitoring devices, such a model would allow reliable assessment and monitoring of pet emotional well-being in scenarios such as veterinary clinics and hospitals, shelters, boarding kennels, day care centers, research institutions, or home environments where the owner is not present. This can allow identification of pets at risk of a decrease in well-being and implementation of early interventions to address the issue. Furthermore, such a model can improve the feasibility of studies aimed at better understanding the current state of pet emotional well-being and the effects of different interventions.

[0021] Accordingly, there is a need for techniques that analyze attributes of a pet (e.g., motion, posture) to determine the pet’s emotions. The techniques disclosed herein can analyze image data of a pet to determine the pet’s emotions. Such techniques can also include analyzing the breed of the pet, which can result in more accurate determinations of the pet’s emotions. Furthermore, these techniques can utilize machine learning models in performing the analysis, thereby improving accuracy and efficiency.

[0022] As will be discussed in greater detail below, in various embodiments, systems and methods for receiving image data from at least one user device are described, where the image data includes one or more frames. The systems and methods can include detecting, in the one or more frames, at least one pet outline including at least one pet. The systems and methods can include detecting, based on the at least one pet outline, one or more emotions of the at least one pet. The systems and methods can include displaying, on at least one user interface of the user device, the one or more emotions.

[0023] Although the terms used below are used in connection with the detailed description of certain specific examples of the present disclosure, these terms can be interpreted in their most generous reasonable manner. In fact, certain terms can even be emphasized below; however, any term intended to be interpreted in any limiting manner will be explicitly and specifically defined in the detailed description of the specific embodiments. Both the foregoing general description and the following detailed description are exemplary and explanatory only, and are not restrictive of the features.

[0024] In this disclosure, the term “based on” means “based, at least in part, on.” The singular forms “a,” “an,” and “the” include plural referents unless the context clearly dictates otherwise. The term “exemplary” means “an example of’ and not “ideal.” The terms “comprise,” “contain,” “comprise,” “comprise,” or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. The term “or” is used in the disjunctive sense, so that “at least one of A or B” includes (A), (B), (A and A), (A and B), and the like. Relative terms, such as “substantially” and “generally,” are used to indicate that the stated or understood value can vary by ±10%.

[0025] As used herein, the term “user” or the like generally encompasses one future pet owner, caregiver, a plurality of future pet owners, one pet owner, and / or a plurality of pet owners. The term “pet” or the like generally encompasses a domesticated animal, such as a domesticated canine, feline, rabbit, mink, horse, cow, and the like. In exemplary embodiments, “pet” can refer to a canine.

[0026] Further, the following techniques can be applied to monitor the health of dogs in shelters, veterinary clinics, beauty institutions, boarding kennels, dog daycares, and / or research institutions using video monitoring equipment. Further, the automated model can support the quantification of pet emotions for research purposes. For example, the automated model can allow for the assessment of the emotional health of a pet, as well as the assessment of the impact of different interventions.

[0027] Example method for detecting emotion of one or more pets

[0028] FIG. 1 An exemplary method 100 for detecting one or more emotions of one or more pets is shown in accordance with one or more embodiments. Notably, the method 100 can be performed by one or more processors of a server in communication with one or more user devices and other external systems via a network. However, it should be noted that the method 100 can be performed by any one or more of the server, one or more user devices, or other external systems.

[0029] The method can include receiving, by one or more processors, image data from at least one user device, where the image data includes one or more frames (step 102). The one or more frames can correspond to one or more still images of the image data. In some embodiments, the image data can include video data. In some embodiments, the video data can have a threshold amount of duration. For example, the threshold amount can be 5 seconds, where the video can have a maximum length of 5 seconds. Further, for example, the one or more frames can include a segment of time (e.g., a 5 second segment) of the video data. The image data can include at least one video of a pet, where the one or more frames can correspond to one or more still images of the video. The image data can also include one or more pixels. The user device can have captured and / or stored the image data. For example, a user (e.g., a pet owner) can have recorded one or more videos of a pet using the user device. In some embodiments, the image data can have been collected and / or stored by one or more mobile applications. In some embodiments, the image data can include more than one video segment. Additionally or alternatively, at least one data store can store the image data.

[0030] In some embodiments, the method can include receiving pet data from one or more devices. For example, the one or more devices can include sensors, wearable devices, thermometers, medical devices, accelerometers, etc. Further, for example, the pet data can include vocalizations, heart rate, heart rate variability, skin temperature, and activity data of a pet. In some embodiments, the pet data can include a timestamp to indicate a time at which the pet data was generated.

[0031] The method can include detecting, by one or more processors, at least one pet contour including at least one pet in the one or more frames (step 104). In some embodiments, a convolutional neural network (CNN) model can be trained to detect the contour of the at least one pet. The pet contour can include a portion of the pet (e.g., a paw) or the entire pet (e.g., the entire body of a dog). In some embodiments, each frame of the image data can be analyzed to detect the at least one pet contour. However, not all frames can include a pet contour. In some embodiments, a frame can include two pets, where the pet contour can include one of the two pets. The other pet can be considered part of the background of the frame.

[0032] The method can include identifying, by the one or more processors, at least one mask corresponding to a background surrounding at least one pet outline of the one or more frames. For example, a pet outline can outline a pet in a frame, where the outline can appear a specified distance from the pet image (e.g., 2 millimeters from the pet). The mask can correspond to the background, where everything in the frame other than the pet outline can be included in the mask. The method can also include updating, by the one or more processors, the one or more frames, the update including removing the background from the one or more frames using the at least one mask. The mask can be used to remove the original background of the frame. For example, the mask can be used to add a color (e.g., the same color) to each pixel of the background, thereby removing the background and isolating the pet outline. The method can also include creating, by the one or more processors, new image data based on the one or more updated frames. For example, the new image data can include the frames with the background removed. The new image data can include a video composed of the frames, where the video has isolated the pet from the received image data. In some embodiments, each pet in the image data can have corresponding new image data, where the new image data can include the isolated pet. For example, the received image data can include a video of three dogs running. The new image data can include three separate videos, where each video can correspond to one of the dogs. The method can also include annotating such new image data. For example, the background-masked video can include annotations indicating specific behaviors and / or emotions of the pet.

[0033] The method can include dilating, by the one or more processors, the at least one pet outline to maximize coverage of the at least one pet outline. For example, dilating the pet outline can include expanding the pet outline by one or more pixels and including some pixels in the background as part of the pet outline. Further, dilating the pet outline can help to account for model errors and / or maximize coverage of the pet outline.

[0034] The method can include detecting, by the one or more processors, one or more emotions of the at least one pet based on the at least one pet silhouette (step 106). The one or more emotions can include happy, sad, frustrated, curious, fearful or anxious, aggressive emotions with fear, aggressive emotions without fear, predatory emotions, and ambivalent emotions. In some embodiments, the detecting can be based on the received pet data. In some embodiments, the detecting can be performed by one or more trained machine learning models. In some embodiments, the detecting can include receiving, by the trained machine learning model, the at least one pet silhouette. For example, the machine learning model can receive the at least one pet silhouette as input. In some embodiments, the trained machine learning model can also receive the pet data. The trained machine learning model can have been previously trained to learn a relationship between a pet in a pet silhouette and one or more pet emotions. Upon receiving the at least one pet silhouette and / or the pet data, the trained machine learning model can determine the one or more emotions. Further, in some embodiments, training the machine learning model can include receiving training data, such as one or more training pet silhouettes and one or more training pet emotions. The machine learning model can then analyze the training data to determine and store one or more relationships between a pet in a pet silhouette and one or more emotions.

[0035] In some embodiments, the method can include classifying, by the one or more processors, the at least one pet included in the at least one pet silhouette as corresponding to at least one pet breed. The at least one pet breed can correspond to a breed of the pet. For example, the pet breed can include Afghan Hound, Airedale, Akita, Alaskan Malamute, Basset Hound, Beagle, Belgian Sheepdog, Black and Tan Coonhound, Border Collie, Border Terrier, Borzoi, Boxer, Bulldog, Bulldog, Cairn Terrier, Chihuahua, Chow Chow, Cocker Spaniel, Collie, Corgi, Dachshund, Dalmatian, Doberman Pinscher, English Setter, Fox Terrier, German Shepherd, Golden Retriever, Great Dane, Greyhound, Brussels Griffon, Irish Setter, Irish Wolfhound, King Charles Spaniel, Labrador Retriever, Lhasa Apso, Mastiff, Newfoundland, Old English Sheepdog, Poodle, Pekingese, Pointer, Pomeranian, Pug, Rottweiler, Saint Bernard, Saluki, Samoyed, Siberian Husky, Skye Terrier, Springer Spaniel, West Highland White Terrier, Yorkshire Terrier, mixed breed, etc.

[0036] In some embodiments, the classification can be performed by one or more trained machine learning models. In some embodiments, the classification can include receiving, by the trained machine learning model, at least one pet silhouette. For example, the machine learning model can receive the at least one pet silhouette as input. The trained machine learning model can include a breed classifier. The trained machine learning model can receive and process the at least one pet silhouette to determine at least one pet breed. The classification can include analyzing, by the trained machine learning model, the at least one pet silhouette to determine at least one body feature (e.g., a posture) of the at least one pet. For example, the trained machine learning model can have previously been trained to learn a relationship between a pet in a pet silhouette and a pet breed. The classification can include determining, by the trained machine learning model, that the at least one pet corresponds to the at least one pet breed based on the at least one body feature. The body feature can correspond to a body attribute of the breed (e.g., short legs), a motion of the breed (e.g., wide strides), and / or a posture of the breed. For example, the trained machine learning model can have been trained to associate short legs, short strides of a pet with a Dachshund.

[0037] In some embodiments, the method can include analyzing, by the one or more processors, at least one pet silhouette in each frame of the one or more frames. For example, the trained machine learning model can repeat the above process for each frame. The method can further include determining, by the one or more processors, that the at least one pet breed occurs most frequently in the one or more frames. The classification of each of the pet silhouettes can be analyzed, where the classification that occurs most frequently can be determined to be the pet breed. For example, the image data can include 10 frames, where the trained machine learning model can have classified the pet silhouette in 6 frames as a Golden Retriever and classified the pet silhouette in 4 frames as a Labrador Retriever. Accordingly, the pet can be classified as a Golden Retriever because the Golden Retriever classification occurs most frequently.

[0038] Additionally or alternatively, in some embodiments, training the machine learning model can include receiving training data, such as one or more training pet silhouettes and one or more corresponding training pet breeds. The machine learning model can then analyze the training data to determine and store one or more relationships between a pet in a pet silhouette and a pet breed.

[0039] In some embodiments, the method can include associating, by the one or more processors, the at least one pet breed with at least one breed cluster, where each of the at least one breed cluster includes a plurality of emotion detectors. The breed cluster can include a plurality of breeds, where the plurality of breeds can share at least one physical and / or emotional attribute. For example, the 544 breeds can be divided into 5 clusters based on various physical characteristics. Further, for example, the breeds in each cluster can share a physical attribute, such as a similar body type, tail posture, ear type, etc. The emotion detectors can include one or more relationships between one or more emotions and one or more physical attributes. In some embodiments, the physical attributes can correspond to a pet’s posture and / or movement. The one or more physical attributes can include: high set ears, low set ears, laid back ears, high set tail, low set tail, wagging tail, etc. In some embodiments, the one or more relationships can be breed specific. For example, if the breed is a Labrador Retriever, a low tail posture can be associated with a fearful or anxious emotion. However, if the breed is a Greyhound, a low tail posture can be associated with a happy or relaxed emotion. In some embodiments, the pet can be associated with the at least one breed cluster based on the at least one pet profile without classifying the at least one pet included in the at least one pet profile as corresponding to the at least one pet breed. For example, the method can include analyzing the at least one pet profile to determine the associated at least one breed cluster.

[0040] In some embodiments, the method can include detecting, by the one or more processors, one or more emotions of the at least one pet based on the plurality of emotion detectors of the at least one breed cluster. The one or more emotions can include happy, sad, depressed, curious, fearful or anxious, aggressive emotion with fear, aggressive emotion without fear, predatory emotion, and ambivalent emotion. The detecting can include analyzing the emotion detectors of the at least one breed cluster. In some embodiments, the detecting can include analyzing the pet data to detect the one or more emotions. In some embodiments, the detecting can include analyzing the plurality of emotion detectors, where a particular combination of emotion detectors can indicate a particular emotion. In some embodiments, the detecting can include detecting the one or more emotions over a period of time. For example, a certain emotion can be detected in a number of frames, but not in all frames of the image data. Additionally or alternatively, more than one emotion can be detected, where each emotion can be detected over a particular period of time corresponding to a particular number of frames.

[0041] In some embodiments, detecting one or more emotions can be performed by a convolutional neural network (CNN) model or a Transformer-based model. A combination of CNN models and Transformer-based models can reduce spatial redundancy and capture complex global dependencies. CNN models can generate rich spatial features, while Transformer-based models can capture temporal relationships among such spatial features. Temporal relationships can help capture emotions expressed by a pet through actions / body movements. For example, a CNN model can capture actions such as a pet’s tail movement, running, eating, jumping, etc. through temporal relationships among one or more frames, and then the CNN model can translate such actions into one or more emotions.

[0042] The method can also include displaying, by the one or more processors, the one or more emotions on at least one user interface of the user device (step 108). For example, the user device can display an emotion (e.g., “happy”) with a visual presentation (e.g., a smiling face). In some embodiments, the user can be able to respond to the displayed emotion to indicate whether the user believes the pet can be experiencing the emotion. Additionally, displaying the one or more emotions can also include displaying at least one emotion confidence level, at least one pet breed, and / or at least one pet breed confidence level. For example, the method can include determining a confidence level corresponding to the one or more emotions, where the confidence level can indicate a confidence of the emotion determination. Additionally or alternatively, for example, the method can include determining a confidence level corresponding to the breed, where the confidence level can indicate a confidence of the breed determination. In some embodiments, the machine learning model can indicate the confidence of the emotion determination and / or the breed determination.

[0043] The method can include storing, by the one or more processors, the at least one pet and the one or more emotions in one or more data stores. For example, the data stores can be located within the user device and / or located within an external system (e.g., data stores of a system performing the analysis). Additionally, storing can also include storing details about the pet’s emotions. For example, additional details can include information about the pet’s location (e.g., the pet is outside), weather conditions (e.g., the weather is sunny), posture, and / or movement. In such an example, the information can be collected via user input, GPS data, data collected from a wearable device, etc.

[0044] In some embodiments, the method can include creating, by the one or more processors, a customized plan for the at least one pet based on the one or more emotions. The customized plan can be created to improve the pet’s emotions. For example, the customized plan can include more outdoor time, indoor time, etc. In some embodiments, the customized plan can include educational materials that describe how the pet owner can identify the one or more emotions and / or general steps to take to address the one or more emotions. The customized plan can also include other resources for the pet owner. The method can also include displaying, by the one or more processors, the customized plan on at least one user interface of the user device. Displaying the customized plan can include displaying one or more tasks for the pet to complete.

[0045] In some embodiments, the method can include determining, by the one or more processors, at least one recommendation based on the one or more emotions, where the at least one recommendation includes at least one physical activity. For example, if the pet has an unhappy emotion, the system can recommend that the pet go for a 10-minute walk each day. In some embodiments, the at least one recommendation can correspond to a particular pet product (e.g., food, toys, etc.). The method can also include displaying, by the one or more processors, the at least one recommendation on at least one user interface of the user device.

[0046] In some embodiments, the method can include comparing, by the one or more processors, the one or more emotions to at least one prior emotion of the at least one pet. For example, the method can include retrieving at least one stored prior emotion corresponding to the at least one pet. The system can analyze the emotion and the prior emotion to determine similarities and / or differences in the emotions and their associated contexts. The method can also include displaying, by the one or more processors, information corresponding to the comparison on at least one user interface of the user device. For example, the system can track the pet’s emotional state over a period of time and / or provide updates (e.g., daily, weekly, monthly, etc.) on the pet’s progress.

[0047] In some embodiments, the method can include displaying, by the one or more processors, at least one visual dashboard. For example, the visual dashboard can include a visual presentation of the pet’s emotional progress, a recommendation for the pet’s physical activity, and / or provide real-time feedback to the pet owner.

[0048] In some embodiments, the analysis can be provided as a subscription service, where pet owners can pay a monthly or yearly fee to access the functionality of the tool and receive personalized guidance and recommendations. Additionally or alternatively, the tool can also be provided on a pay-per-use basis, where pet owners can pay a fee for each individual use of the tool. In some embodiments, the tool can provide consultation and training for veterinarians on how to use and interpret the data generated by the tool. In some embodiments, the tool can recommend professional services and / or products to pet owners. In some embodiments, the tool can integrate with other systems, such as electronic health records (dog collars, accelerometers, etc.) and / or telemedicine. In some embodiments, the system can be implemented in different scenarios, where the tool can analyze the emotion differently depending on the scenario (e.g., dog daycares, grooming establishments, veterinary clinics, kennels, etc.). In some embodiments, the system can be integrated into a "smart home," where pet owners can monitor the emotional well-being of their dogs via an interconnected ecosystem.

[0049] Although FIG. 1 While example blocks of the example method 100 are illustrated, in some implementations, the example method 100 can include additional blocks, fewer blocks, different blocks, or differently arranged blocks than those depicted in FIG. 1 Additionally or alternatively, two or more of the blocks of the example method 100 can be performed in parallel.

[0050] Example interface for determining emotion

[0051] FIGS. 2A-2C An example interface for analyzing pet image data to determine emotion is depicted in accordance with one or more embodiments. FIG. 2A A frame 200 is illustrated, which can include a pet outline and a mask. A machine learning model (e.g., a CNN-based model) can be trained to detect the outline of one or more pets in each frame of an input video, as well as to track each of the one or more pets individually. The pet outline can include a pet and a radius range of pixels around the pet. In some embodiments, the radius can include a distance threshold from the pet in the frame (e.g., a minimum of 3 millimeters from the pet). The detected outline or mask can be dilated to account for model errors and maximize the coverage of the pet. The mask can be used to remove the original background and add a color (e.g., the same color) to each pixel of the background. Since each unique pet can be tracked in the video, all other pets besides the currently tracked pet can be part of the background. For example, this tracking and masking process can enable the creation of a separate masked video for each unique pet in the input video.

[0052] FIG. 2BA frame 202 is shown, which can display the recognized breed of the pet (e.g., “Breed: Labrador Retriever”). In some embodiments, the system can also display a confidence level corresponding to the breed classification (e.g., “0.75”). A machine learning model (e.g., a CNN-based classification model) can be trained to recognize the breed of the dog in the input frame. For example, each breed of pet can be associated with a breed cluster. This association can be trained on data learned by the machine learning model, or in some cases, this association can be based on a manually encoded correspondence between breed clusters and breeds. In some embodiments, the pet outlined in each frame of the video can be sent as input to a breed classifier, where each unique dog can be assigned a breed that can have a maximum count across all frames. For example, 544 breeds can be divided into 5 clusters based on various physical features. Creating clusters can allow for the implementation of removing the influence of the dog’s appearance on the emotion analysis by creating a separate emotion detector for each breed cluster.

[0053] FIG. 2C A frame 204 is shown, which can display the detected emotion of the pet (e.g., “Emotion: Happy”). In some embodiments, the system can also display a confidence level corresponding to the detected emotion (e.g., “0.77”). A CNN-and-Transformer-based model can be trained to detect the emotion of the dog in the video, where the background can have been masked. The hybrid model based on CNN and Transformer can help reduce spatial redundancy and capture complex global dependencies. The CNN part of the model can generate rich spatial features, while the Transformer part can capture the temporal relationship between such features. In some embodiments, a semi-supervised knowledge distillation training method can be used to train different models for each breed cluster, which can allow for further improvement in overall accuracy.

[0054] Example environment and example device

[0055] FIG. 3 An example environment 300 that can be used with the techniques described herein is depicted. One or more user devices 305, one or more external systems 310, and one or more server systems 315 can communicate over a network 301. As will be discussed in further detail below, the one or more server systems 315 can communicate with one or more other components of the environment 300 over the network 301. The one or more user devices 305 can be associated with a user.

[0056] In some embodiments, components of environment 300 are associated with a common entity, such as a pet adoption service, a pet breeder, a pet advertiser, a pet service, a veterinarian, a clinic, an animal specialist, a research center, a pet owner, etc. In some embodiments, one or more components of the environment are associated with a different entity. Systems and devices of environment 300 can communicate in any arrangement.

[0057] User device 305 can be configured to enable a user to access and / or interact with other systems in environment 300. For example, user device 305 can be a computer system, such as a desktop computer, a mobile device, a tablet, etc. In some embodiments, user device 305 can include one or more electronic applications, such as programs, plug-ins, browser extensions, etc., installed on a memory of user device 305.

[0058] User device 305 can include a display / user interface (UI) 305A, a processor 305B, a memory 305C, and / or a network interface 305D. User device 305 can execute an operating system (O / S) and at least one electronic application (each stored in memory 305C) through processor 305B. The electronic application can be a desktop program, a browser program, a web client, or a mobile application (which can also be a browser program in a mobile O / S), a proponent-specific program, system control software, system monitoring software, software development tools, etc. For example, environment 300 can extend information on a web client that can be accessed through a web browser. In some embodiments, the electronic application can be associated with one or more other components in environment 300. The application can manage memory 305C (e.g., a database) to transmit streaming data to network 301. Display / UI 305A can be a touchscreen or a display with other input systems (e.g., a mouse, a keyboard, etc.) so that a user can interact with the application and / or the O / S. Network interface 305D can be a TCP / IP network interface for, e.g., Ethernet or wireless communication with network 301. Processor 305B, when executing the application, can generate data; and / or, receive user input from display / UI 305A; and / or, receive / transmit messages from / to server system 315; and, can perform one or more operations further before providing output to network 301.

[0059] The external system 310 can be, for example, one or more third-party and / or auxiliary systems that integrate with and / or communicate with the server system 315 in performing various image capture and / or sentiment analysis tasks. For example, the external system 310 can include one or more services that include storage of pet images (e.g., pet videos). Further, for example, such pet images can correspond to videos of one or more pets. The external system 310 can communicate with other devices or systems in the environment 300 over one or more networks 301. For example, the external system 310 can communicate with the server system 315 via an API (application programming interface) accessed over one or more networks 301, and can communicate with the user device 305 via a web browser accessed over one or more networks 301.

[0060] In various embodiments, the network 301 can be a Wide Area Network (“WAN”), a Local Area Network (“LAN”), a Personal Area Network (“PAN”), and the like. In some embodiments, the network 301 includes the Internet, and information and data provided between the various systems is transmitted online. “Online” can mean connecting or accessing source data or information from a remote location from other devices or networks connected to the Internet. Alternatively, “online” can refer to connecting or accessing a network via a mobile communication network or device (wired or wirelessly). The Internet is a global system of computer networks - in this network, a party connected to the network can obtain information from any other computer and communicate with other computers or devices. The most widely used part of the Internet is the World Wide Web (usually abbreviated as “WWW” or called “Web”). “Website pages” generally encompass locations, data stores, and the like, for example, hosted and / or operated by a computer system for online access, and can include data configured to cause a program (e.g., a web browser) to perform operations (e.g., send, receive, or process data, generate visual displays and / or interactive interfaces, and the like).

[0061] The server system 315 can include an electronic data system, for example, a computer-readable memory, such as a hard drive, a flash drive, a disk, and the like. In some embodiments, the server system 315 includes and / or interacts with an application programming interface for exchanging data with other systems (e.g., one or more of the other components of the environment).

[0062] The server system 315 can include a database 315A and at least one server 315B. The server system 315 can be a computer, a computer system (e.g., a rack-mounted server), and / or a cloud service computer system. The server system can store or access the database 315A (e.g., a database hosted on a third-party server or in memory 315E). The server can include a display / UI 315C, a processor 315D, a memory 315E, and / or a network interface 315F. The display / UI 315C can be a touchscreen or a display with other input systems (e.g., a mouse, a keyboard, etc.) to allow an operator of the server 315B to control the functions of the server 315B. The server system 315 can execute an operating system (O / S) and at least one servlet program instance (each stored in the memory 315E) by the processor 315D.

[0063] Although depicted as separate components in FIG. 3 , it should be understood that, in some embodiments, a component or a portion of a component in the environment 300 can be integrated with or incorporated into one or more other components. For example, a portion of the display 315C can be integrated into the user device 305, etc. In some embodiments, operations or aspects of one or more of the components discussed above can be distributed among one or more other components. Any suitable arrangement and / or integration of the various systems and devices of the environment 300 can be used.

[0064] In the previous and following methods, various actions can be described as being performed or implemented by a component (e.g., the server system 315, the user device 305, or a component thereof) in FIG. 3 . However, it should be understood that, in various embodiments, the various components of the environment 300 discussed above can execute instructions or perform actions (including the actions discussed above). Actions performed by a device can be considered as being performed by a processor, an actuator, etc. associated with the device. Further, it should be understood that, in various embodiments, various steps can be added, omitted, and / or rearranged in any suitable manner.

[0065] Generally, any of the processes discussed in the present disclosure are contemplated to be computer-implemented, and thus, any of the processes illustrated in FIGS. 1-2C may be performed by a computer system (e.g., the server system 315, the user device 305, etc.). Further, it should be understood that, in various embodiments, various steps can be added, omitted, and / or rearranged in any suitable manner. FIG. 3one or more processors of any system or device in the environment 300, as described above. Processes or process steps executed by one or more processors can also be referred to as operations. The one or more processors can be configured to perform such processes by accessing instructions (e.g., software or computer-readable code) that, when executed by the one or more processors, cause the one or more processors to perform the processes. The instructions can be stored in a memory of a computer system. The processors can be central processing units (CPUs), graphics processing units (GPUs), or any suitable type of processing units.

[0066] A computer system (e.g., a system or device implementing the processes or operations in the above examples) can include one or more computing devices, such as FIG. 3 one or more systems or devices in the environment. The one or more processors of the computer system can be included in a single computing device or distributed among multiple computing devices. The memory of the computer system can include a respective memory of each of the multiple computing devices.

[0067] FIG. 4 is configured to perform the processes of FIGS. 1-2C A simplified functional block diagram of a computer 400 of a device that can be configured to perform the processes of the environment and / or methods according to example embodiments of the present disclosure. For example, the device 400 can include a central processing unit (CPU) 420. The CPU 420 can be any type of processor device, including, for example, any type of special-purpose or general-purpose microprocessor device. As will be appreciated by those skilled in the relevant art, the CPU 420 can also be a single processor in a multi-core / multi-processor system that runs individually, or in a cluster or server farm of computing devices that run in concert. The CPU 420 can be connected to a data communications infrastructure 410, such as a bus, message queue, network, or multi-core message passing scheme.

[0068] The device 400 can also include a main memory 440, such as a random access memory (RAM), and can also include a secondary memory 430. The secondary memory 430 (e.g., read only memory (ROM)) can be, for example, a hard disk drive or a removable storage drive. Such a removable storage drive can include, for example, a floppy disk drive, a magnetic tape drive, an optical disk drive, flash memory, etc. The removable storage drive in this example reads from and / or writes to a removable storage unit in a known manner. The removable storage unit can include a floppy disk, a magnetic tape, an optical disk, etc. which is read by and written to by the removable storage drive. As will be appreciated by persons skilled in the relevant art, such a removable storage unit typically includes a computer usable storage medium having stored therein computer software and / or data.

[0069] In alternative implementations, secondary memory 430 can include other similar devices for allowing computer programs or other instructions to be loaded into device 400. Examples of such devices can include program cartridge and cartridge interface (such as those found in video game devices), removable memory chips (e.g., EPROM or PROM) and associated socket, and other removable storage units and interfaces, which allow software and data to be transferred from the removable storage unit to device 400.

[0070] Device 400 can also include a communications interface ("COM") 460. Communications interface 460 allows software and data to be transferred between device 400 and external devices. Communications interface 460 can include a modem, a network interface (such as an Ethernet card), a communications port, a PCMCIA slot and card, or the like. Software and data transferred via communications interface 460 can be in the form of signals, which can be electronic, electromagnetic, optical or other signals capable of being received by communications interface 460. These signals can be provided to communications interface 460 via a communications path, which can carry signals and be implemented using, for example, wire or cable, fiber optics, a phone line, a cellular phone link, an RF link or other communications channels.

[0071] The hardware elements, operating systems and programming languages of such devices are conventional in nature, and it is presumed that those skilled in the art are adequately familiar therewith. Device 400 can also comprise input / output ports 450 to connect various input / output devices, such as keyboard, mouse, touch screen, monitor, displays, etc. Of course, various server functions can be implemented in a distributed fashion across a number of similar platforms, to distribute the processing load. Alternatively, a server can be implemented through a suitable programming of one computer hardware platform.

[0072] Program aspects of the technology can be thought of as “products” or “articles of manufacture” typically in the form of executable code and / or associated data that is carried or embodied on or in a type of machine readable medium. Storage-type media include any or all of the tangible memory of the computers, processors or the like, or of machines usually related to the computing device, such as various semiconductor memories, tape drives, disk drives and the like, which can provide non-transitory storage of software programming at a particular time. All or portions of the software can at times be communicated through the Internet or various other telecommunication networks. Such communications, for example, can enable loading of the software from one computer or processor into another, for example, from the management server or host computer of a mobile communication network into the computer platform of a server and / or from the server into the mobile device. Thus, another type of media that can carry software elements includes optical, electrical and electromagnetic waves, such as that used across physical interfaces, through wired and optical landlines, and over various air-links, as well as wires comprising a wireless link to carry the software elements. The physical elements that carry the waves, such as wired or wireless links, optical networks, etc., also can be considered a medium as long as those physical elements present at their place of use function to carry the software elements. Such a medium can take many forms, which will be apparent to those skilled in the art. As used herein, unless restricted to non-transitory, tangible “storage” media, terms such as computer or machine “readable medium” refer to any medium that participates in providing instructions to a processor for execution.

[0073] Mention of any particular activity in this disclosure is for convenience, and is not intended to limit the disclosure. One of ordinary skill in the art will recognize that the concepts underlying the disclosed devices and methods can be used in any suitable activity. The disclosure can be understood with reference to the description and drawings herein, in which like elements are referred to with like reference numerals.

[0074] It should be understood that throughout the foregoing description and claims of the present disclosure, various features are sometimes grouped together in a single embodiment, figure, or description of a related aspect in order to streamline the disclosure and help understand one or more of the various inventive aspects. However, this method of disclosure should not be construed to limit the application to less than is permitted by the claims attached hereto. Additionally, any individual element or combination of elements that modulates, responds to, or provides a desired property, function, or result of the present disclosure can be claimed.

[0075] Furthermore, while certain embodiments described herein include some features and do not include other features, combinations of the features of the different embodiments are intended to fall within the scope of the application, and form different embodiments, as can be apparent to those skilled in the art. For example, in the claims that follow, any of the claimed embodiments can be used in any combination.

[0076] Thus, although certain embodiments have been described, it is understood that the scope of coverage of this patent is not limited thereto, but is intended to extend to other embodiments which come within the spirit and scope of the present application. For example, functions described in the specification can be added or deleted, and operations between functions can be interchanged. Methods described within the scope of the present application can add or delete steps.

[0077] The above-disclosed subject matter is to be considered illustrative, and not restrictive, and the appended claims are intended to cover all such modifications, enhancements, and other embodiments, which fall within the true spirit and scope of the present disclosure. Thus, to the maximum extent allowed by law, the scope of the present disclosure is to be determined by the broadest permissible interpretation of the following claims and their equivalents, and shall not be restricted or limited to the abovementioned detailed description. While various embodiments of the present disclosure have been described above, it should be understood that they have been presented by way of example only, and not limitation. Numerous changes to the disclosed embodiments can be made in accordance with the disclosure herein without departing from the spirit or scope of the disclosure. Accordingly, the disclosure is not limited, except as by the appended claims and their equivalents.

Claims

1. A computer-implemented method for detecting one or more emotions of one or more pets, the method comprising: receiving, by one or more processors, image data from at least one user device, wherein the image data comprises one or more frames; detecting, by the one or more processors, at least one pet contour comprising at least one pet in the one or more frames; detecting, by the one or more processors, one or more emotions of the at least one pet based on the at least one pet contour; and displaying, by the one or more processors, the one or more emotions on at least one user interface of a user device.

2. The computer-implemented method of claim 1, the method further comprising: classifying, by the one or more processors, the at least one pet included in the at least one pet contour to correspond to at least one pet breed; and associating, by the one or more processors, the at least one pet breed with at least one breed cluster, wherein each of the at least one breed cluster comprises a plurality of emotion detectors.

3. The computer-implemented method of claim 2, the classifying further comprising: receiving, by a trained machine learning model, the at least one pet contour; analyzing, by the trained machine learning model, the at least one pet contour to determine at least one physical feature of the at least one pet; and determining, by the trained machine learning model, that the at least one pet corresponds to the at least one pet breed based on the at least one physical feature.

4. The computer-implemented method of claim 2, the method further comprising: analyzing, by the one or more processors, the at least one pet contour in each of the one or more frames; and determining, by the one or more processors, that the at least one pet breed appears most frequently in the one or more frames.

5. The computer-implemented method of claim 1, wherein, the detecting is performed by a convolutional neural network (CNN) model or a Transformer-based model.

6. The computer-implemented method of claim 1, wherein, the image data comprises video data.

7. The computer-implemented method of claim 1, the method further comprising: dilating, by the one or more processors, the at least one pet contour to maximize coverage of the at least one pet contour.

8. The computer-implemented method of claim 1, the method further comprising: identifying, by the one or more processors, at least one mask corresponding to a background surrounding the at least one pet contour of the one or more frames; updating, by the one or more processors, the one or more frames, the updating comprising utilizing the at least one mask to eliminate the background in the one or more frames; and creating, by the one or more processors, new image data based on the one or more updated frames.

9. The computer-implemented method of claim 1, the method further comprising: creating, by the one or more processors, a customized plan for the at least one pet based on the one or more emotions; and displaying, by the one or more processors, the customized schedule on at least one user interface of the user device.

10. The computer-implemented method of claim 1, further comprising: determining, by the one or more processors, at least one recommendation based on the one or more moods, wherein the at least one recommendation comprises at least one physical activity; and displaying, by the one or more processors, the at least one recommendation on at least one user interface of the user device.

11. The computer-implemented method of claim 1, further comprising: storing, by the one or more processors, the at least one pet and the one or more moods in one or more data stores.

12. The computer-implemented method of claim 1, further comprising: comparing, by the one or more processors, the one or more moods to at least one prior mood of the at least one pet; and displaying, by the one or more processors, information corresponding to the comparison on at least one user interface of the user device.

13. A computer system for detecting one or more moods of one or more pets, the computer system comprising: at least one memory storing instructions; and at least one processor configured to execute the instructions to perform operations comprising: receiving image data from at least one user device, wherein the image data comprises one or more frames; detecting, in the one or more frames, at least one pet outline comprising at least one pet; detecting, based on the at least one pet outline, one or more moods of the at least one pet; and displaying, on at least one user interface of a user device, the one or more moods.

14. The computer system of claim 13, the operations further comprising: classifying the at least one pet included in the at least one pet outline as corresponding to at least one breed of pet; and associating the at least one breed of pet with at least one breed cluster, wherein each breed cluster of the at least one breed cluster comprises a plurality of mood detectors.

15. The computer system of claim 14, the classifying further comprising: receiving, by a trained machine learning model, the at least one pet outline; analyzing, by the trained machine learning model, the at least one pet outline to determine at least one physical characteristic of the at least one pet; and determining, by the trained machine learning model, that the at least one pet corresponds to the at least one breed of pet based on the at least one physical characteristic.

16. The computer system of claim 13, the operations further comprising: analyzing the at least one pet outline in each frame of the one or more frames; and determining that the at least one breed of pet appears most frequently in the one or more frames.

17. The computer system of claim 13, the operations further comprising: dilating the at least one pet outline to maximize coverage of the at least one pet outline.

18. The computer system of claim 13, the operations further comprising: identifying at least one mask corresponding to a background surrounding at least one pet contour of the one or more frames; updating the one or more frames, the updating including eliminating the background in the one or more frames with the at least one mask; and creating new image data based on the one or more updated frames.

19. A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform operations for detecting one or more emotions of one or more pets, the operations comprising: receiving image data from at least one user device, wherein the image data includes one or more frames; detecting at least one pet contour including at least one pet in the one or more frames; detecting one or more emotions of the at least one pet based on the at least one pet contour; and displaying the one or more emotions on at least one user interface of the user device.

20. The non-transitory computer-readable medium of claim 19, the operations further comprising: comparing the one or more emotions to at least one prior emotion of the at least one pet; and displaying information corresponding to the comparison on at least one user interface of the user device.

Images