Animal emotion determination device and animal emotion determination system

Abstract

To provide an emotion determination device and an emotion determination system capable of determining the emotion of an animal from a still image of the animal.SOLUTION: The emotion of an animal is determined from a still image by using an emotion determination device 1 that includes: an image extraction unit 11 that extracts images of a mouth region and an eye region from an image of an animal to be determined; a mouth-region determination unit 12 that assigns an emotion score indicating a degree of emotion to the image of the mouth region by using a learned model constructed with mouth-region images of animals and emotions of the animals at the time of imaging as teacher data; an eye-region determination unit 13 that assigns an emotion score indicating a degree of emotion to the image of the eye region by using a learned model constructed with eye-region images of animals and emotions of the animals at the time of imaging as teacher data; and an emotion determination unit 14 that determines the emotion of the animal to be determined on the basis of the emotion score assigned by the mouth-region determination unit 12 and the emotion score assigned by the eye-region determination unit 13.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an animal emotion determination device and emotion determination system, and more particularly to a technique for determining the emotion of a pet animal from a facial image of the pet animal. [Background technology]

[0002] Pets such as dogs, cats, and rabbits have become irreplaceable members of the family. Each animal kept as a pet is an individual with its own personality and emotions. If we could understand the emotions of animals, communication between pets and their owners would become smoother and deeper, preventing misunderstandings and mismatches between pets and their owners, and ultimately reducing unfortunate cases such as abandonment.

[0003] Against this background, devices have been proposed that determine the emotion of an animal from an image of the animal (see, for example, Patent Documents 1 and 2). For example, the information processing device described in Patent Document 1 determines the emotion of an animal from the posture, facial expression, vocalizations, etc. of the animal in the image. In addition, the emotion determination system described in Patent Document 2 uses a trained model to extract information on specific behaviors, such as wagging a tail or scratching a body, from the image of the animal, and determines the emotion of the animal from the behavioral information. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2020-170916 [Patent Document 2] Japanese Patent Application Publication No. 2023-94426 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the techniques described in Patent Documents 1 and 2 above are both premised on making judgments using moving images, and have a problem in that a large amount of information is required to make emotion judgments.

[0006] Therefore, an object of the present invention is to provide an emotion determination device and an emotion determination system that can determine the emotion of an animal from a still image of the animal. [Means for solving the problem]

[0007] The animal emotion determination device of the present invention is a device that determines the emotion of an animal from an image of the animal, and has an image extraction unit that extracts images of the area around the mouth and the area around the eyes from the image, a mouth area determination unit that assigns an emotion score indicating the degree of emotion to the image of the mouth area using a trained model constructed using the image of the animal's area around the mouth and the emotion of the animal at the time of photographing as training data, an eye area determination unit that assigns an emotion score indicating the degree of emotion to the image of the eye area using a trained model constructed using the image of the animal's area around the eyes and the emotion of the animal at the time of photographing as training data, and an emotion determination unit that determines the emotion of the animal based on the emotion scores assigned by the mouth area determination unit and the eye area determination unit. The emotion determination device of the present invention can determine whether or not an animal is smiling, and in this case, the emotion score can be, for example, the probability that the animal is smiling. In the emotion determination device of the present invention, the image of the area around the eyes is, for example, an image including the medial levator canthus muscles, and the image of the area around the mouth is, for example, an image including the buccinator muscles and / or the orbicularis oris muscles. The emotion determination unit may perform determination by placing more importance on the emotion score assigned by the mouth-area determination unit than on the emotion score assigned by the eye-area determination unit.

[0008] The animal emotion determination system according to the present invention is an animal emotion determination system including the emotion determination device described above. The animal emotion determination system of the present invention may be configured to include a user terminal connectable to the emotion determination device via a network, wherein an image of the animal to be determined is transmitted from the user terminal to the emotion determination device, and the emotion determination result of the animal is transmitted from the emotion determination device to the user terminal. [Effects of the Invention]

[0009] According to the present invention, since the determination is made using an image of the area around the mouth and the area around the eyes of the animal, it is possible to accurately determine the emotion of the animal even from a still image. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a conceptual diagram showing an example of the configuration and operation of an emotion determination device according to a first embodiment of the present invention. [Figure 2] FIG. [Figure 3] FIG. [Figure 4] FIG. 10 is a conceptual diagram showing an example of the configuration of an emotion determination system according to a second embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0011] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings. However, the present invention is not limited to the following embodiments.

[0012] (First embodiment) First, an emotion determination device according to a first embodiment of the present invention will be described. Fig. 1 is a conceptual diagram showing an example of the configuration and operation of the emotion determination device of this embodiment. As shown in Fig. 1, the emotion determination device 1 of this embodiment determines the emotion of an animal to be determined from a still image of the animal provided by a user or the like, and includes at least an image extraction unit 11, a mouth area determination unit 12, an eye area determination unit 13, and an emotion determination unit 14.

[0013] [Animals to be judged] Examples of animals whose emotions can be determined by the emotion determination device 1 of this embodiment include pet animals such as dogs, cats, rabbits, hamsters, and ferrets; working animals such as sheep, goats, cows, and horses; and wild animals such as bears, raccoon dogs, and foxes. In particular, dogs, cats, rabbits, hamsters, ferrets, goats, and bears, whose body hair tends to prevent the relaxation of their facial muscles, are preferred, but the device is not limited to these and can be applied to various animals kept by humans.

[0014] [Emotion to be judged] Emotions that can be determined by the emotion determination device 1 of this embodiment include, for example, joy, relief, fear, anger, pain, suffering, sadness, sleepiness, surprise, and boredom.

[0015] [Image Extraction Section 11] The image extraction unit 11 extracts images of the area around the mouth and the area around the eyes from an image of the animal to be judged (hereinafter also referred to as a "judgement image"). FIG. 2 is a diagram showing the area around the mouth, with the image shown in the upper part of FIG. 2 being the judgment image and the image shown in the lower part of FIG. 2 being a diagram showing the area around the mouth. As shown in FIG. 2, the "area around the mouth" is the area including the mouth and its surroundings, such as the buccinator muscle, the orbicularis oris muscle, or both.

[0016] 3 is a diagram showing the area around the eyes, with the image shown in the upper part of FIG. 3 being a judgment image and the image shown in the lower part of FIG. 3 being a diagram showing the area around the eyes. As shown in FIG. 3, the "area around the eyes" refers to the area including the eyes and their surroundings, such as the area including the medial levator canthus muscles. By extracting images of these areas related to facial expressions and using them for judgment, emotions can be judged with high accuracy.

[0017] The method for extracting the images around the mouth and eyes from the judgment image is not particularly limited, but can be performed by using, for example, an algorithm based on a deep neural network, a convolutional neural network, etc. Examples of algorithms based on convolutional neural networks include YOLO, R-CNN, Fast R, and Faster R-CNN.

[0018] [Mouth area detection unit 12] The mouth area determination unit 12 assigns an emotion score indicating the degree of emotion to the extracted image of the mouth area using a trained model constructed using the image of the animal's mouth area and the emotion of the animal at the time the image was captured as training data. For example, when using the emotion determination device 1 of this embodiment to determine whether a pet animal is happy or not, the "emotion score" assigned by the mouth area determination unit 12 can be the "probability that the pet animal is smiling." When determining whether a pet animal is sleepy or not, the "emotion score" assigned by the mouth area determination unit 12 can be the "probability that the pet animal is sleeping."

[0019] The trained model used in the mouth area determination unit 12 is preferably artificial intelligence (AI). The AI ​​used here may be either a general-purpose or specialized type, and may be any of the well-known types such as logistic regression, decision tree, k-means algorithm, multilayer perceptron, recurrent neural network, deep neural network, and convolutional neural network, and publicly available software can be used.

[0020] The method of training artificial intelligence to generate a trained model may be either machine learning or deep learning, but deep learning is preferred. Deep learning is an advanced version of machine learning and is characterized by its ability to automatically find features. The training method for generating a trained model is not particularly limited; publicly available software or libraries can be used, or transfer learning can be employed. Specifically, DIGITS (the Deep Learning GPU Training System) published by NVIDIA can be used. A trained model can be generated by transfer learning using ResNet, MobileNet, or EfficientNet as the artificial intelligence (neural network) and Pytorch or the like as the machine learning (deep learning) library. Other known support vector machines, such as those published in "Introduction to Support Vector Machines" (Kyoritsu Publishing), may also be used.

[0021] [Eye area detection section 13] The eye area determination unit 13 assigns an emotion score indicating the degree of emotion to the extracted image of the eye area using a trained model constructed using the image of the animal's eye area and the emotion of the animal at the time the image was captured as training data. The trained model used in the eye area determination unit 13 is also preferably an artificial intelligence (AI). Note that the other configurations of the eye area determination unit 13 are the same as those of the mouth area determination unit 12 described above.

[0022] [Emotion judgment section 14] The emotion determination unit 14 determines the emotion of the target animal based on the emotion score assigned by the mouth-area determination unit 12 and the emotion score assigned by the eye-area determination unit 13. In this case, it is preferable to make the determination by placing more importance on the emotion score assigned by the mouth-area determination unit 12 than on the emotion score 13 assigned by the eye-area determination unit. In pet animals, the area around the mouth changes more significantly with emotion than the area around the eyes, so by placing more importance on the emotion score assigned by the mouth-area determination unit 12, the accuracy of the determination can be improved.

[0023] Specifically, to determine whether a pet is happy or not, a "probability of smiling" is assigned as an "emotion score," and if the probability of a smile around the mouth assigned by the mouth area judgment unit 12 is x% and the probability of a smile around the eyes assigned by the eye area judgment unit 13 is y%, the emotion judgment unit 14 will determine whether the pet is smiling (happy or not) based on, for example, whether x × (0.6 to 0.8) + y × (0.4 to 0.2) is equal to or greater than a reference value.

[0024] [program] The operations of the emotion determination device 1 of this embodiment can be implemented by an emotion determination program that causes a computer to execute a program that automatically determines the emotion of an animal from an image of the animal, and includes an image extraction function that extracts images of the area around the mouth and the area around the eyes from the determination image, a mouth area determination function that assigns an emotion score indicating the degree of emotion to the extracted image of the area around the mouth using a trained model constructed using the image of the area around the animal's mouth and the emotion of the animal at the time of photographing as training data, an eye area determination function that assigns an emotion score indicating the degree of emotion to the extracted image of the area around the eyes using a trained model constructed using the image of the area around the animal's eyes and the emotion of the object at the time of photographing as training data, and an emotion determination function that determines the emotion of the animal based on the emotion score assigned by the mouth area determination unit and the emotion score assigned by the eye area determination unit.

[0025] As described above in detail, the emotion determination device of this embodiment extracts images of the area around the mouth and the area around the eyes from images of the animal to be determined, and determines the emotion of the animal using these images with a trained model, so it can accurately determine the emotion of the animal with less information than determination using the movement of the entire body or video. The emotion determination device of this embodiment is particularly effective for determining "smiles."

[0026] (Second embodiment) Next, an emotion determination system according to a second embodiment of the present invention will be described. Fig. 4 is a diagram showing an example of the configuration of the emotion determination system of this embodiment. As shown in Fig. 4, the emotion determination system 10 of this embodiment includes the emotion determination device 1 of the first embodiment described above, and also includes user terminals 2a to 2d that can be connected to the emotion determination device 1 via the Internet 3 or the like.

[0027] [user] Users of the emotion determination system 10 of this embodiment are, for example, the owner of the animal to be determined, live animal sellers (breeders, pet shops, etc.) who sell the animal to be determined, and animal protection organizations that protect the animal to be determined.

[0028] [User terminals 2a to 2d] User terminals 2a to 2d are terminals owned by the owners of the animals to be judged. By connecting the emotion determination device 1 to user terminals 2a to 2d, users themselves can send or upload images of the animals to be judged to emotion determination device 1, and can also receive emotion determination results of the animals to be judged from emotion determination device 1.

[0029] As described above in detail, the emotion determination system of this embodiment can accurately determine the emotion of an animal from an image taken by a user and provide the result to the user, thereby contributing to improving communication between pet owners. Furthermore, the emotion determination system of this embodiment can accurately determine the emotion of an animal with a smaller amount of information than conventional devices that use video, and is particularly effective in determining a "smile." [Example]

[0030] The effects of the present invention will be specifically described below with reference to examples and comparative examples. In these examples, whether or not a dog is smiling was determined for each breed using the emotion determination device shown in Fig. 1. In this case, the trained models for the mouth area determination unit and eye area determination unit were constructed using 3,264 facial images (672 smiling images and 2,592 non-smiling images) of dogs (toy poodles, chihuahuas, dachshunds, and Shiba Inu) as training data.

[0031] In addition, in the judgment by the emotion judgment unit, the probability that the area around the mouth assigned by the mouth area judgment unit 12 is a smile is x%, the probability that the area around the eyes assigned by the eye area judgment unit 13 is a smile is y%, and x × 0.7 + y × 0.3 ≥ 30% is considered to be a "smile". The results are shown in Tables 1 to 6 below. Table 1 shows the judgment results for toy poodles, Table 2 shows the judgment results for chihuahuas, Table 3 shows the judgment results for dachshunds, Table 4 shows the judgment results for Shiba Inu dogs, Table 5 shows the judgment results for French bulldogs, and Table 6 shows the judgment results for Shih Tzu dogs.

[0032] [Table 1]

[0033] [Table 2]

[0034] [Table 3]

[0035] [Table 4]

[0036] [Table 5]

[0037] [Table 6]

[0038] As shown in Tables 1 to 4 above, the accuracy rate for smiling faces was 96% (70 / 73) for Toy Poodles, 97% (76 / 78) for Chihuahuas, 95% (93 / 98) for Dachshunds, and 98% (89 / 90) for Shiba Inu, which were dog breeds used in the training data when building the trained model, confirming that smiling faces can be detected with high accuracy. Furthermore, the accuracy rate for smiling faces was 100% for French Bulldogs (shown in Table 5 above) and Shih Tzus (shown in Table 6 above), confirming that smiling faces can be detected with high accuracy even for dog breeds not included in the training data.

[0039] Furthermore, when the criteria for determining a "smile" was set to x × 0.7 + y × 0.3 ≧ 40%, the accuracy rate for smiles was 95% (457 / 481) for toy poodles, 93% (142 / 152) for chihuahuas, 96% (196 / 204) for dachshunds, and 97% (189 / 195) for shiba inu. Smiles could be detected with high accuracy in all dogs, but the accuracy rate for smiles was generally higher when the criteria for determining a "smile" was set to x × 0.7 + y × 0.3 ≧ 30%.

[0040] Furthermore, when the criteria for determining a "smile" was set to x × 0.8 + y × 0.2 ≧ 30%, the accuracy rate for smiles was 95% (459 / 481) for toy poodles, 93% (142 / 152) for chihuahuas, 96% (197 / 204) for dachshunds, and 97% (189 / 195) for shiba inu. Smiles could be detected with high accuracy in all dogs, but the accuracy rate for smiles was generally higher when the criteria for determining a "smile" was set to x × 0.7 + y × 0.3 ≧ 30%.

[0041] From the above results, it was confirmed that the emotion determination device of the present invention can be used to determine the emotion of an animal from an image of the animal with high accuracy. [Explanation of symbols]

[0042] 1 Emotion determination device 2a~2d User terminal 3. Internet 10 Emotion detection system 11 Image extraction section 12 Mouth area detection section 13 Eye area detection section 14 Emotion Judgment Department

Claims

1. A device for determining an emotion of an animal from an image of the animal, an image extraction unit that extracts images of a mouth area and an eye area from the image; a mouth area determination unit that assigns an emotion score indicating the degree of emotion to the image of the mouth area using a trained model constructed using an image of the mouth area of ​​an animal and the emotion of the animal at the time of photographing as training data; an eye area determination unit that assigns an emotion score indicating the degree of emotion to the image of the eye area using a trained model constructed using images of the area around the animal's eyes and the emotion of the animal at the time of shooting as training data; an emotion determination unit that determines the emotion of the animal based on the emotion score assigned by the mouth-area determination unit and the emotion score assigned by the eye-area determination unit; An animal emotion determination device having the above.

2. The animal emotion determination device according to claim 1, wherein the device determines whether the animal is smiling.

3. The animal emotion determination device according to claim 2 , wherein the emotion score is a probability of a smile.

4. The animal emotion determination device according to claim 1 , wherein the image of the area around the eyes is an image including the medial levator canthus oculi muscle.

5. The animal emotion determination device according to claim 1 , wherein the image of the area around the mouth includes buccinator muscles and / or orbicularis oris muscles.

6. 2. The animal emotion determination device according to claim 1, wherein the emotion determination unit makes a determination by placing more importance on the emotion score assigned by the mouth-area determination unit than on the emotion score assigned by the eye-area determination unit.

7. An animal emotion determination system comprising the emotion determination device according to any one of claims 1 to 6.

8. a user terminal connectable to the emotion determination device via a network; an image of an animal to be determined is transmitted from the user terminal to the emotion determination device; The system for determining an emotion of an animal according to claim 7 , wherein the emotion determination device transmits a result of the emotion determination of the animal to the user terminal.

Images