Posture detection system

Abstract

The objective is to provide an attitude detection system with a simple equipment configuration. [Solution] The posture detection system is a posture detection system for detecting the lying-down position of a cow, and comprises an imaging device for acquiring images and a detection device for detecting the lying-down position of the target cow captured in the images. The detection device includes a skeletal estimation unit that generates a skeletal model that estimates the skeleton of the target cow, and performs posture determination of the target cow based on the skeletal model.

Description

Technical Field

[0001] The present disclosure relates to a cow posture detection system, and particularly relates to estimating a cow skeleton model based on an image acquired by an imaging device and detecting the posture of a cow based on the estimated skeleton model.

Background Art

[0002] Cows may die due to asphyxiation or the like due to abnormalities such as difficulty in standing up. Therefore, livestock farmers who raise cows need to patrol the cowshed to monitor whether the cows they are raising are experiencing difficulty in standing up.

[0003] Conventionally, an abnormality detection device that detects the difficulty of a cow in standing up by attaching a device equipped with an acceleration sensor and an air pressure sensor to the cows being raised is known. The abnormality detection device estimates whether a cow is standing or in a lying posture based on air pressure data obtained from the air pressure sensor, and further scores the degree of restlessness of the cow based on acceleration data obtained from the acceleration sensor, and detects that an abnormality has occurred in the cow based on the score (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, the abnormality detection device disclosed in Patent Document 1 has a problem that it is necessary to attach a device (such as a tag) to each individual cow, and when raising a large number of cows, the management of the device is complicated.

[0006] The present disclosure solves the above problems and provides a posture detection system with a simple device configuration. [Means for solving the problem]

[0007] The posture detection system according to this disclosure is a posture detection system for detecting the lying-down state of a cow, comprising: an imaging device for acquiring an image; and a detection device for detecting the lying-down state of a target cow captured in the image, wherein the detection device includes a skeletal estimation unit for generating a skeletal model that estimates the skeleton of the target cow, and performs posture determination of the target cow based on the skeletal model. [Effects of the Invention]

[0008] According to the above method, the skeletal structure of the cow captured in the image obtained by the imaging device is estimated to detect the recumbent position. Therefore, it is not necessary to attach special equipment to each cow, and the posture of the cow can be determined with a simple equipment configuration. This makes it possible to detect when a recumbent position, which is particularly harmful to livestock, is occurring. [Brief explanation of the drawing]

[0009] [Figure 1] This is a schematic diagram showing the posture detection system 100 according to Embodiment 1. [Figure 2] This is an example of the hardware configuration of the detection device 10, imaging device 20, audio playback device 30, and external terminal 40 according to Embodiment 1. [Figure 3] This is an example of a functional block of the posture detection system 100 according to Embodiment 1. [Figure 4] This is an example of data processing for each device and functional block of the posture detection system 100 according to Embodiment 1. [Figure 5] This is an example of a bovine skeletal model M estimated in the posture detection system 100 according to Embodiment 1. [Figure 6] This is an example of a bovine skeletal model M estimated in the posture detection system 100 according to Embodiment 1. [Figure 7] This flowchart shows an example of the overall configuration of the process of the skeletal estimation unit 12a of the posture detection system 100 according to Embodiment 1. [Figure 8] This shows an example of an image obtained by the imaging device 20 of the posture detection system 100 according to Embodiment 1. [Figure 9] This shows an example of estimating the skeletal model M in the posture detection system 100 according to Embodiment 1. [Figure 10] This is an example of an image of a target cow used to detect a lying-down position in the posture detection system 100 according to Embodiment 1. [Figure 11] This flowchart shows an example of the configuration of a posture determination model 72 using a skeletal model M of the posture detection system 100 according to Embodiment 1. [Figure 12] This is an example of the information processing flow performed in the posture detection system 100 according to Embodiment 1. [Figure 13] Figure 12 shows an example of the information processing flow for step S5. [Modes for carrying out the invention]

[0010] Preferred embodiments of the attitude detection system of the present disclosure are described in detail below with reference to the drawings. The embodiments described below are preferred examples and are subject to various technically preferred limitations; however, the scope of the present disclosure is not limited to these embodiments unless otherwise stated in the following description.

[0011] Embodiment 1. 1. Configuration of the posture detection system 100 FIG. 1 is a schematic configuration diagram showing a posture detection system 100 according to Embodiment 1. The posture detection system 100 shown in FIG. 1 is a system that detects a predetermined posture of a cow being raised. The posture detection system 100 includes an imaging device 20, an audio playback device 30, and a detection device 10 that processes data from these devices and detects a predetermined posture of the cow. The posture detection system 100 estimates a skeletal model M of a cow (A1, A2, A3 in FIG. 1) based on an image from the imaging device 20 installed in a breeding environment 80 where cows are raised, such as a cowshed, and detects a predetermined posture of the cow based on the state of the skeletal model M.

[0012] The cows being raised may have difficulty standing up. This symptom can be caused by various factors. When a cow has difficulty standing up, it cannot secure its airway, and the accumulated gas compresses the gastrointestinal tract. If this state persists for a long time, it may lead to death. Especially if a cow lies down with its legs thrown out for a long time, it may lead to death. The posture detection system 100 can detect the posture of a cow, including its lying state. In particular, the posture detection system 100 is used to detect a lying state that is dangerous for the cows being raised.

[0013] In addition, when the posture detection system 100 detects the lying state of a cow, it can play a predetermined sound from the audio playback device 30 installed in the breeding environment of the cow to prompt the cow to stand up. Also, when the posture detection system 100 detects the lying state of a cow, it can notify an external terminal 40 that the detection has been made.

[0014] The network 90 has the role of connecting the detection device 10, the imaging device 20, the audio playback device 30, and the external terminal 40. The network 90 is a communication network for establishing a connection path to connect the detection device 10, the imaging device 20, the audio playback device 30, and the external terminal 40 so that data can be transmitted and received. However, at least a part of the detection device 10, the imaging device 20, the audio playback device 30, and the external terminal 40 may be an integrated device or a separate device. Also, at least a part of the detection device 10, the imaging device 20, the audio playback device 30, and the external terminal 40 may be directly connected to each other by wire or wirelessly.

[0015] At least a part of the network 90 may be a wired network or a wireless network. Also, the network 90 may be an IP (Internet Protocol) network. The network 90 may be, for example, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), an ad hoc network, a metropolitan area network (MAN), a part of the Internet, a part of the Public Switched Telephone Network (PSTN), a mobile phone network, ISDN (integrated service digital networks), a wireless LAN, LTE (long term evolution), CDMA (code division multiple access), Bluetooth (registered trademark), satellite communication, etc., or a combination of two or more of these, enabling communication between each terminal.

[0016] 2. Hardware Configuration of the Posture Detection System 100 Figure 2 shows an example of the hardware configuration of the detection device 10, imaging device 20, audio playback device 30, and external terminal 40 according to Embodiment 1. The detection device 10, imaging device 20, audio playback device 30, and external terminal 40 are information processing devices having the hardware configuration shown in Figure 2(a) or Figure 2(b), for example. However, the imaging device 20, audio playback device 30, and external terminal 40 may be dedicated devices that realize only specific functions and are capable of sending and receiving necessary data to and from the detection device 10. In other words, the imaging device 20, audio playback device 30, and external terminal 40 may not have all of the hardware configurations shown in Figure 2(a) or Figure 2(b).

[0017] The information processing device, comprising a detection device 10, an imaging device 20, an audio playback device 30, and an external terminal 40, includes at least a control unit 51, a storage unit 52, and a communication unit 53. The control unit 51 is, for example, a CPU (Central Processing Unit). The control unit 51 may also include ROM (Read Only Memory) and RAM (Random Access Memory). The CPU is also called a central processing unit, central computing unit, processor, microprocessor, microcomputer, or DSP (Digital Signal Processor). In the information processing device, the CPU reads programs and data stored in ROM and uses RAM as a work area to comprehensively control each information processing device. Based on the information stored in the storage unit 52, such as ROM and RAM, the CPU controls communication between information processing devices via the communication unit 53, and the content displayed in the output unit 55, such as a display device and a printing device. Figure 2(b) shows the hardware configuration when the information processing device has an input unit 54 and an output unit 55 in addition to the control unit 51, storage unit 52, and communication unit 53. If the information processing device has the hardware configuration shown in Figure 2(b), the control unit 51 controls the content displayed on the output unit 55, such as a display device and a printing device, based on the content input by the user via the input unit 54, and also controls various functions such as recognizing the content input by the user via the input unit 54 and storing the information in the storage unit 52.

[0018] Furthermore, the control unit 51 may implement each process not only using a CPU with a control circuit, but also using logic circuits (hardware) formed on an integrated circuit (IC (Integrated Circuit) chip, LSI (Large Scale Integration)), etc., or dedicated circuits.

[0019] The storage unit 52 is a non-volatile semiconductor memory such as flash memory, EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), or HDD (Hard disk drive), and plays the role of a so-called secondary storage device. In Embodiment 1, the storage unit 52 may be installed in the terminal or it may be a server connected via the network 90. ​​Note that the storage unit 52 is not necessarily installed in the terminal, but also includes those installed on the network 90 in a communicable manner, and the network 90 may also be referred to as the storage unit 52. In other words, the control unit 51 can retrieve information from a communicable storage device, including the storage unit 52 installed in the terminal or the storage unit 52 connected to the network 90. ​​The control unit 51 also retrieves information necessary to display on the display device from the storage unit 52 as appropriate. The storage unit 52 can also store user response information via the input unit 54.

[0020] The communication unit 53 is the part that transmits and receives various data via the network 90, and any communication protocol can be used as long as communication between each information processing device can be performed, whether wired or wireless. For example, wired communication methods such as USB, IEEE1394, Thunderbolt®, and wired LAN network communication can be used. The communication unit 53 may also be configured to be connected to a communication network via wireless communication methods such as wireless LAN network communication, 3G / LTE / 5G mobile communication, or Bluetooth® communication. Furthermore, the communication unit 53 may be configured to use both the aforementioned wired and wireless communication methods in combination. The communication unit 53 transmits various data to other information processing devices based on instructions from the control unit 51. The communication unit 53 also receives various data from other information processing devices and sends it to the control unit 51.

[0021] The input unit 54 is configured to receive operation inputs made by a user of the information processing device, for example. The input unit 54 may be included in the housing of the information processing device or it may be external. The input unit 54 can be, for example, a touch panel, switch buttons, a mouse, a keyboard, etc. Whether or not the information processing device includes an input unit 54 is optional. For example, as a modification, operation inputs to the information processing device may be received by the information processing device 1 via an information processing terminal located in a separate location independent of where the information processing device is installed.

[0022] The output unit 55 is, for example, a display unit of the information processing device. The output unit 55 may be included in the housing of the information processing device, for example, or it may be externally mounted. The output unit 55 displays a graphical user interface (GUI) screen that can be operated by the user. The output unit 55 may employ display devices such as a CRT display, liquid crystal display, organic EL display, plasma display, electronic paper display, head-mounted display, or smart glasses, as well as a display device that can be turned on, such as a light or projector. It is optional whether or not the information processing device includes an output unit 55. For example, as a modification, the output of the information processing device may be displayed on a display unit located in a separate location independent of where the information processing device 1 is installed. The output unit 55 may also have a device that outputs sound.

[0023] 2.1. Example of hardware configuration of detection device 10 The control unit 51, storage unit 52, and communication unit 53 of the detection device 10 in the posture detection system 100 may be configured in any way. For example, the detection device 10 may be a single information processing terminal. Alternatively, if the detection device 10 is formed as a network computer system consisting of multiple server computers, each function may be distributed and installed across multiple server computers. Furthermore, these server computers may be installed across computer systems owned by multiple vendors or server administrators (for example, computer systems owned or managed by other information providers, ISPs, hosting system providers, etc.). The server computers may be formed as a so-called cloud computer system. Moreover, at least some of the components of the control unit 51, storage unit 52, and communication unit 53 may be provided in the client terminals that constitute the posture detection system 100. Note that the detection device 10 may also include an input unit 54, an output unit 55, and an imaging unit 56, as shown in Figure 2(b), and each function may be distributed and installed across multiple server computers.

[0024] 2.2. Example of Hardware Configuration of Imaging Device 20 The imaging device 20 in the posture detection system 100 is a device that visually captures objects and scenes and records and outputs them as images. The images recorded by the imaging device 20 may be either moving images or still images. The imaging device 20 includes an optical system and an image sensor as an input unit 54 to acquire images. The obtained images are converted into image data by an image processing unit. The functions of the image processing unit are realized, for example, in the control unit 51 shown in Figure 2. The image data is transmitted from the communication unit 53 to the detection device 10. The image data may also be configured to be stored in a storage unit 52 of the imaging device 20. The image data may also be configured to be viewable on an output unit 55 such as a display device of the imaging device 20. The imaging device 20 may be part of the detection device 10. The imaging device 20 may also be an integrated device with the audio playback device 30, which will be described later.

[0025] 2.3. Hardware configuration of the audio playback device 30 The voice playback device 30 in the posture detection system 100 is a device for playing back audio data and producing a predetermined sound. The purpose of the voice playback device 30 is to encourage the cows being raised to stand up by playing back the sound. The voice playback device 30 is configured to play back a predetermined sound when the detection device 10 detects that the cow is lying down. The voice playback device 30 is equipped with at least a speaker as an output unit 55, which converts the audio signal into vibrations and vibrates the air to produce sound. When outputting sound from the speaker, it may be equipped with an amplifier or the like to amplify the audio signal and increase the volume. The voice playback device 30 may be configured to receive and play back audio stored in the detection device 10 using the communication unit 53, or it may play back audio data stored in its own storage unit 52. The voice playback device 30 may also be equipped with an audio processing unit that converts audio data stored as digital data into an analog signal. The function of the audio processing unit is performed by the control unit 51 of the voice playback device 30. Furthermore, the audio playback device 30 may be equipped with an input unit 54 and configured to receive predetermined audio data and store it in the storage unit 52.

[0026] 2.4. Example of hardware configuration of external terminal 40 In the posture detection system 100, the external terminal 40 is a device for notifying the user of the information detected by the detection device 10. The external terminal 40 can include any device that can notify the user, such as a personal information terminal, personal computer, telephone, display, speaker, buzzer, lamp, etc. For example, if the external terminal 40 is a personal information terminal, it can notify the user that a predetermined posture has been detected by means of text notification such as email or messenger, voice notification, visual notification such as the lighting or flashing of an LED, or physical notification such as vibration. Alternatively, these notification means may be combined to notify the user. The external terminal 40 receives signals from the detection device 10 with its communication unit 53 and notifies the user with its output unit 55. Control associated with these notifications is performed by the control unit 51 of the external terminal 40.

[0027] Furthermore, the external terminal 40 may communicate with the detection device 10 via the communication unit 53 and be configured to, for example, change settings related to the detection device 10. For example, the external terminal 40 may have application software installed that allows the user to change the settings of the posture detection system 100. The external terminal 40 may be configured to display the contents shown on the output unit 55, operate via the input unit 54, and transmit the settings to the detection device 10 from the communication unit 53.

[0028] 2.5. Others Furthermore, the information processing devices, namely the detection device 10, imaging device 20, audio playback device 30, and external terminal 40, can realize the functions of each functional unit shown in the embodiment by reading a program stored in a storage medium and executing the read program.

[0029] Furthermore, the program may be provided to the detection device 10, imaging device 20, audio playback device 30, and external terminal 40 via any transmission medium capable of transmitting the program (such as a communication network or broadcast waves). The detection device 10, imaging device 20, audio playback device 30, and external terminal 40 may realize the functions of each functional unit shown in each embodiment by executing a program downloaded, for example, via the Internet.

[0030] Furthermore, embodiments of this disclosure may also be realized in the form of data signals embedded in a carrier wave, where the program is embodied by electronic transmission. At least a portion of the processing in the detection device 10, imaging device 20, audio playback device 30, and external terminal 40 may or may not be realized by cloud computing consisting of one or more computers.

[0031] Unless otherwise explicitly stated, the configuration of the determination in the embodiments of this disclosure is not essential, and a predetermined process may be performed if the determination conditions are met, or if the determination conditions are not met.

[0032] The programs described in this disclosure are implemented using, for example, scripting languages ​​such as JavaScript® and Python®, object-oriented programming languages ​​such as Java®, markup languages ​​such as HTML5, and functional programming languages ​​such as Elixir.

[0033] An example of information processing according to Embodiment 1 will be described below. In the following description, Figure 1 will be referenced as appropriate for each component of the posture detection system 100.

[0034] 3. Overview of information processing in the posture detection system 100 Figure 3 is an example of a functional block of the posture detection system 100 according to Embodiment 1. Figure 4 is an example of data processing for each device and functional block of the posture detection system 100 according to Embodiment 1. The detection device 10 acquires images from the imaging device 20 installed in the cattle rearing environment 80 at the image receiving unit 11. The imaging device 20 is arranged, for example, to be able to image the entire area within a predetermined section of a cattle barn, and to be able to identify cattle A1, A2, and A3 present in that section within the image. The acquired images are processed by the skeleton estimation unit 12a to estimate the skeleton model M of the target cattle A1, A2, and A3 that are captured in the image. The skeleton model M will be explained in Chapter 3.1.1 using Figures 6 and 7. The estimated skeleton model M is processed by the recumbent detection unit 13. The recumbent detection unit 13 detects that the target cattle A1, A2, and A3 are in a recumbent state if their skeletons are in a predetermined state. The skeletal estimation unit 12a may provide the skeletal model M to the lateral position detection unit 13 as the coordinates of each joint.

[0035] In addition to estimating the skeletal model M described above, the posture detection system 100 also determines the recumbent state using image recognition. The determination of the posture of target cows A1, A2, and A3 using the skeletal model M determines whether or not they are in a recumbent state based on the state of the cow's forelimbs and hindlimbs in the skeletal model M. However, the skeletal model M alone may not be able to accurately determine the recumbent state. For example, when a target cow is standing, its forelimbs and hindlimbs are extended, which may be misdetected as a recumbent state. This is because the skeletal model M is an estimate of the target cow's skeleton and does not have information on the positional relationship between the ground and the skeletal model M. The image recognition unit 12b determines the posture of the target cows included in the image from the imaging device 20 using image recognition technology. This image recognition will be explained in Chapter 3.2. The image recognition unit 12b provides the posture determination result of the target cows using image recognition to the recumbent state detection unit 13.

[0036] The recumbent position detection unit 13 acquires the posture determination result of the target cow by the skeletal model M and the posture determination result of the target cow by image recognition, and detects the recumbent state by combining these determination results.

[0037] If the reclining detection unit 13 detects that any of the imaged target cows A1, A2, or A3 is in a reclining state, the reclining detection unit 13 sends the reclining determination result to the audio playback unit 14 and the notification unit 18. The audio playback unit 14 transmits a signal to the audio playback device 30, causing the audio playback device 30 to play a predetermined sound. The notification unit 18 also notifies the external terminal 40 that the target cow is in a reclining state.

[0038] The detection device 10 estimates a skeletal model M based on images transmitted from the imaging device 20 at predetermined intervals. Therefore, a skeletal model M is estimated for each image, but depending on the posture and position of the target cows A1, A2, and A3 captured in the image, there may be variations in the estimated skeletal model M. The error standardization module 62 is designed to suppress false detections caused by variations in multiple skeletal models M estimated from multiple images obtained over time. In addition, the detection device 10 calculates the distance between each joint from the skeletal model M using the distance calculation module 61. The error standardization module 62 and the distance calculation module 61 are described in Chapters 3.1.2 and 3.1.3.

[0039] Furthermore, the images acquired by the imaging device 20 may be configured to be stored in the image storage unit 16 of the database unit 15. The image storage unit 16 may be configured to erase the captured images after a predetermined period of time has elapsed.

[0040] Furthermore, the audio data to be played back by the audio playback device 30 may be stored in the audio storage unit 17. The audio playback unit 14 can transmit audio data to the audio playback device 30 when a lying-down position is detected. Alternatively, the audio playback device 30 may be configured to store the audio data to be played back and to play it back in response to an audio playback instruction signal transmitted from the audio playback unit 14.

[0041] The skeletal model M estimated by the skeletal estimation unit 12a of the detection device 10 can also be viewed by the user as image data. The display unit 19 receives the image data and the skeletal model M estimated based on the image data from the skeletal estimation unit 12a, processes it by overlaying the skeletal model M onto the image data acquired by the imaging device 20 (for example), and displays it on a display device such as a display. The display device may be a display of the detection device 10, or it may be a screen of an external terminal 40.

[0042] 3.1. About Skeletal Model M 3.1.1. Overview of Skeletal Model M Figures 5 and 6 show examples of a bovine skeletal model M estimated by the posture detection system 100 according to Embodiment 1. Figure 5 shows the skeletal model M of a bovine in an upright position, and Figure 6 shows the skeletal model M of a bovine in a prone position. Note that the prone position, unlike the lateral position, is a normal state in which the bovine can transition to an upright position on its own. The posture detection system 100 estimates the bovine skeletal model M shown in Figures 5 and 6 from images acquired by the imaging device 20 using the skeletal estimation unit 12a. The skeletal estimation unit 12a estimates the positions of each joint, indicated by nodes H in the bovine's head, nodes B in the torso (node ​​B1 at the base of the neck and node B2 in the lumbar region), nodes RH and LH in the forelimbs, and nodes RL and LL in the hindlimbs, from the state of the target bovine captured in the image.

[0043] In Figures 5 and 6, node H1 represents the tip of the head, and nodes H2 and H3 represent the temples on both sides. Node B1 represents the base of the neck, and node B2 represents the lumbar region of the torso. Nodes RH1 and LH1 represent the shoulder joints, nodes RH2 and LH2 represent the elbow joints, and nodes RH3 and LH3 represent the carpal joints. Nodes RL1 and LL1 represent the hip joints, nodes RL2 and LL2 represent the knee joints, and nodes RL3 and LL3 represent the hocks. Skeletal model M is formed by connecting the nodes obtained by estimating each of these joints along the body of the cow. Note that skeletal model M is not limited to being composed only of the nodes shown in Figures 5 and 6, but may also include nodes corresponding to other joints or parts of the body.

[0044] The estimation of the skeletal model M is performed by collecting images of multiple cows, creating data for the skeletal model M corresponding to those images, and using a skeletal estimation model 70 trained with machine learning. The skeletal estimation model 70 is constructed, for example, using a convolutional neural network (CNN), which extracts features from image data through multiple layers and ultimately estimates the position of each joint. This skeletal estimation model 70 automatically detects the position of each joint captured in the images of the target cows and generates the skeletal model M.

[0045] The posture detection system 100 has a mechanism that updates the skeletal model M in real time based on image data acquired by the imaging device 20. This allows the system to accurately reflect changes in the cow's posture, corresponding to the changes in the skeletal model M. Furthermore, learning using multiple image data enables highly accurate estimation that takes into account individual differences in cows and differences in the environment.

[0046] Figure 7 is a flowchart showing an example of the overall configuration of the process of the skeletal estimation unit 12a of the posture detection system 100 according to Embodiment 1. The posture detection system 100 takes an image of a cow as input and estimates the cow's skeletal model M using the skeletal estimation model 70. The skeletal estimation model 70 is trained using multiple images of cows collected in advance and data of the corresponding skeletal models (training data 71), thereby enabling accurate estimation of the skeletal model.

[0047] The skeletal model M generated by the skeletal estimation model 70 is sent to the evaluation and comparison unit 74, where user evaluations and feedback may be incorporated. The evaluation and comparison unit 74 determines whether the estimated skeletal model M is accurate, and the result is fed back to the skeletal estimation model 70, thereby continuously improving the accuracy of the skeletal model M. In this way, the skeletal estimation model 70 can learn based on data and evaluations obtained during operation. As shown in Figure 7, the user's evaluation results are reflected in the model through the evaluation and comparison unit 74 and used to improve accuracy.

[0048] Furthermore, the training data 71 is a dataset used during the initial training of the system and includes combinations of images of cows and their corresponding skeletal models M (or node data corresponding to joints). Based on this dataset, a machine learning algorithm trains the skeletal estimation model 70. Through this training, the skeletal estimation model 70 acquires the ability to accurately estimate the posture and joint positions of cows.

[0049] Furthermore, the evaluation and comparison unit 74 may compare not only the results of manual evaluation by the user, but also the estimated results in actual operation with known ground truth data. For example, when a cow assumes a specific posture, the accuracy of the joint position estimation results based on that posture can be automatically evaluated. In this way, a feedback loop for improving the accuracy of the skeletal estimation model 70 may be formed.

[0050] The skeletal model M is created in the skeletal estimation unit 12a based on image data of a cow. The data of the skeletal model M is then provided to the recumbent position detection unit 13, the distance calculation module 61, and the error standardization module 62 in an appropriate format. The skeletal model M consists, for example, of positional information and corresponding coordinate data for each joint. This data is first passed to the distance calculation module 61 in order to calculate the distance between joints. The distance calculation module 61 calculates the relative distance between each joint and sends the result to the error standardization module 62. The error standardization module 62 performs error correction through comparison with a reference model and sends the final data of the skeletal model M to the recumbent position detection unit 13. Specifically, this data may include numerical formats such as coordinate data (values ​​of the x, y, and z axes) for each joint point, the distance between joints, and angle information. Various data formats can be used as examples, such as JSON, XML, CSV, and Prtobf.

[0051] Furthermore, the skeletal model M may be provided as image data to the lying-down detection unit 13, the distance calculation module 61, and the error standardization module 62. The lying-down detection unit 13, the distance calculation module 61, and the error standardization module 62 may process the data of the skeletal model M by image recognition.

[0052] 3.1.2. Calculation of Inter-Joint Distances Figure 8 shows an example of an image obtained by the imaging device 20 of the posture detection system 100 according to Embodiment 1. The target cow and skeletal model M shown in Figures 5 and 6 were obtained from images of cows captured in images of the cow rearing environment 80 obtained by the imaging device 20. As shown in Figure 8, the posture detection system 100 can estimate the skeletal model M shown in Figures 5 and 6 from images of an environment where many cows are being raised.

[0053] The image shown in Figure 8 is a photograph of a rearing environment 80 with depth, where the cows in the foreground appear larger and those in the background appear smaller. Therefore, even if the cows are actually of similar size, the cows in the foreground appear larger and those in the background appear smaller in the image. Consequently, the number of pixels between joints (number of pixels between nodes) in the estimated skeletal model M is smaller for the cows in the background and larger for the cows in the foreground. The posture detection system 100 performs a process to generalize the difference in the number of pixels between joints depending on the position of the cows in this image. This process is performed in the distance calculation module 61 (see Figures 3 and 4).

[0054] As an example, if RHx is the number of pixels from node RH1 to RH3 on the right forelimb of the subject cow, and Bx is the number of pixels from node B1 at the base of the neck to node B2 in the lumbar region of the subject cow, then the distance L(RH) from node RH1 to RH3 on the right forelimb of the subject cow can be expressed as follows. L(RH)=RHx / Bx (Formula 1)

[0055] By calculating the distance L between joints using (Equation 1) above, the distance L between joints of the target cow in the background and the target cow in the foreground of the image can be compared. In other words, although the target cow in the background appears with fewer pixels in the image, in the example above, the distance L(RH) is calculated as the ratio of the number of pixels RHx from the nodes RH1 to RH3 of the right forelimb to the number of pixels Bx from node B1 at the base of the neck to node B2 in the lumbar region, so it becomes a value that can be compared with the distance L(RH) of the target cow in the foreground. This distance L(RH) is useful for determining the distance L between joints of the target cow using a certain threshold L1 when determining the recumbent state, as described later.

[0056] In Equation 1 above, the distance L(RH) between nodes RH1 and RH3 on the right forelimb was calculated using, as an example, the number of pixels Bx from node B1 at the base of the neck to node B2 in the lumbar region of the subject cow. However, the calculation may be performed using the number of pixels of other parts instead of Bx. For example, the distance L may be calculated using the number of pixels from nodes H1 to H2 or H3 on the head, or the distance from node H1 on the head to node B1 at the base of the neck. Furthermore, although Equation 1 shows the distance L(RH) from nodes RH1 to RH3 on the right forelimb, it can be calculated similarly for other parts of the subject cow.

[0057] 3.1.3. Correction of false positives (error standardization process) Figure 9 shows an example of skeletal model M estimation in the posture detection system 100 according to Embodiment 1. In Figure 9, the estimation results of multiple skeletal models M over time are displayed. In this example, the state in which the cow is correctly facing right should be continuously estimated, but the processing when the third skeletal model M3 incorrectly detects that the cow is facing left is shown.

[0058] In the posture detection system 100, the error standardization module 62 analyzes the positional relationships of multiple skeletal models (such as the second skeletal model M2 in the figure) obtained over the first period Δt1 in a time series. The second skeletal model M2 estimated in the second period Δt2 shows that the cow is facing to the right, similar to the previously acquired skeletal model, and the positions of the joint markers are consistently within a predetermined range over time.

[0059] However, the third skeletal model estimated in the third period Δt3 incorrectly detects the cow as facing left, even though it is actually facing right. In this third skeletal model M3, the position of at least one joint marker (e.g., the head or the base of the neck) deviates significantly from the previous model. The error standardization module 62 detects this abnormal change in marker position and treats the third skeletal model M3 as an error.

[0060] This process maintains the correct skeletal model in the second period Δt2, allowing the system to ignore inaccurate skeletal models resulting from false positives. The error standardization module 62 thus effectively filters out incorrectly acquired models based on a time-series consistent skeletal model.

[0061] In the above process, the second period Δt2 must be at least the predetermined first hour. For example, if the second period Δt2 is made extremely short, the posture detection system 100 may incorrectly detect a wrong posture for the target cow. This first hour is set appropriately according to the resolution of the image obtained by the imaging device 20 and the accuracy of the skeletal estimation unit 12a that estimates the skeletal model M. The third period Δt3 is shorter than the second period Δt2 and must be at least the predetermined second hour. The second hour may be less than or equal to the time of one frame of the image obtained by the imaging device 20, and can be set appropriately as long as it is shorter than the first hour. If a skeletal model M is obtained for more than the second hour (more than the first hour), there is a high probability that the skeletal model M is correct. In this case, the posture detection system 100 considers the skeletal model M to be correct and uses it for posture detection.

[0062] 3.1.4. Detection of the recumbent state of target cattle using skeletal model M Figure 10 is an example of an image of a target cow used for detecting a recumbent state in the posture detection system 100 according to Embodiment 1. The target cow shown in Figure 10(a) is in a recumbent state that may cause difficulty in standing up, and if this posture continues for a long time, it may result in death of the cow. Figure 10(a) shows a cow in a recumbent state, and the forelimbs and hindlimbs are extended straight. Figure 10(b) is an example showing a state where the forelimbs are bent, which indicates a state of the forelimbs that does not correspond to a recumbent state. Also, Figure 10(b) is an example showing a state where the forelimbs are extended, and if the forelimbs are in this state, there is a possibility that the cow is in a recumbent state. In the posture detection system 100, markers are set for multiple joints of the forelimbs and hindlimbs by the skeletal estimation unit 12a, and the recumbent state can be determined based on their positions.

[0063] The recumbent position detection unit 13 analyzes the posture based on the positions of each joint detected by the skeletal estimation unit 12a. This analysis calculates the angle θ of the virtual line formed between the joints and the distance L between the joints. If these values ​​exceed a predetermined threshold, the system determines that the cow is recumbent. This function allows the posture detection system 100 to detect the posture of the target cow in real time and accurately determine whether or not it is in a recumbent state. The above-mentioned angle θ and distance L are sometimes referred to as skeletal features. Furthermore, skeletal features are not limited to those described above and include any values ​​obtained based on the positions of each joint output by the skeletal estimation unit 12a. For example, skeletal features include the coordinates of the nodes representing each joint and numerical values ​​calculated using these coordinates.

[0064] 3.1.5. Determination of the condition of the limbs based on the distance L between joints The recumbent position detection unit 13 can determine whether the target cow is in a recumbent position based on the state of the forelimbs and hindlimbs of the skeletal model M. For this determination, as an example, the distance L of the nodes RH1 and RH3, LH1 and LH3 corresponding to the forelimbs, and the distance L of the nodes RL1 and RL3, LL1 and LL3 corresponding to the hindlimbs are used. When each distance L is determined to be such that the forelimbs and hindlimbs are extended straight (or nearly straight), the recumbent position detection unit 13 can determine that the target cow is in a recumbent position.

[0065] Specifically, as shown in (Equation 1) above, a standardized distance L is used to determine whether the forelimbs and hindlimbs are extended. For example, the forelimb nodes RH1 and LH1 correspond to the shoulder joint of a cow, and the forelimb nodes RH3 and LH3 correspond to the carpal joint of a cow. Therefore, if the distance between nodes RH1 to RH3 and LH1 to LH3 is equal to or close to the length of the cow's forelimb, it can be determined that the cow's forelimb is extended straight (or nearly so). In other words, the posture detection system 100 determines that the forelimb is extended straight (or nearly so) when the distance L between these nodes exceeds a certain threshold L1. The threshold L1 is adjusted considering the error in the number of pixels between nodes obtained from the image, the variation in forelimb length among individual cows, and the error in the estimated skeletal model M.

[0066] The posture detection system 100 determines that the hind limbs are in a straightened state (or close to a straightened state) when the distance L exceeds a certain threshold L1, similar to the forelimb nodes RH and LH. The threshold L1 may be set to a different value than that of the forelimbs, and preferably a threshold L1 unique to the hind limbs is set.

[0067] The image and skeletal model M shown in Figure 10(a) represent a cow that was determined to be in a recumbent position. The distances between the nodes RH1-RH3, LH1-LH3, RL1-RL3, and LL1-LL3 of the cow's right forelimb exceed the threshold L1. In particular, although node RL2 of the right hindlimb is not estimated (node ​​RL2 is not displayed), the positions of nodes RL1 and RL3 are estimated, so the distance L between nodes RL1-RL3 can be determined, and by comparing this with the threshold L1, it is possible to determine that the cow is in a recumbent position.

[0068] 3.1.6. Determination of the condition of the limbs based on the angle formed by imaginary lines between joints. Furthermore, the recumbent position detection unit 13 can also determine whether the animal is in a recumbent position by using the angles between virtual lines connecting the nodes of the forelimbs and hindlimbs. For example, it can determine whether the virtual line connecting nodes RH1 (LH1) and RH2 (LH2), which correspond to the forelimbs, and the virtual line connecting nodes RH2 (LH2) and RH3 (LH3), are straight or nearly straight. Similarly, it can determine whether the virtual line connecting RL1 (LL1) and RL2 (LL2), which correspond to the hindlimbs, and the virtual line connecting RL2 (LL2) and RL3 (LL3), which correspond to the hindlimbs, are straight or nearly straight.

[0069] To determine whether the virtual lines are straight or nearly straight, using the nodes of the right forelimb as an example, this can be determined by the angle between vector v1 between RH1 and RH2 and vector v2 between RH2 and RH3. Whether the right forelimb is extended or not is determined by whether the angle θ between vector v1 and vector v2 exceeds a threshold θ1. The threshold θ1 is set to, for example, 150°. The left forelimb is determined to be straight or not in a similar manner.

[0070] For the hind limbs, the same determination is made as with the example of the right foreleg above: whether or not they are straight.

[0071] In the image and skeletal model M shown in Figure 10(a), the angles of the lines connecting nodes RH1 and RH2, and RH2 and RH3, in the right forelimb are close to 180°. This is also true for the left forelimb and left hindlimb.

[0072] As described above, the recumbent position detection unit 13 determines whether the forelimbs and hindlimbs are extended straight (or nearly straight) based on whether the distance L between nodes corresponding to joints or the angle θ formed by the imaginary lines between joints exceeds a threshold. The recumbent position detection unit 13 basically determines that the cow is in a recumbent position if all of the forelimbs and hindlimbs are extended straight (or nearly straight). In other words, the recumbent position detection unit 13 does not determine that the cow is in a recumbent position if only the forelimbs or only the hindlimbs are extended (see the determination in step S55 of Figure 13 described later).

[0073] In addition, the posture detection system 100 may not be able to identify either the forelimb or hindlimbs from the image. In other words, there may be cases where one of the forelimbs or hindlimbs is not visible in the image. Even in this case, the posture detection system 100 generates a skeletal model M in the skeletal estimation unit 12a, and determines the state of the forelimbs and hindlimbs based on this skeletal model M.

[0074] In the skeletal model M generated by the skeletal estimation unit 12a, some nodes may not be able to be estimated, as shown in Figure 10(a). Even in such cases, the posture detection system 100, as explained above, determines whether each limb is extended using two criteria: the distance L between joints and the angle θ formed by the virtual lines between joints, thus achieving high accuracy in determining the lying-down position.

[0075] Furthermore, the system incorporates a mechanism to detect abnormalities if the patient remains in a lying-down position for a certain period of time. This monitoring of the duration prevents false detections due to temporary body movements, resulting in highly accurate assessments.

[0076] 3.1.7 Posture determination model using skeletal model M Figure 11 is a flowchart showing an example of the configuration of a posture determination model 72 using a skeletal model M of the posture detection system 100 according to Embodiment 1. As described above, the recumbent detection unit 13 can detect a recumbent state using two criteria: the distance L between joints and the angle θ formed by the virtual lines between joints. However, as shown in Figure 11, the recumbent state may also be detected using a posture determination model 72 that has been trained in advance using training data 73, which consists of at least one of image data and the skeletal model M, along with the corresponding recumbent state determination results. In this embodiment, the posture determination model 72 is applied to the skeletal model M to determine the recumbent state of the target cow (for example, the probability of being in a recumbent state), and the final recumbent state determination value is calculated by combining it with the recumbent state determination value obtained by image recognition (for example, the probability of being in a recumbent state), which will be described later. After that, the state of the forelimbs and hindlimbs, as described in Chapters 3.1.5 and 3.1.6 above, is determined, and then the recumbent state is detected. This processing flow will be described in Chapter 4.

[0077] The posture determination result of the target cow generated by the posture determination model 72 is sent to the evaluation and comparison unit 75, where user evaluations and feedback may be incorporated. The evaluation and comparison unit 75 determines whether the estimated determination result is accurate, and the result is fed back to the posture determination model 72, thereby continuously improving the accuracy of detecting the reclining state. In this way, the posture determination model 72 can learn based on data and evaluations obtained during operation. As shown in Figure 11, the user's evaluation results are reflected in the model through the evaluation and comparison unit 75 and used to improve accuracy.

[0078] Furthermore, the training data 73 is a dataset used during the initial training of the system and includes at least a combination of the skeletal model M and its corresponding judgment result. Based on this dataset, the posture judgment model 72 is trained using a machine learning algorithm. Through this training, the posture judgment model 72 acquires the ability to detect the recumbent position with high accuracy.

[0079] Furthermore, the evaluation and comparison unit 75 may compare not only the results of manual evaluation by the user, but also the estimated results in actual operation with known ground truth data. For example, when a cow assumes a specific posture, the system can automatically evaluate whether the judgment result based on that posture is accurate. In this way, a feedback loop for improving the accuracy of the posture judgment model 72 may be formed.

[0080] 3.2. Determination of the lateral recumbent position using image recognition. The posture detection system 100 uses both a means for determining a reclining state using the skeletal model M described above and a means for determining a reclining state using image recognition technology when detecting a reclining state. The image recognition unit 12b targets the entire image, including the target cow and the cow's living environment, and determines the posture of the target cow included in the image. Since the image acquired by the imaging device 20 includes not only the target cow but also the environment surrounding the target cow, the image recognition unit 12b determines the posture of the target cow based on factors such as the positional relationship between the ground and the target cow, and the positional relationship between the target cow and the surrounding environment such as pillars.

[0081] For example, looking at the target cow shown in Figure 5, both forelimbs and hindlimbs of the skeletal model M are extended. Therefore, if the determination is made based solely on the skeletal model M, the recumbent state will be determined from the state of the cow's limbs, potentially leading to a false detection of the cow being recumbent when it is standing. For this reason, the posture detection system 100 also uses the acquired image to determine the cow's posture, including the surrounding environment. However, the posture detection system 100 is not necessarily limited to using both posture determination by the skeletal model M and posture determination by image recognition; it is also possible to detect a recumbent state using only one of these methods.

[0082] The image recognition unit 12b can perform the following processing, for example, when determining the posture of the target cow.

[0083] The image recognition unit 12b may perform preprocessing such as noise reduction, brightness adjustment, and contrast adjustment as preprocessing for image recognition.

[0084] The image recognition unit 12b may detect the target cow from the image using object detection algorithms (e.g., YOLO (You Only Look Once), Faster R-CNN (Region-based Convolutional Neural Networks)), segmentation (e.g., Mask R-CNN), etc.).

[0085] The image recognition unit 12b may extract features necessary for determining the posture of the target cow from the image. Since the target cow and the background are separated from the image by the object detection algorithm and segmentation described above, the image recognition unit 12b may use this result to output numerical features such as the shape of the target cow, the positional relationship between the target cow and the ground or surrounding equipment (such as pillars), and the tilt of the target cow's body.

[0086] As a specific example, the image recognition unit 12b outputs a numerical value between 0 and 1 indicating the probability that the target cow captured in the image is in a reclining position, based on the above-mentioned features. The reclining state probability output by the image recognition unit 12b may be output for each of the multiple target cows captured in the image, or it may be output for all of the multiple target cows. The outputted reclining state probability is sent to the reclining detection unit 13, which then decides whether to provide notification and play audio.

[0087] To determine the probability of the cow lying down, a configuration using a machine learning model similar to the posture determination model 72 described in Figure 11 can be used. In posture determination by the image recognition unit 12b, the posture determination model 72 is trained using a combination of images in which the target cow is captured and the posture determination result of the target cow as training data 73.

[0088] 4. Flow of information processing in the posture detection system 100 4.1. Overall configuration of information processing for posture detection system 100 Figure 12 shows an example of the information processing flow performed in the posture detection system 100 according to Embodiment 1. The posture detection system 100 according to Embodiment 1 performs the following information processing using an imaging device 20 and an audio playback device 30 installed in the cattle rearing environment 80, a detection device 10, and an external terminal 40.

[0089] First, images of the cow's living environment 80 are acquired at time intervals of Δt using the imaging device 20 (step S1). Time Δt can be set arbitrarily, but it should be set within a range that does not impair the cow's health when it is lying down. Also, time Δt may be a video set to 24, 30, 60, 120 frames per second, etc.

[0090] Next, the skeletal estimation unit 12a generates a skeletal model M for the target cows captured in the images acquired at time intervals Δt. The skeletal estimation unit 12a estimates the positions of the joints of the target cows captured in the images and outputs the skeletal model M (step S2). In addition, the coordinates of the nodes located at each estimated joint are obtained. The coordinates of the nodes are generated for each target cow.

[0091] Next, the distance L between each joint is determined (step S3). After the distance L between each joint is determined, error standardization processing is performed in the error standardization module (step S4). In this error standardization processing, the positions of each joint estimated in the skeletal model M are analyzed in a time series and verified to see if there are any abnormal positional relationships. For example, if an anomaly is detected, such as the node, which is the estimated position of the joint, exceeding a predetermined range l from the position of the node estimated in the previously obtained image, it is processed as an error, and an accurate skeletal model M is maintained.

[0092] In parallel with the processing in the skeletal estimation unit 12a, the image recognition unit 12b recognizes the acquired image and evaluates the posture of the target cow captured in the image. The evaluation result is sent to the reclining detection unit 13 (step S10).

[0093] Once the determination of the target cow's posture and error standardization processing using image recognition are complete, the next step is to detect the cow's recumbent state (step S5). The recumbent state detection unit 13 determines whether the cow is in a recumbent state based on the skeletal model M and the probability of the target cow being in a recumbent state determined by image recognition. Specifically, if the probability of the target cow being in a recumbent state determined by image recognition is above a threshold and the positional relationship and distance of the joints exceed predetermined criteria, the system determines that the cow is in a recumbent state (Yes in step S5). If the cow is not in a recumbent state, it is determined to be in a normal posture (No in step S5), and the process returns to step S1. Details of the process in step S5 will be explained later using Figure 13.

[0094] If the cow is determined to be in a reclining position (Yes in step S5), the reclining detection unit 13 outputs the detection result and sends it to the sound playback unit 14. The sound playback unit 14 plays a sound using the sound playback device 30 (step S6). The sound should preferably be one that will attract the cow's attention, and can be a human voice (owner's voice) or other sound. This allows the posture detection system 100 to quickly take action to correct the cow's reclining position when it detects that the cow is lying down. Alternatively, other stimuli may be given to the cow instead of sound.

[0095] If the cow is detected to be lying down (if the answer is Yes in step S5), the lying-down detection unit 13 outputs the detection result and sends it to the notification unit 18. The notification unit 18 notifies the external terminal 40 (step S9). The external terminal 40 is, for example, a portable information terminal held by the user, and notification is made quickly after the cow has fallen into a lying-down state. Alternatively, the external terminal 40 may be a device that emits an alarm sound and may be installed, for example, in an office where barn staff are located. After notification, the user can go to the barn where the cow that has fallen into a lying-down state is located and actually check the cow's condition, and it may also be possible to manually resolve the cow's lying-down state.

[0096] In step S6, the sound playback device 30 plays sound, but it may be set to stop playing sound after a predetermined time (step S7). Alternatively, in step S6, the same process as in steps S1 to S4 may be performed, and the sound may be stopped after detecting that the cow's lying-down state has been resolved (step S7). Generally, if sound is played continuously from the time the cow becomes lying down until it is resolved, it may affect other healthy cows, so it is desirable to stop sound playback after a predetermined time.

[0097] After the audio playback stops, the posture detection system 100 will, after a predetermined time has elapsed (step S8), repeat the process of detecting the lying-down state from step S1. Alternatively, the posture detection system 100 may start processing from step S1 immediately after detecting the lying-down state. In other words, the posture detection system 100 constantly acquires images of the cow's living environment 80 and constantly performs estimation of the skeletal model M, but it may be configured not to provide notification and play audio again until a predetermined time has elapsed when the lying-down state occurs.

[0098] Through the above series of processes, the posture detection system 100 efficiently detects when a cow is lying down, enabling early intervention.

[0099] 4.2. Detailed information processing flow for lateral recumbent state detection Figure 13 shows an example of the information processing flow for step S5 in Figure 12. The process of detecting the recumbent state in step S5 is performed by integrating the results of the skeletal model and image recognition. Specifically, as shown in Figure 12, first, the posture determination result based on features is extracted from the skeletal model M (step S51), then the posture determination result by the image recognition algorithm is obtained (step S52), and these are integrated to calculate a recumbent state determination value (step S53). It is determined whether the recumbent state determination value obtained by integration is above a threshold (step S54). After that, it is determined using the skeletal model M whether the forelimbs and hindlimbs are extended (step S55). Furthermore, if it is determined in step S55 that the forelimbs and hindlimbs are extended (Yes in step S55), it is determined whether that state has continued for a predetermined time (step S56). If it is determined in step S56 that the recumbent state has continued for a predetermined time (Yes in step S56), notification actions and sound playback are performed. This integrated approach improves the accuracy of posture detection, which was difficult to detect using a single method, and suppresses false detections, notifications, and audio playback.

[0100] The posture determination results obtained in steps S51 and S52 include, for example, the probability of lying down based on an image recognition algorithm and the probability of lying down based on the features of the skeletal model M. The lying-down detection unit 13 calculates the final lying-down state determination value based on these lying-down state probabilities (step S53), and determines that a lying-down state has been detected if the lying-down state probability exceeds a certain threshold (Yes in step S54).

[0101] Steps S55 and S56 are processes to improve the accuracy of the reclining state detection in the reclining detection unit 13 and to suppress false detections, notifications, and audio playback, and may be performed, for example, before step S54. In other words, the decisions in steps S54 to S56 can be performed in any order.

[0102] In the above explanation, we described a method in which the final reclining state determination value is calculated in step S54 using the reclining state probability based on image recognition and the reclining state probability based on the features of the skeletal model M. However, the detection of the reclining state may be performed in other ways. For example, the reclining state probability based on image recognition may be referenced only when the reclining state probability based on the skeletal model M is above a certain threshold, and if it is above a certain threshold, the process may proceed to the next step S55. Alternatively, the reverse pattern may be used, where the reclining state probability based on image recognition is determined first, and then the determination is made using the skeletal model M.

[0103] The posture determination result by the skeletal model M is expressed as a binary value such as "yes" or "no" for the recumbent state, and the posture determination result by image recognition is expressed as a probability of the recumbent state (e.g., a value between 0 and 1). The recumbent detection unit 13 may be configured to refer to the posture determination result by image recognition only when the posture determination result by the skeletal model M indicates a recumbent state, and to proceed to the next step S55 only if the posture determination result by image recognition is above a certain threshold. Alternatively, the posture determination result by image recognition may be expressed as a binary value such as "yes" or "no" for the recumbent state, the posture determination result by the skeletal model M may be expressed as a probability of the recumbent state, and the posture determination result by the skeletal model M may be referred to only when the posture determination result by image recognition indicates a recumbent state.

[0104] As described above, the output content of the judgment results based on image recognition and the judgment results based on the skeletal model can be changed as appropriate. Alternatively, one of the judgment results may be prioritized when outputting the detection result for a reclining state. These settings can be adjusted as appropriate depending on the images acquired by the posture detection system 100, the number of target cows, the environment surrounding the target cows, etc.

[0105] 5. Effects of Embodiment 1 The posture detection system 100 according to Embodiment 1 is a posture detection system 100 that can also detect when a cow is lying down, and comprises an imaging device 20 that acquires images and a detection device 10 that detects when a target cow is lying down as captured in the images. The detection device 10 includes a skeleton estimation unit 12a that generates a skeleton model M that estimates the skeleton of the target cow, and performs posture determination of the target cow based on the skeleton model M. With this configuration, the posture detection system 100 determines the posture of a cow in the breeding environment 80 where the imaging device 20 is installed using an estimated skeletal model M, thereby enabling highly accurate management of the cow's condition. Furthermore, since the skeletal model M is estimated based on the image of the target cow captured in the image acquired by the imaging device 20, there is no need to attach devices to the cow. In addition, the posture of the target cow can be detected in the same way whether the image contains one target cow or multiple target cows.

[0106] In the posture detection system 100 described above, the skeletal estimation unit 12a estimates the positions of multiple joints in the forelimbs and hindlimbs of the target cow. The recumbent position detection unit 13 obtains skeletal features from the positional relationships of the estimated multiple joints in the forelimbs and hindlimbs. Furthermore, in the posture detection system 100 described above, the skeletal features are parameters based on the distances between multiple joints of the forelimb and multiple estimated joints of the hindlimbs. Furthermore, in the posture detection system 100 described above, the skeletal estimation unit 12a estimates the positions of three consecutive joints in the forelimb and hindlimbs of the target cow. The skeletal feature is the angle formed by two virtual lines connecting adjacent joints among the three estimated joints in the forelimb and hindlimbs. With this configuration, the posture detection system 100 can detect the state of the forelimbs and hindlimbs, which are characteristic of a cow's reclining position, from images. The posture detection system 100 directly detects the state of the forelimbs and hindlimbs using a skeletal model M obtained from images, resulting in high accuracy in detecting the cow's reclining position. In particular, when a cow is reclining, its forelimbs and hindlimbs are extended straight, so the posture detection system 100 can accurately detect this state by understanding the positional relationship of the nodes in the skeletal model M. The state in which the cow's forelimbs and hindlimbs are extended straight can be easily estimated if the positional relationship of the nodes, which are the joints of the cow's forelimbs and hindlimbs, is known from the skeletal model M. Therefore, the accuracy of detecting the reclining position can be improved by using the distance between the nodes corresponding to the forelimbs and hindlimbs, or the angle formed by the virtual lines connecting the nodes.

[0107] Furthermore, in the posture detection system 100 described above, the detection device 10 includes an image recognition unit 12b that determines the posture of the target cow based on the target cow and the background surrounding the target cow captured in the image, and a reclining detection unit 13 that outputs a detection result indicating that the target cow is in a reclining position. The reclining detection unit 13 outputs a detection result based on the posture determination result from the image recognition unit 12b in addition to the posture determination result from the skeletal features. With this configuration, the posture detection system 100 can detect a recumbent state using posture determination that takes into account the relationship between the target cow and the environment surrounding the target cow, performed by the image recognition unit 12b. This improves accuracy compared to detecting a recumbent state using only the skeletal model M.

[0108] Furthermore, in the posture detection system 100 described above, the reclining detection unit 13 outputs a detection result when the posture determination result that the target cow is in a reclining state continues for a preset first period of time. Furthermore, in the posture detection system 100 described above, the imaging device 20 transmits multiple images to the detection device during a predetermined first period. With this configuration, the posture detection system 100 uses multiple images obtained during the first period Δt1 to detect the bovine skeletal model M, and only after confirming that the reclining state has continued for a predetermined period does it play audio and notify the user. Therefore, it is possible to suppress the posture detection system 100 from mistakenly detecting the reclining state from the images and playing audio and notifying the user.

[0109] Furthermore, in the above posture detection system 100, the detection device 10 further includes an error standardization module 62 that standardizes the multiple skeletal models M estimated by the skeletal estimation unit 12a for each of the multiple images. The error standardization module 62 defines, from the multiple skeletal models M estimated in the first period Δt1, a plurality of second skeletal models obtained in the second period Δt2 which is included in the first period Δt1 and shorter than the first period Δt1, and at least one third skeletal model obtained in the third period Δt3 which is included in the first period Δt1 and shorter than the first period Δt1 and follows the second period Δt2. Furthermore, the third skeletal model is treated as an error if the positions of the markers for each of the estimated multiple joints of the multiple second skeletal models are located within a predetermined range relative to the corresponding markers of the skeletal model obtained in the previous time series, and the position of at least one of the markers for the estimated multiple joints of the third skeletal model is located outside a predetermined range relative to the corresponding markers of the skeletal model obtained in the previous time series, and the second period Δt2 is at or above a predetermined 1 hour, and the third period Δt3 is shorter than the second period Δt2 and at or below a predetermined 2 hours. With this configuration, the posture detection system 100 can reliably and correctly detect the posture of the target cow captured in the image, even if it partially misdetects the cow's posture in multiple images obtained during the first period Δt1, by processing this as an error.

[0110] Furthermore, in the posture detection system 100 described above, the skeletal estimation unit 12a includes a skeletal estimation model 70 that has been learned based on a sample image in which a cow is captured and the position information of the cow's joints captured in the sample image. As a result, the posture detection system 100 can accurately detect the posture of the target cow from the images obtained by the imaging device 20. Furthermore, by applying AI, the posture detection system 100 can predict the skeletal model M even if not all parts of the target cow are captured in the images obtained by the imaging device 20. In addition, the posture detection system 100 can improve the accuracy of the skeletal estimation unit 12a and the recumbent position detection unit 13 by evaluating the images obtained during operation and the skeletal model M estimated from those images.

[0111] Furthermore, in the posture detection system 100 described above, the skeleton estimation unit 12a includes a sample image in which a cow is captured, a skeleton model of the cow captured in the sample image, and a posture determination model 72 that has been learned based on the results of the posture determination of the cow captured in the sample image. Furthermore, in the posture detection system 100 described above, the image recognition unit 12 includes a sample image in which a cow is captured and a posture determination model 72 that has been learned based on the results of the posture determination of the cow captured in the sample image. As a result, the posture detection system 100 can accurately detect the posture of the target cow from the image obtained by the imaging device 20. Furthermore, the posture detection system 100 can determine the posture of the target cow using the posture determination model 72, and can output posture determination values ​​such as the probability of lying down, and can also use these posture determination values ​​to detect and output the state of lying down.

[0112] Furthermore, the posture detection system 100 described above includes an audio playback device that plays a predetermined sound when the reclining detection unit 13 outputs a detection result indicating that the target cow is in a reclining position. This allows the posture detection system 100 to respond quickly when a cow being raised in a state of abnormality occurs.

[0113] Furthermore, in the posture detection system 100 described above, when the reclining detection unit 13 outputs a result indicating that the target cow is in a reclining position, it notifies an external terminal. This allows the posture detection system 100 to quickly notify the user when a cattle being raised is in an abnormal state, enabling the user to quickly check the health status of the cattle and take appropriate action manually.

[0114] Although the present disclosure has been described above based on embodiments, the present disclosure is not limited to the configuration of the embodiments described above. Furthermore, in the above embodiments, the posture detection system 100 was described as a system for detecting the lying-down position of a cow, but it is not limited to this and can also detect other postures based on the skeletal model M. In addition, notification and sound playback can also be performed by detecting postures other than the lying-down position.

[0115] Furthermore, in the above embodiments, the posture detection system 100 was implemented using a system including a client-server system of a network computer system. However, the same functions as the posture detection system 100 can also be implemented in various computers such as personal computers that do not constitute a client-server system, or in various communication terminals and mobile information terminals such as mobile terminals and tablets. It is also possible to implement at least a part of the functions of the posture detection system 100 by implementing a computer program in various computers or various communication terminals and mobile information terminals. In other words, this disclosure also includes a program to make various computers function as at least a part of the posture detection system 100. In addition, the above embodiments and modifications may be implemented in combination as appropriate. For the sake of clarity, it should be noted that the scope of various modifications, applications, and uses that a person skilled in the art may make as needed is also included in the gist (technical scope) of this disclosure.

[0116] Furthermore, the posture detection system 100 described above may also include combinations of the features shown in the following appendices 1 to 13. These combinations are shown below.

[0117] [Note 1] A posture detection system for detecting when a cow is lying down, An imaging device that acquires images, The system includes a detection device that detects the reclining position of the target cow captured in the aforementioned image, The detection device is The system includes a skeletal estimation unit that generates a skeletal model by estimating the skeleton of the target cow. The posture of the subject cow is determined based on the aforementioned skeletal model. Posture detection system. [Note 2] The posture detection system described in Appendix 1, The aforementioned skeleton estimation unit is The positions of multiple joints in the forelimbs and hindlimbs of the aforementioned target cows are estimated. The detection device is Skeletal features are determined from the estimated positional relationships of the multiple joints in the forelimb and hindlimbs, respectively. Posture detection system. [Note 3] The posture detection system described in Appendix 2, The aforementioned skeletal features are, A parameter based on the distance between the multiple joints of the forelimb and the estimated multiple joints of the hindlimbs, Posture detection system. [Note 4] A posture detection system as described in Appendix 2 or 3, The aforementioned skeleton estimation unit is The positions of three consecutive joints in the forelimbs and hindlimbs of the aforementioned subject cow are estimated. The aforementioned skeletal features are, This is the angle formed by two imaginary lines connecting adjacent joints among the three estimated joints of the forelimb and hindlimbs, respectively. Posture detection system. [Note 5] A posture detection system described in any one of the appendices 1 to 4, The detection device is An image recognition unit that determines the posture of the target cow based on the target cow and the background surrounding the target cow captured in the aforementioned image, The system includes a reclining detection unit that outputs a detection result indicating that the target cow is in a reclining position, The aforementioned lying-down detection unit is In addition to the results of posture determination based on the skeletal features, the detection results are output based on the results of posture determination of the target cow by the image recognition unit. Posture detection system. [Note 6] A posture detection system described in any one of the appendices 1 to 5, The system includes a reclining detection unit that outputs a detection result indicating that the target cow is in a reclining position, The aforementioned lying-down detection unit is The detection result is output when the result of the posture determination that the target cow is in a lying-down position continues for a predetermined first period. Posture detection system. [Note 7] A posture detection system as described in Appendix 5 or 6, The system includes an audio playback device that plays a predetermined sound when the reclining detection unit outputs the detection result. Posture detection system. [Note 8] A posture detection system as described in Appendix 5 or 6, When the aforementioned lying-down detection unit outputs the detection result, it notifies an external terminal. Posture detection system. [Note 9] A posture detection system described in any one of the appendices 1 to 8, The imaging device is Multiple images are transmitted to the detection device during the predetermined first period Δt1. Posture detection system. [Note 10] The posture detection system described in Appendix 9, The detection device is The system further includes an error standardization module that standardizes the multiple skeletal models estimated by the skeletal estimation unit for each of the multiple images, The error standardization module is When we define a plurality of second skeletal models obtained in a second period Δt2 which is included in the first period Δt1 and shorter than the first period Δt1, and at least one third skeletal model obtained in a third period Δt3 which is included in the first period Δt1 and shorter than the first period Δt1 and follows the second period Δt2, The positions of the markers for each of the estimated multiple joints of the multiple second skeletal models are located within a predetermined range relative to the corresponding markers of the skeletal model obtained in the previous chronological order. The position of at least one of the estimated joint markers of the third skeletal model is outside a predetermined range relative to the corresponding marker of the skeletal model obtained in the previous time series, If the second period Δt2 is at least a predetermined first hour, and the third period Δt3 is shorter than the second period Δt2 and at least a predetermined second hour, the third skeletal model is treated as an error. Posture detection system. [Note 11] A posture detection system described in any one of the appendices 1 to 10, The aforementioned skeleton estimation unit is The system comprises a sample image containing a cow and a skeletal estimation model trained based on the positional information of the cow's joints captured in the sample image. Posture detection system. [Note 12] The posture detection system described in Appendix 11, The aforementioned skeleton estimation unit is The system comprises a sample image containing a cow, a skeletal model of the cow captured in the sample image, and a posture determination model learned based on the results of the posture determination of the cow captured in the sample image. Posture detection system. [Note 13] A posture detection system described in any one of the appendices 5 to 12, The image recognition unit, The system includes a sample image containing a cow and a posture determination model trained based on the results of posture determination of the cow in the sample image. Posture detection system. [Explanation of symbols]

[0118] 1: Information Processing Device 10: Detection device 11: Image receiving unit 12: Skeletal Estimation Section 13: Lying position detection unit 14: Audio playback unit 15: Database Department 16: Image storage unit 17: Voice Memory Unit 18: Hochi Department 19:Display section 20: Imaging device 30: Audio playback device 40: External terminals 51: Control Unit 52: Storage section 53: Communications Department 54: Input section 55: Output section 56: Imaging Department 61: Distance Calculation Module 62: Error Standardization Module 70: Skeletal Estimation Model 71: Training data 72: Error Standardization Module 80: Rearing environment 90: Network 100: Posture detection system

Claims

1. A posture detection system for detecting when a cow is lying down, An imaging device that acquires images, The system includes a detection device that detects the reclining position of the target cow captured in the aforementioned image, The detection device is The system includes a skeletal estimation unit that generates a skeletal model by estimating the skeleton of the target cow. The posture of the subject cow is determined based on the aforementioned skeletal model. Posture detection system.

2. A posture detection system according to claim 1, The aforementioned skeleton estimation unit is The positions of multiple joints in the forelimbs and hindlimbs of the aforementioned target cows are estimated. The detection device is Skeletal features are determined from the estimated positional relationships of the multiple joints in the forelimbs and hindlimbs of the subject cow. Posture detection system.

3. A posture detection system according to claim 2, The aforementioned skeletal features are, These are parameters based on the distance between multiple joints in the forelimbs and the estimated distance between multiple joints in the hindlimbs of the subject cow. Posture detection system.

4. A posture detection system according to claim 2 or 3, The aforementioned skeleton estimation unit is The positions of the three consecutive joints in the forelimbs and hindlimbs of the aforementioned subject cow are estimated. The aforementioned skeletal features are, This is the angle formed by two imaginary lines connecting adjacent joints among the three estimated joints of the forelimb and hindlimbs of the subject cow. Posture detection system.

5. A posture detection system according to claim 1 or 2, The detection device is An image recognition unit that determines the posture of the target cow based on the target cow and the background surrounding the target cow captured in the aforementioned image, The system includes a reclining detection unit that outputs a detection result indicating that the target cow is in a reclining position, The aforementioned lying-down detection unit is In addition to the results of posture determination based on the skeletal features, the detection results are output based on the results of posture determination of the target cow by the image recognition unit. Posture detection system.

6. A posture detection system according to claim 5, The aforementioned lying-down detection unit is The detection result is output when the result of the posture determination that the target cow is in a lying-down position continues for a predetermined first period. Posture detection system.

7. A posture detection system according to claim 5, The system includes an audio playback device that plays a predetermined sound when the reclining detection unit outputs the detection result. Posture detection system.

8. A posture detection system according to claim 5, When the aforementioned lying-down detection unit outputs the detection result, it notifies an external terminal. Posture detection system.

9. A posture detection system according to any one of claims 1 to 3, The imaging device is Multiple images are transmitted to the detection device during the predetermined first period Δt1. Posture detection system.

10. A posture detection system according to claim 9, The detection device is The system further includes an error standardization module that standardizes the multiple skeletal models estimated by the skeletal estimation unit for each of the multiple images, The error standardization module is When we define a plurality of second skeletal models obtained in a second period Δt2 which is included in the first period Δt1 and shorter than the first period Δt1, and at least one third skeletal model obtained in a third period Δt3 which is included in the first period Δt1 and shorter than the first period Δt1 and follows the second period Δt2, The positions of the markers for each of the estimated multiple joints in the aforementioned multiple second skeletal models are located within a predetermined range relative to the corresponding markers of the skeletal model obtained in the previous chronological order. The position of at least one of the markers for multiple joints estimated in the third skeletal model is outside a predetermined range relative to the corresponding marker of the skeletal model obtained in the time series immediately preceding, If the second period Δt2 is at least a predetermined first hour, and the third period Δt3 is shorter than the second period Δt2 and at least a predetermined second hour, the third skeletal model is treated as an error. Posture detection system.

11. A posture detection system according to any one of claims 1 to 3, The aforementioned skeleton estimation unit is The system comprises a sample image containing a cow and a skeletal estimation model trained based on the positional information of the cow's joints captured in the sample image. Posture detection system.

12. A posture detection system according to claim 11, The aforementioned skeleton estimation unit is The system comprises a sample image containing a cow, a skeletal model of the cow captured in the sample image, and a posture determination model trained based on the results of the posture determination of the cow captured in the sample image. Posture detection system.

13. A posture detection system according to claim 5, The image recognition unit, The system includes a sample image containing a cow and a posture determination model trained based on the results of posture determination of the cow in the sample image. Posture detection system.

Images