Information processing device, information processing method, and program

Abstract

To enable the individual identification of an animal related to a picked-up image on the basis of the picked-up image of the animal in a specific area.SOLUTION: An animal appearance determination device 100 includes an image data analysis part 112 for acquiring a picked-up image including a cow raised in a cow stall, and acquiring the position of the cow in the cow stall calculated on the basis of the image of the cow in the picked-up image, as first position information, a position data analysis part 114 for acquiring identification information from a sensor device 200 attached to the cow and second position information showing the position of the cow in the cow stall, and an individual identification part 115 for associating the first position information with the identification information related to the first second information for recording in the case of determining that difference between the first position information and the second position information is equal to or less than a prescribed threshold.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] Efficient health management of animals is becoming increasingly important for maintaining stable dairy farming. For example, Patent Document 1 describes a livestock disease management system that includes a rearing status acquisition unit that acquires rearing status information including image information of livestock, and a disease risk estimation unit that extracts livestock shown in the image information acquired by the rearing status acquisition unit, analyzes the status of the livestock, and estimates disease risk according to the status of the livestock. Patent Document 2 describes an activity amount calculation program that causes a computer to execute a process of acquiring multiple captured images of multiple animals, detecting movement of attachments worn by each of the multiple animals based on the multiple captured images, and calculating the activity amount of each of the multiple animals based on the detected movement of the attachments. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2021-135956 [Patent Document 2] Japanese Patent Application Laid-Open No. 2018-148843 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the technology in Patent Document 1 has the problem that it can estimate disease risk for a group, but cannot estimate disease risk or manage physical condition for each individual.The technology in Patent Document 2 has the problem that the accuracy of individual identification may decrease when the number of livestock being observed increases or when the attached items cannot be confirmed in the image.

[0005] One of the objects of the present invention is to provide an information processing device, an information processing method, and a program that can accurately identify the position of an animal in a specific area based on an image of the animal in the specific area, thereby enabling individual identification of the animal related to the image. [Means for solving the problem]

[0006] One aspect of the present invention is an information processing device that includes a first location information acquisition unit that acquires an image including an animal kept in a predetermined breeding area and acquires the location of the animal within the breeding area as first location information calculated based on an image of the animal in the image; a second location information acquisition unit that acquires identification information of the animal and second location information indicating the location of the animal within the breeding area from a terminal device provided on the animal; and an individual identification unit that, when it is determined that the difference between the first location information and the second location information is less than a predetermined threshold, associates the first location information with the identification information related to the second location information and records them.

[0007] The information processing device may be equipped with an appearance determination unit that, based on a captured image of the animal, makes a determination on pre-set appearance determination items regarding the appearance of the animal, uses the determination result as appearance information of the animal, obtains the first location information calculated from the image of the animal that was the subject of the determination, and associates and records the identification information associated with the first location information with the appearance information.

[0008] The appearance determination unit may determine the appearance of the animal using a trained model that has learned the relationship between images acquired of the animal and appearance information, which is the determination result for the appearance determination items that have been set in advance.

[0009] The appearance determination unit may determine the appearance of the animal based on an image acquired of the animal and criteria for determining characteristics in the image of the animal that are set in advance for the appearance determination items.

[0010] The second location information may be calculated based on the direction of arrival of radio waves transmitted from a terminal device provided on the animal.

[0011] The appearance information for each individual animal recorded in chronological order by the appearance determination unit may be used to predict at least one of the timing of illness, estrus, parturition, growth, and weaning of the animal.

[0012] Another aspect of the present invention includes a process of acquiring a captured image including an animal kept in a predetermined breeding area, and acquiring the position of the animal in the breeding area calculated based on an image of the animal in the captured image as first location information; a process of acquiring identification information of the animal and second location information indicating the position of the animal in the breeding area from a terminal device provided on the animal; and a process of associating the first location information with the identification information related to the second location information and recording them when it is determined that the difference between the first location information and the second location information is equal to or less than a predetermined threshold; This is an information processing method for performing the above.

[0013] Yet another aspect of the present invention is a program that causes an information processing device to perform the following processes: acquire an image including an animal being kept in a predetermined breeding area; acquire the position of the animal within the breeding area calculated based on an image of the animal in the image; acquire identification information of the animal and second position information indicating the position of the animal within the breeding area from a terminal device provided on the animal; and, if it is determined that the difference between the first position information and the second position information is less than a predetermined threshold, associate the first position information with the identification information related to the second position information and record them. [Effects of the Invention]

[0014] According to the present invention, it is possible to identify the position of an animal in a specific area based on a captured image of the animal in the specific area, and also to identify the individual animal in the captured image. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a schematic diagram illustrating the overall configuration of an animal appearance determination system according to an embodiment of the present invention; [Figure 2] 1 is a block diagram showing an outline of the functions of an animal appearance determination system according to an embodiment of the present invention. [Figure 3] 1 is a block diagram illustrating an example of the configuration of hardware and functional blocks of an animal appearance determining device according to an embodiment of the present invention. [Figure 4] 1 is a block diagram illustrating an example of the configuration of hardware and functional blocks of a sensor device and a position detector according to an embodiment of the present invention. [Figure 5] FIG. 10 is an explanatory diagram showing an example of processing an image of an animal. [Figure 6] 10 is a schematic diagram showing a comparison between the position of an animal identified from an image and the position of an animal identified from identification information from a sensor device; FIG. [Figure 7] FIG. 10 is an explanatory diagram showing examples of appearance determination items. [Figure 8] FIG. 10 is a diagram illustrating an example of the configuration of an appearance determination data storage unit. [Figure 9] FIG. 2 illustrates an example of the configuration of a position information storage unit; [Figure 10] FIG. 10 is a diagram illustrating an example of the configuration of an appearance information storage unit. [Figure 11] FIG. 2 illustrates an example of the configuration of a processing data storage unit; [Figure 12] 10 is a flowchart showing an example of a position information calculation process executed by the animal appearance determining device according to an embodiment of the present invention. [Figure 13] 10 is a flowchart showing an example of an appearance determination process executed by an animal appearance determining device according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0016] The present invention will be described below based on embodiments with reference to the accompanying drawings. Note that the present invention is not limited to the following embodiments. Furthermore, each drawing referred to in the following description merely shows a schematic representation of the shape, size, and positional relationship to the extent that the contents of the present disclosure can be understood. In other words, the present invention is not limited to only the shape, size, and positional relationship illustrated in each drawing.

[0017] (Embodiment) <Animal Appearance Judgment System 1> FIG. 1 is a schematic diagram illustrating the overall configuration of an animal appearance determination system 1 according to one embodiment of the present invention. As shown in FIG. 1 , the animal appearance determination system 1 according to this embodiment has a function for chronologically recording appearance information, which is the result of assessment of predefined appearance assessment items for each individual animal, based on images of the appearance of, for example, a cow C raised by a dairy farmer. The animals to be recorded are not limited to cows C, but may include all animals for which the acquisition and use of appearance information is beneficial, including animals that fall into the category of so-called industrial animals, such as pigs, sheep, horses, and goats, animals kept in zoos, and so-called pets. In this embodiment, four items are set as appearance assessment items: "ear position," "eye opening degree," "head position," and "back curvature." The appearance assessment items are not limited to these items and can be set appropriately depending on the target animal, the purpose of use of the appearance information, and the like.

[0018] Figure 1 is a plan view showing a rectangular cow pen CB placed in a cowshed. For simplicity, Figure 1 shows two cows C1 and C2 being kept in the cow pen CB.

[0019] Within the cow pen CB, a feeding area F, a nursing area M, and a watering area W are provided at appropriate locations so that the cows C1 and C2 can eat feed, drink milk, and drink water at each location. Note that the locations of the feeding area F, nursing area M, and watering area W within the cow pen CB are not limited to the example in Figure 1 and can be determined as appropriate.

[0020] Cows C1 and C2 are fitted with sensor devices 200 equipped with sensor devices for monitoring each cow's behavior, heart rate, and other biological information. Each sensor device 200 is assigned unique identification information, which is used as the identification information for each cow. The sensor devices 200 are provided with a data transmission function, and data acquired from the sensor devices is transmitted together with the identification information by, for example, near field communication (NFC).

[0021] An animal appearance determining device 100, which is an information processing device that implements the main functions for determining the animal appearance in this embodiment, is installed near a cowshed that includes stalls CB, for example, in an administration building of a dairy farm. The animal appearance determining device 100 receives various data and identification information from sensor devices 200 attached to cows C, and uses this data to identify the position of each cow C within stall CB.

[0022] A position detector 300 is provided in the pen CB. The position detector 300 receives radio waves transmitted from the sensor device 200 and generates data for the animal appearance determining device 100 to analyze the position of the sensor device 200 (i.e., the positions of the cows C1 and C2 within the pen CB). In the example of FIG. 1, two position detectors 300 are installed at appropriate locations within the pen CB, detect the angle of the radio waves arriving from the sensor device 200, and transmit the data to the animal appearance determining device 100. The animal appearance determining device 100 uses the angle data from each position detector 300 to analyze and obtain the position of the cow C within the pen CB. As illustrated in FIG. 1, for convenience, X- and Y-axes defining a plane coordinate system, as well as a Z-axis perpendicular to the X- and Y-axes, can be set within the pen CB. The animal appearance determining device 100 may be configured to be able to communicate with an external communication network N, such as the Internet, a WAN, or a LAN. The configurations and functions of the animal appearance determining device 100, the sensor device 200, and the position detector 300 will be described later with reference to the related drawings.

[0023] One or more cameras 400 are installed in the stall CB to capture images of the cows C. It is desirable that images of the cows C kept in the stall CB be captured on a regular basis. Therefore, if the camera 400 is installed so as to capture images of a location in the stall CB where the cows C appear on a regular basis (for example, once a day), such as near a feeding area or a watering area, it is possible to regularly obtain images of each cow C in the stall CB, which is preferable. The camera 400 is, for example, a digital camera having an imaging element such as a CCD, and captures moving images of the inside of the stall CB or still images at predetermined time intervals, and transmits the image data to the animal appearance determining device 100.

[0024] FIG. 2 shows a schematic configuration and functions of the animal appearance determination system 1 of FIG. 1. The animal appearance determination system 1 of this embodiment includes an animal appearance determination device 100. Radio wave reception angle data from a position detector 300 installed in a cow pen CB is input to the animal appearance determination device 100, along with identification information of the sensor device 200. The identification information of the sensor device 200 is included in the data received by the position sensor 300 from the sensor device 200. The animal appearance determination device 100 analyzes the input radio wave reception angle data and the installation position of the position detector 300 using a position data analysis function F1, and calculates the position information of the sensor device 200 within the cow pen CB using a position information calculation function F2. The identification information of the sensor device 200 is linked to each cow C, so the position information of the sensor device 200 indicates the position of the cow C to which the sensor device 200 is attached within the cow pen CB.

[0025] Meanwhile, the animal appearance determining device 100 acquires image data of the inside of the cow pen CB captured by the camera 400. The acquired image data is analyzed by the image analysis function F3, and the images of the cows C contained in each image are extracted by the animal extraction function F4. At this time, the position of each cow C in the image is calculated. The position of the cow C in the image is converted to its actual position in the cow pen CB based on the installation position, image capturing direction, and angle of view of the camera 400 that captured the image.

[0026] The individual identification function F5 compares the position information (second position information) based on the signal received from the sensor device 200 with the position information (first position information) within the stall CB obtained by converting the position of the cow C in the image. If it is determined that there is a correspondence between the position information based on the image and the position information based on the signal from the sensor device 200, the image is identified as an image of the cow C to which the sensor device 200 is attached, for which a correspondence has been determined. As described above, the identification information of the sensor device 200 is associated with the individual information of each cow C, so the image and the individual information of the cow C are associated. A correspondence between the position information based on the image and the position information based on the signal from the sensor device 200 means that the difference (distance) between the positions within the stall CB identified by each piece of position information is equal to or less than a predetermined threshold. In other words, if the position within the stall CB indicated by the position information based on the image and the position within the stall CB indicated by the position information based on the signal from the sensor device 200 are within a certain range, it is determined that the correspondence exists. The threshold value may be determined appropriately depending on the target animal, etc., but in the case of cattle in this embodiment, it can be determined to be on the order of several tens of centimeters, for example.

[0027] The image of cow C extracted from each image by the animal extraction function F4 is subjected to a judgment process from the perspective of its appearance by the appearance judgment function F6. As described above, four appearance judgment items are set: "ear position," "eye opening degree," "head position," and "back curvature," and the appearance judgment function F6 performs a quantitative judgment for each appearance judgment item based on the image of cow C. The image and individual information of cow C associated by the individual identification function F5, and the judgment results based on each image of cow C obtained by the appearance judgment function F6 are associated and recorded in chronological order by the appearance information recording function F7.

[0028] <Animal Appearance Determination Device 100, Sensor Device 200, Position Detector 300> Next, a specific description will be given of the configurations and functions of the animal appearance determining device 100, the sensor device 200, and the position detector 300 in this embodiment. Figures 3 and 4 are block diagrams illustrating the hardware and functional block configurations of the animal appearance determining device 100, the sensor device 200, and the position detector 300 in this embodiment, respectively.

[0029] <<Sensor device 200, position detector 300>> First, the sensor device 200 and the position detector 300 in this embodiment will be described with reference to Fig. 4. The sensor device 200 is attached to an appropriate part of the body of the cow C, and includes a device that converts the movement of the cow C, which is included in the behavior of the cow C, into a physical quantity and outputs it. In this embodiment, the sensor device 200 is attached to a collar, for example, and attached to the neck of the cow C, but it may also be configured to be attached to another part of the body, such as the ear.

[0030] In this embodiment, the sensor device 200 functions as a terminal device that enables the animal appearance determining device 100 to acquire the position of the sensor device 200. Therefore, in this embodiment, it is sufficient for the sensor device 200 to have the function of a terminal device that transmits its own identification information. Here, the configuration of the sensor device 200 as such a terminal device will be described. As illustrated in FIG. 4, the sensor device 200 includes a processing unit 210 configured by a processor such as a CPU, and a communication unit 220. The processing unit 210 includes an identification information storage unit 211 that stores data that is identification information of the sensor device 200, and an identification information transmission unit 212 that transmits the data of the identification information to the outside. As the identification information, a serial number unique to the device that constitutes the sensor device 200 may be used.

[0031] The communication unit 220 has a function of receiving identification information data from the identification information transmission unit 212 of the processing unit 210 and wirelessly transmitting it using a predetermined communication method. In this embodiment, the communication unit 220 is configured as a communication module with a communication function that complies with Bluetooth (registered trademark). The digital data from the identification information transmission unit 212 that is transmitted from the communication unit 220 is transmitted as Bluetooth data and received by the animal appearance determining device 100. The Bluetooth data is also received by the position detector 300 installed in the cow pen CB. Note that the above communication method is an example and is not limiting.

[0032] The position detector 300 functions as a locator for utilizing the direction detection function included in Bluetooth 5.1, for example. As shown in FIG. 4, the position detector 300 may be configured with a processing unit 310, which may be configured with a processor or the like, an antenna array 320, and a communication unit 330. The processing unit 310 may include a radio wave receiving unit 311 and a reception angle calculation unit 312 as functional blocks. In Bluetooth 5.1, the arrival angle of a Bluetooth signal can be obtained from the reception angle (Angle of Arrival, AoA) of the radio wave. A Bluetooth 5.1 digital signal includes a monotonic sine wave for a certain period called a Constant Tone Extension (CTE) after normal data communication. The position detector 300 receives this CTE with the radio wave receiving unit 311 via the antenna array 320, which includes multiple antennas. The reception angle calculation unit 312 calculates the incident angle of the Bluetooth signal based on the difference in phase angle of the received wave at each antenna. As will be described later, the reception angle data calculated by the reception angle calculation unit 312 of the position detector 300 is transmitted to the animal appearance determining device 100 by the communication unit 330. Note that the position detection method for detecting the position of the sensor device 200 is not limited to the method using the Bluetooth function, and other available methods can be adopted, such as a position detection method using a satellite positioning system such as GPS, or a position identification method using radio waves from a Wi-Fi base station.

[0033] <<Animal Appearance Judgment Device 100>> Next, an animal appearance determining device 100 will be described as an embodiment of an information processing device according to the present invention. As shown in Fig. 3, the animal appearance determining device 100 is an information processing device such as a server computer or a personal computer, and includes a processing unit 110, a storage unit 120, an input / output unit 130, a data interface (data IF) unit 140, and a communication unit 150. The processing unit 110, the storage unit 120, the input / output unit 130, the data IF unit 140, and the communication unit 150 are connected to each other by a bus serving as an internal communication line (not shown).

[0034] The processing unit 110 is an arithmetic unit configured by a processor such as a CPU, and reads and executes various programs and data from the storage unit 120 (described later) to realize the functions of the animal appearance determining device 100. In this embodiment, the processing unit 110 executes data processing for each of the functional units: the image data receiving unit 111, the image data analyzing unit 112, the position data receiving unit 113, the position data analyzing unit 114, the individual identification unit 115, the appearance determining unit 116, the appearance information recording unit 117, and the appearance information output unit 118. The operation of each functional unit will be described later.

[0035] The storage unit 120 is a storage area for various programs and various data for causing the hardware group to function as the animal appearance determining device 100, and can be configured with ROM, RAM, flash memory, a solid-state drive (SSD), a hard disk (HDD), etc. Specifically, the storage unit 120 stores a program for causing the processing unit 110 to execute each function of this embodiment (a control program for the animal appearance determining device 100), various parameters, data used for individual identification and appearance determination, data related to the cow C to be processed, operation input data input from outside, and generated appearance information data, etc.

[0036] The input / output unit 130 is composed of various input devices such as a keyboard, mouse, touch panel, microphone, etc. that enable data input to the animal appearance judgment device 100 from outside, and output devices such as a monitor display, speaker, etc. for outputting and displaying appearance information data, etc.

[0037] The data IF unit 140 has a function of controlling data communication between the processing unit 110 , the storage unit 120 , the input / output unit 130 , and the communication unit 150 .

[0038] The communication unit 150 is a communication module that transmits and receives various data between the sensor device 200 and the position detector 300, and is configured as hardware such as a network interface card (NIC). The communication unit 150 can also be used to enable the animal appearance determining device 100 to communicate with the external communication network N illustrated in Fig. 1. Although omitted in this embodiment, the animal appearance determining device 100 may be provided with various control functions for the camera 400 installed in the cow pen CB.

[0039] Next, the functions of the animal appearance determining device 100 realized by the programs executed by the processing unit 110 will be described.

[0040] Image data receiving unit 111 has a function of receiving image data captured by camera 400 installed in cow pen CB. Camera 400 is typically a digital video camera equipped with an imaging element such as a CCD and having a required resolution, but is not limited to this and may also be configured to capture still images at predetermined time intervals.

[0041] The image data analysis unit 112 has the function of analyzing the captured image data of the cow pen CB received by the image data receiving unit 111 using image analysis data to determine whether a cow C is included in the image and, if so, to calculate the cow C's position within the cow pen CB based on its position within the image. FIG. 5 shows an example of an image of the cow pen CB captured by the camera 400. The image shown in FIG. 5 depicts three cows, C1 to C3. The image data analysis unit 112 uses a learning model that has been trained in advance using image data of various cows stored in the image analysis data storage unit 121 of the storage unit 120 to check whether a cow is included in each image, and frames the area in the image that has been determined to be a cow with rectangles (x1, y1), (x2, y2), (x3, y3), and (x4, y4), for example, as shown in FIG. 5. (x1, y1), ... indicate the coordinates of the rectangle vertices in the screen coordinate system within the image. The training data used for learning is annotation data that identifies the area in which the cow is captured in image data containing various images of cows. Image data determined to contain cow C is used as data for determining appearance, which will be described later. On the other hand, for an image determined to contain cow C, the image data analysis unit 112 calculates the position of cow C in the cow pen CB based on the area of ​​cow C in the image. The area occupied by cow C in the image, i.e., the position coordinates of the vertices of the rectangle in the screen coordinate system, are projected onto a coordinate system of real space (world coordinate system) in the cow pen CB based on the installation position coordinates of the camera 400, the angle indicating the imaging direction, for example, the angle (depression angle) that the imaging direction of the camera 400 makes with the horizontal plane and the angle with respect to the coordinate axes in the cow pen CB, and the angle of view of the camera 400, thereby making it possible to determine the position occupied by cow C in the image in the real space of the cow pen CB. Specifically, a transformation matrix for translating and rotating a rectangle indicating the position of cow C in the screen coordinate system to convert it into a position in the world coordinate system is prepared in advance for each camera 400. The position of cow C calculated based on this image data is used in the individual identification process for each cow C, which will be described later.

[0042] Position data receiving unit 113 has a function of receiving position data from position detector 300. As described above, the position data is, for example, the position coordinates of each position detector 300 within the cow pen CB, and radio wave reception angle data from sensor device 200 calculated by reception angle calculation unit 312 of each position detector 300. The position coordinate data of position detector 300 may be stored in advance in memory unit 120 as a parameter.

[0043] The position data analysis unit 114 calculates the position of the sensor device 200 within the cow pen CB using the position coordinate data and radio wave reception angle of the position detector 300 received and acquired by the position data receiving unit 113. When two sets of position detectors 300 are used as in Figure 1, the position of the sensor device 200, which is the source of the radio waves, can be found as the intersection of lines drawn from the positions of each position detector 300 according to the radio wave reception angle. Additional position detectors 300 can be installed depending on the need for precision in measuring the position of the sensor device 200. By matching the found position of the sensor device 200 with planar coordinates set in advance within the cow pen CB, it is possible to determine the location within the cow pen CB of the cow C to which the sensor device 200 is attached. The position of each sensor device 200 within the cow pen CB calculated by the position data analysis unit 114 is used in the individual identification process for each cow C, which will be described later. The position detection method of the sensor device 200 described above is an example, and other methods may be used, such as calculating position coordinates based on positioning satellite signals of the GPS (Global Positioning System) or using radio waves from a Wi-Fi base station.

[0044] The individual identification unit 115 has the function of identifying each individual cow C captured in the image by comparing the estimated position of the cow C in the stall CB calculated by the image data analysis unit 112 based on the position of the cow C in the image included in the image data with the position of each sensor device 200 in the stall CB calculated by the position data analysis unit 114 based on data from the position detector 300. As described above, the sensor device 200 attached to each cow C is assigned a sensor device ID, which is unique identification information, and this is received by the appearance determination device 100 via the position detector 300. Since the sensor device ID is linked to the individual ID, which is the individual identification code of each cow C, by comparing the position of the cow C estimated from the image data with the position of the sensor device 200, it is possible to determine whether the difference between the position of the cow C estimated from the image data and the position of the sensor device 200 is equal to or less than a predetermined threshold (whether the two positions are close enough to be included within a certain range within the pen CB), and associate the cow C in the image with the individual ID (this association will be described later with respect to the configuration of the processing target data storage unit). Figure 6 schematically shows the position of the cow C in the pen CB estimated from the image data compared with the position of the cow C calculated based on the radio waves received from the sensor device 200. In Figure 6, stars indicate the position of the cow C estimated from the image data, and circles indicate the position of the cow C calculated based on the radio waves received from the sensor device 200. The fact that the number of cows C based on the image data is smaller than the number of sensor devices 200 indicates that there are cows C in the pen CB that could not be detected from the images captured by the camera 400. 6 shows that the position of cow C (star) estimated from the image data is associated with three cows (individual IDs = 001 to 003) marked with diagonally shaded circles. In this way, the individual identification unit 115 makes it possible to identify which individual cow C is captured in the image. The location information identified for each cow C is recorded in the location information storage unit 123 shown in FIG. 9 as a record in which the recording date and time, the individual ID of the cow C, and the location within the pen CB are associated.

[0045] The appearance determination unit 116 has a function of quantitatively determining the appearance of the cow C based on an image of the cow C included in the image data captured by the camera 400. Quantitatively determining the appearance of the cow C can be used for health management, disease prevention, and the like for each cow C. FIG. 7 illustrates examples of appearance determination items determined in the appearance determination in this embodiment. In this embodiment, the appearance determination items for a calf are shown as an example, and four items, "ear position," "eye opening," "head position," and "back curvature," are each determined on a three-level scale, from an appearance score of 0, which is the best score, to an appearance score of 2. For each item, the higher the ear position, the better, the wider the eyes are open, the higher the head position, and the straighter the back without curvature. The determination items may be changed as appropriate depending on the subject of appearance determination, such as a dairy cow. The appearance determination unit 116 performs a determination process for image data including cow C framed in the image by the image data analysis unit 112, using appearance determination data stored in the appearance determination data storage unit 122. Figure 8 shows an example of the configuration of the appearance determination data storage unit 122. The appearance determination data illustrated in Figure 8 stores image data including cow C, the appearance type of cow C contained in images such as the face and side of the body, determination items such as eye opening, and appearance scores assigned to the determination items of the associated image data. For example, the appearance determination unit 116 can compare the image of the cow C to be determined with the appearance determination data, and determine the appearance score for the image to be determined based on the similarity of the determination items in the image. The method of appearance determination is not limited to this, but may be any of the following: - From images accumulated for a large number of cows C, quantitative indices are derived in advance based on each judgment item such as "ear position" and "eye opening," and appearance is judged based on these quantitative indices. For example, for "ear position," the judgment criteria are quantified so that an appearance score can be assigned based on the degree to which the ears droop from the horizontal line (the angle at which the ears droop). Using the data for appearance assessment, a trained model is used to learn the relationship between images of cow C's appearance and the appearance scores for each assessment item, and an appearance assessment is performed on a new image of cow C. It is possible to adopt the following.

[0046] The appearance information recording unit 117 has a function of recording the appearance judgment results for each cow C generated by the appearance judgment unit 116 as appearance information in chronological order. The appearance information is recorded in the appearance information storage unit 124 of the storage unit 120. Fig. 10 shows an example of the configuration of the appearance information storage unit 124. The appearance information storage unit 124 shown in Fig. 10 records the date and time when the appearance information was recorded, the individual ID of cow C, the appearance judgment items, and the appearance score in association with each other.

[0047] The appearance information output unit 118 has a function of extracting and outputting appearance information data from the appearance information storage unit 124 under predetermined conditions in response to input operations from the input / output unit 130, etc. For example, by extracting an appearance score corresponding to a specific appearance assessment item of a cow C having a specific individual ID and outputting it as time-series data, it is possible to use this data to detect changes in the health condition of the cow C, and signs of illness or specific events (weaning, parturition, etc.).

[0048] The processing target data storage unit 125 stores individual information about each cow C that is the subject of appearance determination. FIG. 11 shows an example of the configuration of the processing target data storage unit 125. The processing target data storage unit 125 shown in FIG. 11 stores the following items in association with an individual ID assigned as an identification code to each cow C that is the subject of appearance determination: a sensor device ID, which is unique identification information for the sensor device 200; a breed indicating the type of cow; sex; age; health condition; and medical history. The sensor device ID is attached to the sensor data transmitted from each sensor device 200 attached to the cow C to the animal appearance determination device 100, making it possible to identify which sensor device 200 acquired the received sensor data. The sensor device ID is also received by the position detector 300, so that the location of a specific sensor device 200 can be linked to the sensor device ID. The health condition may be recorded as a numerical value as shown in the figure, or may be specifically recorded in text. An appropriate description format may also be adopted for the medical history. The record items of the processing target data storage unit 125 are not limited to the example of FIG. 11 and can be determined as appropriate.

[0049] <Data processing by the animal appearance determination device 100> Next, a description will be given of data processing executed by the animal appearance determining device 100 in this embodiment. The animal appearance determining device 100 mainly executes a position information calculation process for detecting the position of an animal (a cow in this embodiment) captured in an image, and an appearance determination process for determining the appearance of the animal based on the image.

[0050] <<Location information calculation process>> 12 is a flowchart showing an example of the process flow of the position information calculation process executed by the animal appearance determining device 100 in this embodiment. The position information calculation process is started, for example, when the animal appearance determining device 100 is powered on and started up, and is executed at regular time intervals.

[0051] The image data receiving unit 111 of the animal appearance determining device 100 receives, from the camera 400, the captured image data captured by the camera 400 (step S11).

[0052] The image data analysis unit 112 analyzes the image data received from the camera 400 using the image analysis data stored in the image analysis data storage unit 121 (step S12), and extracts the image area of ​​cow C within the image (step S13).

[0053] The image data analysis unit 112 converts the area occupied by cow C in the image into a position in a spatial coordinate system using the relationship between the screen coordinate system, the position of the camera 400, the camera coordinate system based on the imaging direction, and the spatial coordinate system of the real world, and calculates the position of cow C in the stall CB (step S14).

[0054] Meanwhile, the position data analysis unit 114 calculates the position of each sensor device 200 as the position of the cow C from the angle of arrival of the radio waves received by the position data receiving unit 113 from the position detector 300, and compares this with the position of the cow C estimated based on the image data in step S14 (step S15). As a result, the position of the cow C estimated from the image data is verified with the position of the compared sensor device 200, i.e., the position of the cow C to which the sensor device 200 is attached, and individual identification of each cow C captured in the image is achieved. Note that the verification of the position of the cow C can be performed by first assuming that the first position information indicating the position of the cow C estimated based on the image data is the position of the cow C in the pen CB, and then checking whether the position of the corresponding sensor device 200 is located, in other words, whether the position coordinates of any sensor device 200 are detected within a certain range from the position coordinates of the cow C estimated based on the image data. Alternatively, the position of the sensor device 200 can first be taken as the position of cow C within the cow pen CB, and the position of cow C can be verified based on whether there is first position information indicating the position of cow C estimated based on the corresponding image data, in other words, whether the position coordinates of cow C estimated based on any image data are within a certain range from the position coordinates of the sensor device 200.

[0055] The position data analysis unit 114 records the position of the cow C identified in step S15 in the position information storage unit 123 (step S16).

[0056] The processing unit 110 determines whether a data processing end command has been received through the input / output unit 130 or the communication unit 150 (step S17). If it is determined that a data processing end command has not been received (step S17: NO), the processing unit 110 returns to step S11 and continues executing the position information calculation process. If it is determined that a data processing end command has been received (step S17: YES), the processing unit 110 ends the position information calculation process.

[0057] <<Appearance determination process>> 13 is a flowchart showing an example of the processing flow of the appearance determination process executed by the animal appearance determining device 100 in this embodiment. The appearance determination process is started, for example, when the animal appearance determining device 100 is powered on and started up, and is executed at regular time intervals.

[0058] The appearance determination process by the animal appearance determination device 100 is repeatedly executed for each sensor device 200 attached to the cow C while the animal appearance determination device 100 is operating (loop of step S21). The appearance determination unit 116 acquires images of the cow C extracted from the image data by the image data analysis unit 112 (step S22). The appearance determination unit 116 extracts images related to the appearance determination items from the acquired images of the cow C (step S23). Specifically, the appearance determination unit 116 checks whether the acquired images include the head, face, and side of the body of the cow C and whether the "head," "ears," "eyes," and "back" related to the appearance determination items are identifiable. This determination can be made using, for example, a learning model trained using a large amount of training data including the "head," "ears," "eyes," and "back" of the cow C. Note that when acquiring images of the cow C, it is preferable to capture images of all body parts related to the appearance determination items for each cow C evenly. From this perspective, it is preferable to install multiple cameras 400 in the stall CB where the cows C are kept and capture images of the inside of the stall CB from multiple different directions. Alternatively, it is also possible to configure the camera 400 to emit sound, light, etc. to draw the attention of the cows C to the camera 400 while capturing images.

[0059] The appearance determination unit 116 compares the extracted images related to the appearance determination items with the appearance determination data read from the appearance determination data storage unit 122 (step S24). Specifically, as described above, the appearance determination unit 116 checks the degree of similarity with the appearance determination data for the four items of "ear position," "eye opening," "head position," and "back curvature." Alternatively, it performs processing such as comparing the extracted images related to the appearance determination items with quantified indices derived in advance based on each determination item, or performing processing such as judging a new image of cow C using a trained model that has learned the relationship between the image of cow C's appearance and the appearance score of each determination item using the appearance determination data.

[0060] The appearance determination section 116 determines an appearance score for each determination item based on the comparison result of step S24 (step S25), and the appearance information recording section 117 records the score as appearance information in the appearance information storage section 124 (step S26).

[0061] By executing the above position information calculation process and appearance assessment process, appearance scores for each cow C regarding the appearance assessment items ``ear position,'' ``eye opening,'' ``head position,'' and ``back curvature'' are accumulated in chronological order.

[0062] <Use of Appearance Determination Results by Animal Appearance Determination Device 100> As described above, the animal appearance determining device 100 of this embodiment accumulates, in chronological order, the appearance scores and determination indices for each cow C related to the "ear position," "eye opening," "head position," and "back curvature" selected as appearance determination items. Therefore, for example, by analyzing the correlation between temporal changes in the appearance scores related to "ear position," "eye opening," "head position," and "back curvature" and events such as the cow C's health condition, disease onset, estrus, parturition, and weaning, and having the animal appearance determining device 100 learn the correlation, the animal appearance determining device 100 of this embodiment can be used to detect signs of events such as changes in the cow C's health condition, disease onset, estrus, parturition, and weaning. In this embodiment, images of the cows C in the stall CB are periodically captured, but each cow C is essentially in a natural state. If imaging is continued under such conditions, for example, if a certain cow C is in poor physical condition and moves little or does not change its posture much, it can be inferred that there is a tendency that the number of images that can be used as appearance assessment items is small for that particular cow C, or that the number of images for a particular appearance assessment item is small. In other words, it is thought that it is possible to predict changes in the health condition of cow C not only from the time-series change in the appearance score but also from the time-series change in the number of images for the appearance assessment items.

[0063] According to the embodiment described above, the following effects are achieved.

[0064] The animal appearance determination device 100 in one embodiment of the present invention is equipped with an image data analysis unit 112 that acquires an image including a cow C kept in a barn CB and acquires the position of the cow within the breeding area calculated based on the image of the cow C in the image as first position information, a position data analysis unit 114 that acquires identification information and second position information indicating the position of the cow C within the barn CB from a sensor device 200 installed on the cow C, and an individual identification unit 115 that, when it is determined that the difference between the first position information and the second position information is below a predetermined threshold, associates the first position information with the identification information related to the second position information and records them.

[0065] In this way, the position of the cow C in the stall CB can be identified with high accuracy based on the captured image of the cow C in the stall CB, thereby making it possible to identify the individual cow C in the captured image.

[0066] The animal appearance determination device 100 may be equipped with an appearance determination unit 116 that makes a determination on pre-set appearance determination items regarding the appearance of the animal based on the captured image of the animal, uses the determination result as appearance information of the animal, obtains the first location information calculated from the image of the animal that was the subject of the determination, and associates and records the identification information corresponding to the second location information associated with the first location information with the appearance information.

[0067] In this way, it is possible to record changes in the appearance of each identified cow C in chronological order.

[0068] The appearance determination unit 116 may determine the appearance of cow C using a trained model that has learned the relationship between the image information acquired about cow C and the appearance information that is the determination result for the pre-set appearance determination items.

[0069] In this way, the appearance of each cow C can be judged automatically.

[0070] The appearance determination unit 116 may determine the appearance of the cow C based on the image acquired of the cow C and criteria for determining features in the image of the cow C that are set in advance for the appearance determination items.

[0071] In this way, the appearance of each cow C can be judged automatically.

[0072] The position information to be compared with the position estimated based on the image of the cow C can be calculated based on the direction of arrival of the radio waves transmitted from the sensor device 200 provided on the cow C.

[0073] In this way, the position of the sensor device 200, that is, the cow C, can be detected by the radio waves used for data transmission from the sensor device 200.

[0074] The appearance information for each individual animal recorded in chronological order by the appearance determination unit may be used to predict at least one of the timing of illness, estrus, parturition, growth, and weaning of the animal.

[0075] In this way, events such as animal illness, estrus, parturition, growth, and weaning can be detected in a timely manner without being overlooked based on the animal's appearance.

[0076] The above-described series of processes can be executed by hardware or software. In other words, the functional configurations shown in FIGS. 2 and 3 are merely illustrative and are not particularly limited. That is, it is sufficient for the animal appearance-judging device 100 to have the functionality to execute the above-described series of processes as a whole, and the functional blocks used to realize these functions are not particularly limited to the examples shown in FIGS. 2 and 3. Furthermore, each functional block may be configured as a single piece of hardware, a single piece of software, or a combination thereof. The functional configuration in this embodiment is realized by a processor that executes arithmetic processing. Processors that can be used in this embodiment include those configured as single processing devices such as single processors, multiprocessors, and multicore processors, as well as those that combine these processing devices with processing circuits such as ASICs (Application Specific Integrated Circuits) and FPGAs (Field-Programmable Gate Arrays).

[0077] When a series of processes is executed by software, the programs that make up the software are installed into a computer or the like from a network or a recording medium. The computer may be a computer built into dedicated hardware. Alternatively, the computer may be a computer that can execute various functions by installing various programs, such as a general-purpose personal computer.

[0078] The recording medium containing such a program may be a removable medium such as a USB memory that is distributed separately from the device main body in order to provide the program to the user, or may be a recording medium that is provided to the user in a state where it is pre-installed in the device main body. Removable media may be, for example, a magnetic disk (including a floppy disk), an optical disk, or a magneto-optical disk. Optical disks may be, for example, CD-ROMs (Compact Disk-Read Only Memory), DVDs (Digital Versatile Disks), Blu-ray (registered trademark) Discs, etc. Magneto-optical disks may be, for example, MDs (Mini-Disks), etc. Furthermore, recording media that are provided to the user in a state where they are pre-installed in the device main body may be, for example, a ROM on which the program is recorded, or a hard disk included in the storage unit 120.

[0079] In this specification, the steps of describing a program to be recorded on a recording medium include not only processes that are performed chronologically in accordance with the order, but also processes that are not necessarily performed chronologically but are performed in parallel or individually.

[0080] Although several embodiments of the present invention have been described above, these embodiments are merely illustrative and do not limit the technical scope of the present invention. The present invention can take various other embodiments, and it is also possible to combine the configurations of the above embodiments and modifications. Furthermore, various modifications such as omissions and substitutions can be made without departing from the spirit of the present invention. These embodiments and modifications thereof are included within the scope and spirit of the invention described in this specification, etc., and are included in the invention described in the claims and their equivalents. [Explanation of symbols]

[0081] 1. Animal Appearance Judgment System 100 Animal appearance determination device 110 Processing section 115 Individual Identification Department 116 Appearance Judgment Department 120 Storage section 122 Appearance determination data storage unit 124 Appearance information storage unit 200 Sensor Device 300 Position detector 400 cameras

Claims

1. a first position information acquisition unit that acquires a captured image including an animal being kept in a predetermined breeding area, and acquires, as first position information, the position of the animal in the breeding area calculated by mapping an extracted area of ​​the animal in the captured image onto a coordinate system of real space using a transformation matrix that performs projective transformation on the position coordinates in a screen coordinate system of rectangular vertices that surround an area determined to be the animal in the captured image based on the installation position coordinates of a camera, the imaging direction, and the angle of view; a second location information acquisition unit that acquires, from a terminal device provided on the animal, identification information of the animal and second location information indicating the location of the animal within the breeding area detected by a sensor device that identifies the location using a wireless communication function; an individual identification unit that, when determining that the difference between the first location information and the second location information is equal to or less than a predetermined threshold, associates the first location information with the identification information related to the second location information and records them; An information processing device comprising:

2. 2. The information processing device of claim 1, further comprising an appearance determination unit that, based on a captured image of the animal, makes a determination on pre-set appearance determination items regarding the appearance of the animal, uses the determination result as appearance information of the animal, obtains the first location information calculated from the image of the animal that was the subject of the determination, and records the identification information associated with the first location information in association with the appearance information.

3. The information processing device according to claim 2, wherein the appearance determination unit determines the appearance of the animal using a trained model that has learned the relationship between the image acquired of the animal and appearance information that is the determination result for the appearance determination items that have been set in advance.

4. The information processing device according to claim 2 , wherein the appearance determination unit determines the appearance of the animal based on an image acquired of the animal and criteria for determining characteristics in the image of the animal that are set in advance for the appearance determination items.

5. The information processing device according to claim 1 , wherein the second position information is calculated based on an arrival direction of radio waves transmitted from a terminal device provided on the animal.

6. An information processing device as claimed in any one of claims 2 to 5, wherein the appearance information for each individual animal recorded in chronological order by the appearance determination unit is used to predict at least one of the timing of illness, estrus, parturition, growth, and weaning of the animal.

7. The information processing device a process of acquiring a captured image including an animal being kept in a predetermined breeding area, and acquiring, as first position information, the position of the animal in the breeding area calculated by mapping an extracted area of ​​the animal in the captured image onto a coordinate system of real space using a transformation matrix that performs projective transformation on the position coordinates in a screen coordinate system of vertices of a rectangle that surrounds an area determined to be the animal in the captured image based on the installation position coordinates of a camera, the imaging direction, and the angle of view; a process of acquiring, from a terminal device provided on the animal, identification information of the animal and second location information indicating the location of the animal within the breeding area detected by a sensor device that identifies the location using a wireless communication function; a process of recording the first location information and the identification information related to the second location information in association with each other when it is determined that the difference between the first location information and the second location information is equal to or less than a predetermined threshold; An information processing method that performs the above.

8. In the information processing device, a process of acquiring a captured image including an animal being kept in a predetermined breeding area, and acquiring, as first position information, the position of the animal in the breeding area calculated by mapping an extracted area of ​​the animal in the captured image onto a coordinate system of real space using a transformation matrix that performs projective transformation on the position coordinates in a screen coordinate system of vertices of a rectangle that surrounds an area determined to be the animal in the captured image based on the installation position coordinates of a camera, the imaging direction, and the angle of view; a process of acquiring, from a terminal device provided on the animal, identification information of the animal and second location information indicating the location of the animal within the breeding area detected by a sensor device that identifies the location using a wireless communication function; a process of recording the first location information and the identification information related to the second location information in association with each other when it is determined that the difference between the first location information and the second location information is equal to or less than a predetermined threshold; A program that executes the following.

Images