Memory control device, learning device, livestock identification device, memory control system, memory control method, and program

Abstract

To appropriately identify livestock. [Solution] The memory control device 200 comprises an input unit 201, a setting unit 202, an extraction unit 203, and a memory control unit 204. The input unit 201 receives imaging information of livestock captured from the front. The setting unit 202 sets a predetermined area within the imaging area indicated by the imaging information in which the head of the livestock can move. The extraction unit 203 extracts the face image of the livestock within the predetermined area based on the imaging information. The memory control unit 204 classifies and stores the face images according to the identification information corresponding to the predetermined area.

Description

Technical Field

[0001] The present invention relates to a memory control device, a learning device, a livestock identification device, a memory control system, a memory control method, and a program.

Background Art

[0002] Conventionally, in dairy farms, it is necessary to manage each livestock individually. In this management, if each livestock is manually identified one by one, it is time-consuming. Especially in larger-scale dairy farms, this labor becomes more prominent. Therefore, various technologies for identifying livestock have been disclosed. For example, a method of attaching a communication device to a livestock to determine the position and group of the livestock has been disclosed (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the technology described in Patent Document 1, since a communication device is attached to a part of the livestock's body, wearing it for a long time may cause stress to the cows and there is a risk of deterioration in the quality of the livestock. In addition, since repeaters for communicating with the communication devices attached to the livestock need to be installed throughout the farm, it becomes a great deal of labor and burden for dairy farmers.

[0005] The present invention has been made in view of such circumstances, and an object thereof is to provide a technology capable of easily identifying livestock.

Means for Solving the Problems

[0006] To solve the above-mentioned problems, a memory control device according to one aspect of the present invention includes an input means for inputting imaging information of livestock captured from the front, a setting means for setting a predetermined region within the imaging region indicated by the imaging information in which the head of the livestock can move, an extraction means for extracting a face image of the livestock within the predetermined region based on the imaging information, and a memory control means for classifying and storing the face image according to the identification information corresponding to the predetermined region. The memory control means inputs the face image extracted by the extraction means into a discrimination model that outputs the identification information of the known livestock and its accuracy when the face image of a known livestock is input, classifies the face image according to the discriminated identification information if the accuracy is greater than or equal to a predetermined value, and classifies the face image according to the identification information corresponding to the predetermined region if the accuracy is less than a predetermined value.

[0007] To solve the above-mentioned problems, a learning device according to one aspect of the present invention includes a learning processing means that uses a learning dataset in which the face images stored by the memory control device described above are input samples and the one-hot vector indicating the class to which the face image belongs among a plurality of classes is output as an output sample, to create a discrimination model that outputs a vector indicating the posterior probability of the class to which the face image belongs, corresponding to the input face image. This learning device is characterized by the following features.

[0008] Furthermore, another embodiment of the present invention is a livestock identification device comprising: an imaging information input means for inputting imaging information obtained by imaging livestock from the front; a face image extraction means for extracting a face image of livestock based on the imaging information; an identification means for identifying identification information by inputting the face image extracted by the face image extraction means into a discrimination model learned by the learning device described above; and an output means for outputting the result identified by the identification means.

[0009] Furthermore, another embodiment of the present invention is a memory control system comprising an imaging device for imaging livestock from the front and a memory control device, wherein the memory control device comprises an input means for inputting imaging information from the imaging device, a setting means for setting a predetermined region within the imaging region indicated by the imaging information in which the head of the livestock can move, an extraction means for extracting a face image of the livestock within the predetermined region based on the imaging information, and a memory control means for classifying and storing the face image for each identification information corresponding to the predetermined region, wherein the memory control means inputs the face image extracted by the extraction means into a discrimination model that outputs the identification information of the known livestock and its accuracy when a face image of a known livestock is input, classifies the face image to the discriminated identification information if the accuracy is greater than or equal to a predetermined value, and classifies the face image to the identification information corresponding to the predetermined region if the accuracy is less than a predetermined value.

[0010] Furthermore, another aspect of the present invention is a memory control method in which a computer used in a memory control device performs a process including: an input step of inputting imaging information of livestock captured from the front; a setting step of setting a predetermined region in the imaging region indicated by the imaging information in which the head of the livestock can move; an extraction step of extracting a face image of the livestock within the predetermined region based on the imaging information; and a memory control step of classifying and storing the face image for each identification information corresponding to the predetermined region. The memory control method is characterized in that, when a face image of a known livestock is input, the face image extracted in the extraction step is input to a discrimination model that outputs the identification information of the known livestock and its accuracy; if the accuracy is greater than or equal to a predetermined value, the face image is classified to the discriminated identification information; and if the accuracy is less than a predetermined value, the face image is classified to the identification information corresponding to the predetermined region.

[0011] Furthermore, another aspect of the present invention is a program in which a computer used in a memory control device functions as an input means for inputting imaging information of livestock captured from the front, a setting means for setting a predetermined region within the imaging region indicated by the imaging information in which the head of the livestock can move, an extraction means for extracting a face image of the livestock within the predetermined region based on the imaging information, and a memory control means for classifying and storing the face image for each identification information corresponding to the predetermined region, wherein the memory control means inputs the face image extracted by the extraction means into a discrimination model that outputs the identification information of the known livestock and its accuracy when the face image of a known livestock is input, classifies the face image to the discriminated identification information if the accuracy is greater than or equal to a predetermined value, and classifies the face image to the identification information corresponding to the predetermined region if the accuracy is less than a predetermined value. [Effects of the Invention]

[0012] According to the present invention, livestock can be easily identified. [Brief explanation of the drawing]

[0013] [Figure 1] This is an explanatory diagram showing an example of livestock identification system 1. [Figure 2] This is a block diagram showing an example of the functional configuration of the livestock identification system 1 according to this embodiment. [Figure 3] This is an explanatory diagram illustrating an example of how to extract and store facial images of cows (Bu). [Figure 4] This is an explanatory diagram showing the configuration of the learning processing unit 221 and the classification model MO. [Figure 5] This is an explanatory diagram showing an example of the hardware configuration of the livestock identification device 120. [Figure 6] This flowchart shows an example of the memory control process for facial images 320 performed by the livestock identification device 120. [Figure 7] This flowchart shows an example of the cattle identification process performed by the livestock identification device 120. [Figure 8]It is an explanatory diagram showing an example of the display of the cow ID displayed by the livestock identification device 120.

Embodiments for Carrying Out the Invention

[0014] (Embodiment) FIG. 1 is an explanatory diagram showing an example of the livestock identification system 1. The livestock identification system 1 is an example of a memory control system. The livestock identification system 1 includes an imaging device 100 and a livestock identification device 120. Dozens to hundreds of livestock are raised on a dairy farm. In the present embodiment, the livestock is a dairy cow (cow Bu). However, the livestock is not limited to cow Bu, and for example, it may be a beef cow, or an animal other than a cow such as a pig, a goat, or a chicken.

[0015] On the dairy farm, there is a feeding area 10 where cow Bu eats feed. A fence 20 is provided in the feeding area 10. The fence 20 is provided with a gap through which the head of cow Bu can pass, and cow Bu eats feed by sticking its head out through the gap in the fence 20. The fence 20 may be of a type that can lock the gap so as to restrict the movement of the head when cow Bu passes its head through the gap (equipped with a head lock), or may be of a type that allows cow Bu to freely insert and remove its head without locking the gap. Cow Bu, for example, moves to the feeding area 10 at a fixed time every day to eat feed. After cow Bu finishes eating the feed, it leaves the feeding area 10 and is sequentially replaced with another cow Bu that has not eaten the feed.

[0016] In the feeding area 10, an imaging device 100 for imaging a plurality of cow Bu from the front is fixedly arranged. That is, the imaging device 100 is installed so that the surface on which the gaps of the fence 20 are arranged is imaged within the imaging range. The imaging device 100 images the face of cow Bu from the front. The imaging device 100 images the face of each cow Bu. The number of cow Bu to be imaged (the number of gaps in the fence 20) is shown as 4 in the drawing, but it is not limited to this, and for example, it may be 5 or more (for example, 10), or it may be 3 or less.

[0017] A plurality of imaging devices 100 are arranged at the feeding place 10. Each of the plurality of imaging devices 100 is assigned a number (camera number) for identification. Note that the imaging device 100 is not limited to being arranged in plurality, and may be arranged singly. For the imaging device 100, for example, a CCD (charge coupled device) camera or a CMOS (Complementary Metal Oxide Semiconductor) camera can be used. Also, a 3D camera may be used for the imaging device 100.

[0018] The imaging device 100, for example, captures images at 25 frames per second. The imaging device 100 outputs imaging information indicating the captured imaging images to the livestock identification device 120. The output mode of the imaging information may be an output mode by wired or wireless transmission, or an output mode of storing it in a storage medium (for example, a USB (Universal Serial Bus) memory, etc.).

[0019] The livestock identification device 120 is a computer device such as a personal computer, a notebook computer, a tablet device, a smartphone, etc. The livestock identification device 120 inputs the imaging information output by the imaging device 100. The input mode of the imaging information may be an input mode by wired or wireless reception, or an input mode of reading from a storage medium. The livestock identification device 120 stores the face images of each cow Bu from the input imaging information for each cow Bu, and learns to be able to identify each cow Bu using the stored face images.

[0020] (Regarding the functional configuration of the livestock identification system 1 and the extraction of the face images of cows Bu) FIG. 2 is a block diagram showing an example of the functional configuration of the livestock identification system 1 according to the present embodiment. The livestock identification system 1 is an example of a storage control system. FIG. 3 is an explanatory diagram showing an example of the outline of extracting and storing the face images of cows Bu. In the description of FIGS. 2 and 3, the functional configuration shown in FIG. 2 will be described while appropriately referring to FIG. 3.

[0021] As shown in Figure 2, the livestock identification system 1 comprises an imaging device 100 and a livestock identification device 120. The livestock identification device 120 comprises a memory control device 200 and a learning device 220. The memory control device 200 comprises an input unit 201, a setting unit 202, an extraction unit 203, a memory control unit 204, a detection unit 205, and a storage unit 210. The learning device 220 comprises a learning processing unit 221. The livestock identification device 120 comprises an imaging information input unit 231, a face image extraction unit 232, an identification unit 233, and an output unit 234.

[0022] The input unit 201 receives imaging information from the imaging device 100. The imaging information is image information of cows Bu (Bu1 to Bu4) captured from the front. The imaging information 300 includes both video and still images. As shown in Figure 3, the imaging information 300 includes multiple frame images 301. The imaging area 302 of each frame image 301 indicates that multiple cows Bu are present. Note that in some cases, only one cow Bu may be present in the imaging area 302.

[0023] The setting unit 202 sets the regions of interest 310 (310a to 310d) from the imaging area 302 indicated by the imaging information 300. The regions of interest 310 are examples of predetermined areas. The regions of interest 310 are areas provided corresponding to the cow Bu or the gaps in the fence 20, and each indicates an area where the cow Bu's head can be located. Each region of interest 310 is assigned corresponding identification information.

[0024] The extraction unit 203 extracts face images 320 (320a to 320d) of cow Bu within the region of interest 310 based on the imaging information 300. The extraction unit 203 extracts face images 320 using an object detection function based on image recognition. The extraction unit 203 extracts face images 320 each time a frame image 301 is input to the input unit 201.

[0025] The extracted face images 320 are normalized to, for example, 300 x 300 pixels. If the extracted face images 320 are 240 pixels wide and 300 pixels high, a plain image (e.g., black) is added to the area in the horizontal direction that is less than 300 pixels to make it 300 x 300 pixels. In other words, if one of the dimensions of the extracted face images 320 is 300 pixels and the other is less than 300 pixels, a plain image (e.g., black) is added to the area that is less than 300 pixels to make it 300 x 300 pixels.

[0026] Furthermore, if neither the width nor height of the extracted 320 face images is 300 pixels, they are enlarged or reduced proportionally so that the larger dimension becomes 300 pixels. Then, a blank image is added to the area of ​​the other dimension that is less than 300 pixels. This makes it possible to create 300x300 pixel images even if neither the width nor height of the 320 face images is 300 pixels.

[0027] The memory control unit 204 classifies and stores the face images 320 in the memory unit 210 according to the identification information assigned to each region of interest 310. Specifically, each time a face image 320 is extracted by the extraction unit 203, the memory control unit 204 classifies and stores the face image 320 in folders 330 (330a to 330d) corresponding to each region of interest 310. The memory unit 210 stores a predetermined number of face images 320 for each cow Bu. The predetermined number should be a number that allows the learning processing unit described later to create a trained model (for example, around 300 images).

[0028] Folder 330 is assigned a folder ID (Identity) as identification information. The folder ID is associated with the cow ID. The cow ID may be the same as the folder ID, or it may be a different ID assigned by the operator of the livestock identification device 120 (for example, a dairy farmer), or it may be the name or number of cow Bu. The memory control unit 204 stores a face image 320 in the memory unit 210 for each cow ID (folder ID). In the following explanation, the cow ID and folder ID will be described as the same ID, and specifically, the folder ID will be described as the cow ID.

[0029] (Regarding the calculation of the overlap rate) Here, cow Bu eating feed at the feeding area 10 tends to stay in the same spot and continue eating for a predetermined time (e.g., 5 minutes) after starting to eat, but when the feed in front of it runs out, it may eat the feed in front of the neighboring cow Bu. In such cases, or when cow Bu's hunger is satisfied and it does not stay in the same spot, the cow Bu's face image 320 may fall outside the region of interest 310. Face images 320 that fall outside the region of interest 310 are undesirable to use as training images, for example, because they may only show half of the face. Therefore, in this embodiment, the face image 320 is stored based on the overlap rate between the face image 320 and the region of interest 310.

[0030] Specifically, the memory control unit 204 stores the face image 320 for each cow ID corresponding to the region of interest 310, based on the degree of overlap (overlap rate) between the region indicated by the face image 320 and the region of interest 310. The overlap rate can be expressed as the ratio of the region indicated by the face image 320 within the region of interest 310. For example, a face image 320 with an overlap rate below a threshold may overlap with the region of interest 310 set for an adjacent cow Bu.

[0031] For example, suppose that the face of cow Bu1 in Figure 3 moves in front of the adjacent cow Bu2, and as a result, the face image 320a of cow Bu1 is extracted from regions of interest 310a and 310b. In this case, the face image 320a of cow Bu1 extracted from region of interest 310a will have a low overlap rate in region of interest 310a. Similarly, the face image 320a of cow Bu1 extracted from region of interest 310b will also have a low overlap rate in region of interest 310b.

[0032] Therefore, in this embodiment, if the duplication rate is less than a threshold, the memory control unit 204 prevents the face image 320 from being stored in the folder 330. On the other hand, if the duplication rate is equal to or greater than a threshold, the memory control unit 204 stores the face image 320 in the folder 330 corresponding to the cow ID.

[0033] (Regarding the setting of area of ​​interest 310) As shown in Figure 3, cow Bu is eating near the center of the region of interest 310. However, cow Bu does not necessarily eat at the center of the region of interest 310 in the imaging region 302. Therefore, in this embodiment, the region of interest 310 is set according to the range of movement of cow Bu's head.

[0034] Specifically, the detection unit 205 detects the area of ​​the cow Bu's face within the imaging area 302 for each of the multiple frame images included in the imaging information 300 input to the input unit 201. The detection unit 205 detects the face using an object detection function. The face area is, for example, an area equivalent to the area shown in the face image 320. However, the face area may be smaller or larger than the area shown in the face image 320.

[0035] The setting unit 202 sets the region of interest 310 based on the facial regions in multiple frame images detected by the detection unit 205. Specifically, the setting unit 202 sets the region of interest 310 to a predetermined region that includes the centers of multiple facial regions relating to the same individual detected by the detection unit 205. Individual identification is performed, for example, by considering multiple facial regions in the same frame image as separate individuals, and considering the facial regions that are closest in distance to each other in consecutive frame images as belonging to the same individual. Alternatively, individual identification may be performed simply by identifying the relative position of appearance (e.g., second from the left).

[0036] The area of ​​interest 310 may have a predetermined size and shape (rectangle). In this case, the setting unit 202 sets the area of ​​interest 310 by determining the placement position of the rectangle based on the position of the face area detected by the detection unit 205. Furthermore, when setting the areas of interest 310, it is desirable to leave a gap between adjacent areas of interest 310. This is to make it difficult for each cow Bu to enter the area of ​​interest 310 of a neighboring cow Bu.

[0037] If the rectangle is initially positioned based on the position of the face area to set the region of interest 310, there is a risk that adjacent regions of interest 310 may overlap if the distance between adjacent cows Bu happens to be close. For this reason, the average position of the face area of ​​each cow Bu may be detected during the first few tens of seconds, and the region of interest 310 may be set based on that position. Alternatively, the setting unit 202 may shrink the rectangle or shorten its horizontal length to set the regions of interest 310 so that adjacent regions of interest 310 do not overlap.

[0038] (Regarding the creation of pre-trained models) The learning processing unit 221 creates a trained model (discrimination model) for identifying cow IDs. Specifically, the learning processing unit 221 creates a trained model using a training dataset. The training dataset is a training dataset in which face images 320 stored by the memory control unit 204 are used as input samples, and one-hot vectors indicating the class to which the face image 320 belongs among multiple classes are used as output samples. The trained model is a trained model that outputs a vector indicating the posterior probability of the class to which the input face image 320 belongs, corresponding to the face image 320. The number of classes is the number of folders in the memory unit 210.

[0039] Here, we will explain the learning processing unit 221 in detail using Figure 4. Figure 4 is an explanatory diagram showing the configuration of the learning processing unit 221 and the classification model MO. Figure 4(A) shows the configuration of the learning processing unit 221. The learning processing unit 221 comprises a classification model storage unit 11, a dataset acquisition unit 12, a learning unit 13, and an output unit 14.

[0040] The classification model storage unit 11 stores the classification model M0, which is constructed using a convolutional neural network. As shown in Figure 4(B), the classification model M0 comprises an input unit M01, a feature calculation unit M02, a classification unit M03, and an output unit M04. The input unit M01 outputs the input face image 320 as a vector to the feature calculation unit M02. The input unit M01 forms the input layer of the neural network.

[0041] The feature calculation unit M02 and the classification unit M03 form the intermediate layers of the neural network. The output unit M04 forms the output layer of the neural network. The feature calculation unit M02 converts the vector input from the input unit M01 into a low-dimensional feature vector and outputs it to the classification unit M03. The classification unit M03 converts the feature vector input from the feature calculation unit M02 into a P-dimensional vector representing the posterior probability of the cow ID represented by that feature vector. P is the number of cow IDs to be estimated.

[0042] The dataset acquisition unit 12 acquires a training dataset that associates the input sample, which is a face image 320, with the output sample, which is a cow ID label. The cow ID label is represented by a P-dimensional one-hot vector, where P is the number of cow IDs in the dataset.

[0043] The learning unit 13 uses the training dataset acquired by the dataset acquisition unit 12 to train the parameters of the classification model M0 so that when a face image 320 is input, it outputs a P-dimensional vector representing the posterior probability of the cow ID corresponding to the face image 320.

[0044] Next, the learning method will be explained in detail. The dataset acquisition unit 12 acquires a pre-prepared dataset. Then, the learning unit 13 uses the acquired dataset to train the parameters of the classification model M0 stored in the classification model storage unit 11. At this time, the learning unit 13 updates each parameter using gradient descent to minimize the loss function using the calculation results of the classification model M0.

[0045] The learning unit 13 updates the parameters of the feature extraction unit M02 and the classification unit M03 of the classification model M0. For example, the loss function represents the cross-entropy error between the output values ​​of the classification model M0 and the output samples of the dataset.

[0046] The learning unit 13 terminates the learning process when the evaluation value of the loss function falls below a predetermined threshold, or when the learning process has been repeated a predetermined number of times. When the learning process by the learning unit 13 is completed, the output unit 14 outputs the classification model M0 (trained model) learned by the learning unit 13 to the discrimination unit 233.

[0047] (Regarding the assignment of identification information) At this stage, when training is not yet complete, there may be a mix of cows (Bu) that have not been trained by the trained model and cows that have been trained. In this case, when a trained cow (Bu) finishes eating and is replaced by an untrained cow (Bu), it is necessary to ensure that the face images of these two cows (Bu) are not stored in the same folder. To explain this in more detail, the identification unit 233 identifies the cow ID by inputting the face image 320 extracted by the extraction unit 203 into the trained model. If the identification unit 233 does not identify the cow ID (indicating a low probability) and there is a folder 330 to which the cow ID is assigned for that cow (Bu), the memory control unit 204 stores the extracted face image 320 in the folder 330 corresponding to that cow ID.

[0048] Furthermore, a cow Bu that has just entered the feeding area 10 will not have a cow ID (folder ID) assigned to it. Therefore, it is necessary to assign a cow ID to the cow Bu that has just entered the feeding area 10 and create a folder 330. To explain this in more detail, if there is a cow Bu that has just entered the feeding area 10, specifically if the cow ID is not identified by the identification unit 233 (i.e., not learned) and there is no folder 330 to which the cow ID has been assigned for that cow Bu, the memory control unit 204 will assign a new cow ID corresponding to the region of interest 310 and generate a folder 330.

[0049] Furthermore, in the case of a cow Bu trained with a pre-trained model, the cow ID will be identified by the identification unit 233. In this case, the memory control unit 204 stores the face image 320 extracted by the extraction unit 203 in the folder 330 corresponding to the cow ID. In this case, the learning processing unit 221 can improve the learning accuracy by updating the pre-trained model using the face image 320 stored in the folder 330. However, the memory control unit 204 may choose not to store the extracted face image 320 in the memory unit 210.

[0050] In this manner, the memory control unit 204 inputs the face image 320 extracted by the extraction unit 203 into a discrimination model (for example, a trained model created by the learning processing unit 221) that outputs the cow ID of a known livestock and its accuracy when the face image 320 of a known livestock is input. If the accuracy is greater than or equal to a predetermined value, the memory control unit 204 classifies the face image 320 to the determined cow ID. On the other hand, if the accuracy is less than the predetermined value, the memory control unit 204 classifies the face image 320 to the cow ID corresponding to the region of interest 310. Note that the discrimination model is not limited to a trained model created by the learning processing unit 221, but may also be a trained model created by another device.

[0051] (Regarding the identification of cattle) The identification unit 233 can identify the cow ID of cow Bu eating feed in the feeding area 10 in real time by using a trained model created by the learning processing unit 221. Specifically, the imaging information input unit 231 receives imaging information from the imaging device 100, which is an image of cow Bu taken from the front. The face image extraction unit 232 extracts a face image 320 of cow Bu based on the imaging information. The identification unit 233 identifies the cow ID by inputting the real-time face image 320 extracted by the face image extraction unit 232 into the trained model trained by the learning device 220. More specifically, the identification unit 233 derives an estimated value for each cow ID from the face image 320 input to the trained model and selects (identifies) the cow ID with the highest estimated value.

[0052] The output unit 234 outputs a cow ID indicating the result identified by the identification unit 233. The output unit 234 outputs the cow ID identified by the identification unit 233 and the face image 320. The output unit 234 associates the cow ID with the face image 320 (video) of cow Bu and displays it on the display 405 (see Figure 8). Note that the output mode by the output unit 234 is not limited to display output mode, but may also include output mode by sound or output mode by transmission to an external device. When transmitting to an external device, the external device may display the cow ID and face image 320.

[0053] The identification unit 233 identifies the cow ID each time the face image extraction unit 232 extracts a face image 320. The output unit 234 outputs the cow ID each time the identification unit 233 identifies it. The output unit 234 also outputs the cow ID that was identified most frequently among the cow IDs identified by the identification unit 233 as the final result. Specifically, the output unit 234 outputs the cow ID that was identified most frequently within a predetermined time as the final result. The predetermined time can be any time the cow Bu stays in place and eats (for example, within 5 minutes). In this embodiment, the predetermined time is, for example, 3 minutes.

[0054] In this embodiment, the identification unit 233 identifies the cow ID of the face image 320 using a trained model created by the learning device 220, but this is not limited to this. For example, the identification unit 233 may identify the cow ID using comparison data that compares the feature quantities of the face images 320 corresponding to each folder 330. Specifically, a database (not shown) is used to store the cow ID and the comparison feature quantities extracted from the face image 320 for each cow Bu, in association with each other. The identification unit 233 may calculate feature quantities from the real-time face images 320 extracted by the face image extraction unit 232 and identify (identify) the cow ID corresponding to the feature quantity with the highest similarity to the feature quantity.

[0055] In this embodiment, the memory control device 200 and the learning device 220 are included in the livestock identification device 120, but this is not limited to that. For example, the livestock identification device 120, the memory control device 200, and the learning device 220 may each be separate devices. Alternatively, the livestock identification device 120 and the learning device 220 may be separate devices, with the memory control device 200 included in the learning device 220, or the memory control device 200 may be included in the livestock identification device 120.

[0056] Furthermore, the memory unit 210 is not limited to being provided in the memory control device 200, but may also be provided in an external device such as an external server. Also, the input unit 201 and the imaging information input unit 231 are not limited to being separate functional units, but may be a single functional unit. Similarly, the extraction unit 203 and the face image extraction unit 232 are not limited to being separate functional units, but may be a single functional unit.

[0057] (Hardware configuration of livestock identification device 120) Figure 5 is an explanatory diagram showing an example of the hardware configuration of the livestock identification device 120. In Figure 5, the livestock identification device 120 includes a CPU 401 (Central Processing Unit), memory 402, operation unit 403, microphone 404, display 405, speaker 406, and communication I / F (interface). These can communicate with each other via a bus.

[0058] The CPU 401 is a central processing unit that controls the operation of the livestock identification device 120 by reading and executing various programs stored in the memory 402. The various programs include a memory control program, a learning program, and a livestock identification program, as described in this embodiment.

[0059] Memory 402 is a storage unit 210 and includes, for example, ROM (Read Only Memory), RAM (Random Access Memory), hard disk, SSD (Solid State Drive), etc. ROM is read-only memory and stores various types of information used by the CPU 401, including programs. RAM is read-and-write memory and stores various types of information. For example, RAM stores information obtained from external sources and information generated during processing. Memory 402 also includes removable storage media such as USB memory and optical discs.

[0060] The control unit 403 is an input unit for operation, such as a touch panel display, keyboard, or mouse. The microphone 404 receives sound input, such as voice. The CPU 401 recognizes the voice input to the microphone 404 by executing a voice recognition program.

[0061] The display 405 is a touch panel display or a monitor. The speaker 406 outputs sound. The communication I / F 407 is an interface for sending and receiving information with other devices. The livestock identification device 120 may also be equipped with a scanner to read information, a printer to print information onto a medium, and other such devices.

[0062] Furthermore, the parts 201-205, 221, and 231-234 shown in Figure 2, and the parts 11-14 shown in Figure 4, are implemented by the CPU 401. In other words, the CPU 401 implements the functions of the parts 11-14, 201-205, 221, and 231-234 by executing various programs stored in memory 402.

[0063] (An example of the memory control processing of facial images 320 performed by the livestock identification device 120) Figure 6 is a flowchart illustrating an example of the memory control process for face images 320 performed by the livestock identification device 120. In Figure 6, the livestock identification device 120 determines whether or not to start learning when the operation unit 403 (see Figure 4) receives a predetermined operation indicating the start of learning (step S601). The livestock identification device 120 waits until learning begins (step S601: NO), and when learning begins (step S601: YES), it inputs imaging information 300 (one frame image 301) captured by the imaging device 100 (step S602).

[0064] The livestock identification device 120 then determines whether or not the region of interest 310 has been set (step S603). If the region of interest 310 has been set (step S603: YES), the livestock identification device 120 proceeds to step S605. If the region of interest 310 has not been set (step S603: NO), the livestock identification device 120 detects the face region of cow Bu and, based on the detection result, sets multiple regions of interest 310 (310a to 310d) within the imaging region 302 (step S604).

[0065] The livestock identification device 120 then identifies one of the multiple regions of interest 310 (step S605). Next, the livestock identification device 120 extracts a face image 320 from the identified region of interest 310 (step S606). Then, the livestock identification device 120 calculates the overlap rate based on the region of the extracted face image 320 and the region of interest 310 (step S607).

[0066] Next, the livestock identification device 120 determines whether the overlap rate is above a threshold (step S608). If the overlap rate is below the threshold (step S608: NO), the livestock identification device 120 proceeds to step S614. If the overlap rate is above the threshold (step S608: YES), the livestock identification device 120 inputs the extracted face images 320 into the trained model (step S609).

[0067] The livestock identification device 120 determines whether or not a cow ID has been derived as a result of inputting the face image 320 into the trained model (step S610). Specifically, the livestock identification device 120 determines that a cow ID corresponding to an element has been derived if the probability indicated by the element with the highest value in the output vector of the trained model is above a threshold. On the other hand, the livestock identification device 120 determines that a cow ID has not been derived if the probability indicated by the element with the highest value in the output vector of the trained model is below a threshold. If a cow ID has been derived (step S610: YES), that is, if a cow ID has been identified, the livestock identification device 120 proceeds to step S613. If a cow ID has been derived, the cow ID may be displayed on the display 405. On the other hand, if a cow ID has not been derived (step S610: NO), the livestock identification device 120 determines whether or not a cow ID corresponding to a identified region of interest 310 has been assigned (step S611).

[0068] If a cattle ID corresponding to the said area of ​​interest 310 is assigned (step S611: YES), the livestock identification device 120 proceeds to step S613. On the other hand, if a cattle ID corresponding to the said area of ​​interest 310 is not assigned (step S611: NO), the livestock identification device 120 assigns a new cattle ID corresponding to the said area of ​​interest 310, that is, generates a folder 330 with a new folder ID (step S612).

[0069] Next, the livestock identification device 120 stores the face image 320 in a folder 330 corresponding to the one area of ​​interest 310 (step S613). Then, the livestock identification device 120 determines whether or not all areas of interest 310 have been identified in the target frame image 301 (step S614). If not all areas of interest 310 have been identified (step S614: NO), the livestock identification device 120 returns to step S605 and extracts and stores face images 320 for the other areas of interest 310 that have not been identified.

[0070] On the other hand, if all regions of interest 310 have been identified (step S614: YES), the livestock identification device 120 determines whether a predetermined time (e.g., 5 minutes) has elapsed in which a predetermined number of face images 320 capable of creating a trained model have been obtained (step S615). If the predetermined time has not elapsed (step S615: NO), the livestock identification device 120 returns to step S602 and inputs the next frame image 301.

[0071] On the other hand, if a predetermined time has elapsed (step S615: YES), the livestock identification device 120 creates a trained model to determine the cow ID using the face images 320 stored in each folder 330 (step S616). If a trained model has already been created, the livestock identification device 120 overwrites the trained model. The predetermined time that triggers the start of training, as shown in step S615, can be arbitrarily changed. Furthermore, this trigger is not limited to a predetermined time; it may also be a predetermined number of frames or the number of face images 320 stored.

[0072] Once the creation of the trained model in step S616 is complete, the livestock identification device 120 unassociates the region of interest 310 corresponding to the target cow ID of the trained model with the said cow ID (step S617). Next, the livestock identification device 120 determines whether or not to terminate the learning process when the operation unit 403 (see Figure 4) receives a predetermined operation indicating the end of learning (step S618).

[0073] If learning is not terminated (step S618: NO), the livestock identification device 120 returns to step S602 and inputs the next frame image 301. On the other hand, if learning is terminated (step S618: YES), the livestock identification device 120 terminates the series of processes.

[0074] Through the process described above, the livestock identification device 120 can store the face images 320 extracted from the region of interest 310 in folders 330, one for each cow ID (one for each region of interest 310). The livestock identification device 120 can also create a trained model using the face images 320 in each folder 330 as input samples. Furthermore, by repeatedly executing the process described above, the number of cows Bu that can be identified by the trained model can be sequentially increased.

[0075] (An example of the cattle identification process performed by the livestock identification device 120) Figure 7 is a flowchart illustrating an example of the cattle identification process performed by the livestock identification device 120. In Figure 7, the livestock identification device 120 determines whether or not to start identifying the cattle ID when it receives a predetermined operation from the operation unit 403 (see Figure 4) indicating the start of identification (step S701). The livestock identification device 120 waits until it is time to start identifying the cattle ID (step S701: NO), and when it is time to start identifying the cattle ID (step S701: YES), it inputs the imaging information 300 (one frame image 301) captured by the imaging device 100 (step S702).

[0076] The livestock identification device 120 then extracts face images 320 (step S703). Next, the livestock identification device 120 identifies one of the extracted face images 320 (step S704). Then, the livestock identification device 120 inputs the identified face image 320 into the trained model (step S705).

[0077] The livestock identification device 120 derives an estimated value for each cow ID based on the input face image 320 and selects the cow ID with the highest estimated value (step S706). In step S706, the estimated value for each cow ID is expressed as a percentage. Specifically, it is expressed as follows: there is a 95% probability that the cow is ID "1", a 2% probability that it is ID "2", a 1% probability that it is ID "3", and so on, and the cow ID with the highest probability is selected.

[0078] Next, the livestock identification device 120 displays the selected cow ID on the display 405 (step S707). Then, the livestock identification device 120 determines whether or not all of the face images 320 extracted in step S703 have been input into the trained model (step S708). If all of the face images 320 have not been input into the trained model (step S708: NO), the livestock identification device 120 returns to step S704, identifies one face image 320 that has not yet been identified, and inputs it into the trained model and selects (identifies) the cow ID.

[0079] On the other hand, once all face images 320 have been input into the trained model (step S708: YES), the livestock identification device 120 determines whether a predetermined time (e.g., 3 minutes) has elapsed (step S709). If the predetermined time has not elapsed (step S709: NO), the livestock identification device 120 returns to step S702 and inputs the next frame image. On the other hand, if the predetermined time has elapsed (step S709: YES), the livestock identification device 120 displays the cow ID that was selected most frequently within the predetermined time as the final result on the display 405 (step S710).

[0080] Next, the livestock identification device 120 determines whether to terminate the identification of the cattle ID when the operation unit 403 (see Figure 4) receives a predetermined operation indicating the end of identification (step S711). If the identification of the cattle ID is not terminated (step S711: NO), the livestock identification device 120 returns to step S702 and inputs the next frame image 301. On the other hand, if the identification of the cattle ID is terminated (step S711: YES), the livestock identification device 120 terminates the series of processes.

[0081] Through the process described above, the livestock identification device 120 can extract a face image 320 from the imaging area 302 and identify the cow ID based on the extracted face image 320.

[0082] (Example of the cattle ID displayed by the livestock identification device 120) Figure 8 is an explanatory diagram showing an example of the display of the cattle ID shown by the livestock identification device 120. Figure 8(A) shows the screen before a predetermined time has elapsed since the start of cattle ID identification. Figure 8(B) shows the screen after the predetermined time has elapsed. In Figures 8(A) and (B), the display 405 shows the search screen 800. The search screen 800 includes a camera number input area 810, a search start button 820, an image capture display area 830, and a cattle ID display area 840.

[0083] The camera number input area 810 is an area for receiving the input of a camera number. The camera number is a number that indicates one of the multiple imaging devices 100 located in the feeding area 10. The search start button 820 is a button that accepts the start of a search for a cow ID. The cow ID to be searched is the cow ID of cow Bu that is imaged by the imaging device 100 corresponding to the camera number entered in the camera number input area 810.

[0084] The imaging display area 830 is an area that displays images captured by the imaging device 100. The imaging device 100 is the imaging device 100 corresponding to the camera number entered in the camera number input area 810. The imaging display area 830 also includes an identification result area 831 (831a to 831d). The identification result area 831 is an area corresponding to the face image 320 (320a to 320d) within the imaging display area 830 and shows the identification result of the cow ID. The identification result area 831 is displayed either above or below the face in the face image 320 within the imaging display area 830, depending on the position of the face.

[0085] For example, in the face images 320a and 320c of cows with IDs "1" and "3", the position of cow Bu's face is relatively high in the imaging display area 830. As a result, there is space below the face images 320a and 320c in the imaging display area 830, so the identification result areas 831a and 831c are displayed below the face image 320.

[0086] On the other hand, the position of cow Bu's face in face images 320b and 320d for cow IDs "2" and "4" is relatively low in the imaging display area 830. As a result, there is space above face images 320b and 320d in the imaging display area 830, so the identification result areas 831a and 831c are displayed above face image 320.

[0087] Thus, the identification result area 831 is switched between being displayed at the top or bottom depending on the position of the face in the face image 320. Furthermore, even with the same face image 320, different identification results may be displayed from frame to frame.

[0088] The cattle ID display area 840 is an area that displays the identification results of cattle IDs in a list. Specifically, the cattle ID display area 840 is an area that displays the cattle IDs of cattle Bu displayed in the imaging display area 830 in a list format. Furthermore, the cattle IDs displayed in the cattle ID display area 840 correspond to the identification results in the identification result area 831. For this reason, different cattle IDs may be displayed in the cattle ID display area 840 from frame to frame.

[0089] In Figure 8(A), the identification result area 831 is displayed in a first display mode. In Figure 8(B), the identification result area 831 is displayed in a second display mode, which is different from the first display mode. The second display mode is a display mode that shows the final result of the identification. Once displayed in the second display mode, no different identification results will be displayed for the same face image 320.

[0090] Furthermore, the display mode of the cattle ID display area 840 will also be a display mode that shows the final result. Note that in Figure 8(B), both the identification result area 831 and the cattle ID display area 840 are shown in a display mode that shows the final result, but it is sufficient to have at least one of them in a display mode that shows the final result.

[0091] Furthermore, in Figure 8, when the operator enters a different camera number in the camera number input area 810 and selects the search start button 820, the livestock identification device 120 displays the image captured by the other imaging device 100 in the image display area 830 and searches for the cow ID based on the image. After a predetermined time has elapsed, the livestock identification device 120 displays the cow ID of cow Bu, captured by the other imaging device 100, as the final result.

[0092] This allows the operator to easily determine which of the cows Bu, as captured by the imaging device 100, have eaten or have not eaten.

[0093] As described above, in this embodiment, the livestock identification system 1 (memory control device 200) extracts facial images 320 of cows Bu within the region of interest 310 based on imaging information captured from the front of the cows Bu, and classifies and stores the facial images 320 according to the cow ID corresponding to the region of interest 310. This makes it possible to easily and accurately obtain facial images 320 of cows Bu eating feed in the feeding area 10. Therefore, it is possible to identify cows Bu based on their facial images 320. Consequently, it becomes possible to easily identify cows Bu with a simple configuration without having to attach a communication device to part of the cow's body or install relays throughout the farm. This reduces the burden on dairy farmers in identifying cows Bu.

[0094] Furthermore, in this embodiment, the livestock identification system 1 (memory control device 200) stores the face image 320 for each cow ID corresponding to the region of interest 310, based on the degree of overlap between the region indicated by the face image 320 and the region of interest 310. This prevents the storage of face images 320 in which the area showing the cow Bu's face is small, i.e., low-accuracy face images 320 that are outside the region of interest 310. For example, low-accuracy face images 320 can be avoided in creating a trained model. Consequently, the accuracy of cow ID identification can be improved.

[0095] Furthermore, in this embodiment, the livestock identification system 1 (memory control device 200) detects the area of ​​the cow Bu's face within the imaging area 302 based on the imaging information 300, and sets a region of interest 310 based on the detected face area. This makes it possible to set a suitable region of interest 310 according to the position of the cow Bu. In other words, regardless of where the cow Bu eats its feed within the imaging area 302, a suitable face image 320 can be obtained.

[0096] Furthermore, in this embodiment, the livestock identification system 1 (memory control device 200) inputs the facial image extracted by the extraction unit 203 into the discrimination model, and if the accuracy is above a predetermined value, it classifies the facial image 320 to the identified cow ID, and if the accuracy is below the predetermined value, it classifies the facial image 320 to the cow ID corresponding to the region of interest 310. As a result, even for cows Bu that have just entered the feeding area 10 (cows Bu that have not been assigned a cow ID), a folder 330 with a cow ID can be created and the facial image 320 can be appropriately classified.

[0097] Furthermore, in this embodiment, the livestock identification system 1 (learning device 220) uses a training dataset in which a face image 320 stored for each cow ID is used as an input sample, and a one-hot vector indicating the class to which the face image 320 belongs among multiple classes is used as an output sample. The system is configured to create a trained model (discrimination model) that outputs a vector indicating the posterior probability of the class to which the face image 320 belongs, corresponding to the input face image 320. As a result, a trained model can be automatically created in a few minutes using the face image 320 of cow Bu eating feed at the feeding area 10. Therefore, a trained model can be created easily and quickly, and cow IDs can be identified with high accuracy.

[0098] Furthermore, in this embodiment, the livestock identification system 1 (memory control device 200) uses a pre-trained model created by the learning device 220 as its discrimination model. This allows for the creation of a folder 330 with a cow ID and the creation of a pre-trained model even for cows Bu (cows Bu that have not been assigned a cow ID) that have just entered the feeding area 10. Therefore, pre-trained models can be created easily and quickly for all cows Bu on the dairy farm.

[0099] Furthermore, in this embodiment, the livestock identification system 1 (livestock identification device 120) identifies the cow ID by inputting the face image 320 into a trained model learned by the learning device 220, and outputs the identification result. This makes it possible to easily and accurately identify cows Bu eating feed in the feeding area 10. Therefore, it becomes possible to suitably identify cows Bu with a simple configuration.

[0100] Furthermore, in this embodiment, the livestock identification system 1 (livestock identification device 120) identifies and outputs a cow ID each time it extracts a face image 320 from a predetermined number of consecutive frames of imaging information 300. As a result, the identification result can be displayed on the display 405 in real time each time imaging information 300 is input. Therefore, the identification result can be quickly presented to the operator.

[0101] Furthermore, in this embodiment, the livestock identification system 1 (livestock identification device 120) outputs the cow ID that was identified most frequently as the final result. As a result, although there is a possibility of variation in the identification results depending on the orientation of the cow Bu's face, even if such variation occurs, the cow ID that was identified most frequently within a predetermined time can be used as the final result. Therefore, cow IDs can be identified with higher accuracy.

[0102] (Modified examples of the embodiment) Modifications of the embodiments are described below. Note that in the following modifications, explanations of the contents described in the embodiments above will be omitted as appropriate. Furthermore, the following modifications and the embodiments described above can be combined.

[0103] (modified version) In the embodiment described above, the region of interest 310 was configured to be set based on the region of the cow Bu's face within the imaging region 302 (see Figure 3). In Modification 1, in addition to or instead of this configuration, the region of interest 310 is configured in which the imaging region 302 is divided into pre-defined regions.

[0104] In the modified example, the setting unit 202 sets pre-divided regions of interest 310 within the imaging area 302. Specifically, it is assumed that the area within the imaging area 302 where the cows Bu are located is predetermined. The setting unit 202 pre-sets regions of interest 310 in the area where the cows Bu are assumed to be located. Taking Figure 3 as an example, four regions of interest 310 (310a to 310d) are pre-set within the imaging area 302. A gap is provided between adjacent regions of interest 310. Although the figure assumes that four cows Bu are captured in the imaging area 302, if, for example, eight cows Bu are expected to be captured in the imaging area 302, eight regions of interest 310 should be pre-set.

[0105] According to the modified livestock identification system 1 (memory control device 200), even if adjacent cows Bu are close together at the start of imaging, a face image 320 can be obtained for each predetermined region of interest 310. Face images 320 can be obtained favorably in this way as well.

[0106] While several embodiments of the present invention have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These embodiments can be carried out in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims and their equivalents.

[0107] Furthermore, the program for realizing the livestock identification system 1, livestock identification device 120, memory control device 200, and learning device 220 described above may be recorded on a computer-readable recording medium, and the program may be loaded into a computer system and executed. Here, "computer system" includes hardware such as the OS and peripheral devices. "Computer-readable recording medium" refers to portable media such as flexible disks, magneto-optical disks, ROMs, CD-ROMs, and storage devices such as hard disks built into a computer system. Moreover, "computer-readable recording medium" also includes volatile memory (RAM) inside a computer system that acts as a server or client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line, which retains the program for a certain period of time. Furthermore, the above program may be transmitted from the computer system that stores the program in a storage device, etc., to another computer system via a transmission medium or by transmission waves in the transmission medium. Here, "transmission medium" for transmitting the program refers to a medium that has the function of transmitting information, such as a network such as the Internet or a communication line such as a telephone line. Furthermore, the above program may be for realizing only a part of the functions described above. Furthermore, the aforementioned functions may be implemented in combination with programs already recorded in the computer system, such as so-called differential files (differential programs). [Explanation of Symbols]

[0108] 1…Livestock identification system, 100…Imaging device, 120…Livestock identification device, 200…Memory control device, 201…Input unit, 202…Setting unit, 203…Extraction unit, 204…Memory control unit, 205…Detection unit, 210…Storage unit, 220…Learning device, 221…Learning processing unit, 231…Imaging information input unit, 232…Face image extraction unit, 233…Identification unit, 234…Output unit, 401…CPU, 402…Memory, 403…Operation unit, 404…Microphone, 405…Display, 406…Speaker, 407…Communication I / F

Claims

1. An input means for inputting imaging information obtained by imaging livestock from the front, A setting means for setting a predetermined region within the imaging area indicated by the aforementioned imaging information in which the head of livestock can move, Based on the aforementioned imaging information, an extraction means for extracting facial images of livestock within the predetermined area, A storage control means for classifying and storing the face images according to the identification information corresponding to the predetermined area, Equipped with, The memory control means inputs the face image extracted by the extraction means into a discrimination model that outputs identification information and accuracy of the known livestock when the face image of the known livestock is input, and classifies the face image into the discrimination information if the accuracy is greater than or equal to a predetermined value, and classifies the face image into the identification information corresponding to the predetermined region if the accuracy is less than the predetermined value. A memory control device characterized by the following features.

2. The memory control device according to claim 1 uses the face image stored in it as an input sample and a training dataset using a one-hot vector indicating the class to which the face image belongs as an output sample, to create a discriminant model that outputs a vector indicating the posterior probability of the class to which the face image belongs, corresponding to the input face image. A learning device characterized by the following features.

3. The imaging information input means for inputting the aforementioned imaging information, A face image extraction means for extracting the face image based on the aforementioned imaging information, An identification means for identifying identification information by inputting the face image extracted by the face image extraction means into a discrimination model learned by the learning device described in claim 2, An output means that outputs the result identified by the identification means, A livestock identification device characterized by being equipped with the following features.

4. The aforementioned imaging information is imaging information of a predetermined number of consecutive frames, The identification means identifies identification information each time the face image is extracted by the face image extraction means. The livestock identification device according to feature 3.

5. The output means outputs the identification information that is identified most frequently among the identification information identified by the identification means as the final result. The livestock identification device according to feature 4.

6. In a memory control system comprising an imaging device for imaging livestock from the front and a memory control device, The memory control device is An input means for inputting imaging information from the aforementioned imaging device, A setting means for setting a predetermined region within the imaging area indicated by the aforementioned imaging information in which the head of livestock can move, Based on the aforementioned imaging information, an extraction means for extracting facial images of livestock within the predetermined area, A storage control means for classifying and storing the face images according to the identification information corresponding to the predetermined area, Equipped with, The memory control means inputs the face image extracted by the extraction means into a discrimination model that outputs identification information and accuracy of the known livestock when the face image of the known livestock is input, and classifies the face image into the discrimination information if the accuracy is greater than or equal to a predetermined value, and classifies the face image into the identification information corresponding to the predetermined region if the accuracy is less than the predetermined value. A memory control system characterized by the following:

7. The computer used in the memory control device The input process involves inputting imaging information obtained by photographing livestock from the front, and A setting step of setting a predetermined region within the imaging area indicated by the aforementioned imaging information in which the head of the livestock can move, Based on the imaging information, an extraction step is performed to extract facial images of livestock within the predetermined area. A storage control step that classifies and stores the face image for each identification information corresponding to the predetermined area, Execute the process that includes, In the memory control step, when a facial image of a known livestock is input, the facial image extracted in the extraction step is input to a discrimination model that outputs identification information of the known livestock and its accuracy, and if the accuracy is greater than or equal to a predetermined value, the facial image is classified to the discriminated identification information, and if the accuracy is less than the predetermined value, the facial image is classified to the identification information corresponding to the predetermined region. A memory control method characterized by the following:

8. The computer used in the memory control device An input means for inputting imaging information obtained by imaging livestock from the front. Setting means for setting a predetermined area within the imaging area indicated by the aforementioned imaging information in which the head of livestock can move, Extraction means for extracting facial images of livestock within a predetermined area based on the aforementioned imaging information, A storage control means for classifying and storing the facial images according to the identification information corresponding to the predetermined area, To make it function as, The memory control means inputs the face image extracted by the extraction means into a discrimination model that outputs identification information and accuracy of the known livestock when the face image of the known livestock is input, and classifies the face image into the discrimination information if the accuracy is greater than or equal to a predetermined value, and classifies the face image into the identification information corresponding to the predetermined region if the accuracy is less than the predetermined value. A program characterized by the following features.

Images