A method for identifying individual animals, and a computer program for doing so.

Abstract

This technology provides a method for individual identification using features other than nose prints. [Solution] The disclosed method includes (a) acquiring a p-th type target image relating to the back of the animal to be identified; (b) obtaining a p-th type embedding vector using a p-th deep distance learning model; (c) calculating the p-th type distance between the p-th type registration embedding vector and the p-th type embedding vector for each registered individual; and (d) determining which of the registered individuals the animal to be identified corresponds to using the p-th type distance. Step (d) includes (d1) obtaining an integrated determination distance for each registered individual; (d2) determining that the registered individual is an individual of the animal to be identified if the minimum value of the integrated determination distance is smaller than a preset integrated threshold; and (d3) determining that the animal to be identified is an unregistered individual if it satisfies an unregistered condition, including that the minimum value of the integrated determination distance is larger than the integrated threshold.

Description

Technical Field

[0001] The present disclosure relates to a method for individual identification of animals and a computer program.

Background Art

[0002] Patent Document 1 discloses a technique for collating nose print images of cattle. In this conventional technique, a nose print image is extracted from a face image of a cattle, a feature vector of the nose print image is obtained using a neural network for nose print image classification, and the nose print image is collated by calculating a similarity between the feature vector and a known feature vector.

Prior Art Literature

Patent Literature

[0003]

Patent Document 1

Summary of the Invention

Problem to be Solved by the Invention

[0004] However, the conventional technique requires a high-resolution face image that can identify the fine structure of the nose print, and there has been a problem that it is difficult to acquire such a face image. For example, it is difficult to stop the movement of a cattle for photographing when the cattle is walking continuously. Such a problem is not limited to cattle, but is a problem common to individual identification of other animals such as pigs. Therefore, there is a demand for a technique capable of performing individual identification using features other than nose prints.

Means for Solving the Problem

[0005] A first embodiment of this disclosure provides a method for individual identification of an animal to be identified. This method includes: (a) acquiring N1 p-th type target images of the back of the animal to be identified using a p-th type image sensor, where p is an ordinal number from 1 to 2 and N1 is an integer of 1 or more; (b) obtaining N1 p-th type embedding vectors from the N1 p-th type target images using a p-th deep distance learning model; (c) calculating N1 × N2 p-th type distances between the N1 p-th type embedding vectors and the N2 p-th type registration embedding vectors using registration data that includes N2 p-th type registration embedding vectors pre-generated for each of the n registered individuals, where n and N2 are integers of 2 or more; and (d) determining which of the n registered individuals the animal to be identified corresponds to, using the N1 × N2 p-th type distances for each of the n registered individuals. The step (d) includes (d1) determining an integrated determination distance for each of the n registered individuals by integrating N1 × N2 first-type distances and N1 × N2 second-type distances; (d2) determining that the registered individual corresponding to the minimum value of the integrated determination distance is an individual of the target animal if the minimum value of the integrated determination distance is smaller than a preset integration threshold; and (d3) determining that the target animal is an unregistered individual that does not correspond to any of the n registered individuals if it satisfies an unregistered condition including that the minimum value of the integrated determination distance is larger than the integration threshold.

[0006] According to a second embodiment of the present disclosure, a computer program is provided that causes a processor to perform a process for individual identification of an animal to be identified. The computer program causes the processor to perform the following: (a) when p is an ordinal number from 1 to 2 and N1 is an integer of 1 or more, a process to acquire N1 p-th type target images with respect to the back of the animal to be identified using a p-th type image sensor; (b) a process to obtain N1 p-th type embedding vectors from the N1 p-th type target images using a p-th deep distance learning model; (c) when n and N2 are integers of 2 or more, a process to calculate N1 × N2 p-th type distances between the N1 p-th type embedding vectors and the N2 p-th type registration embedding vectors using registration data that includes N2 p-th type registration embedding vectors that have been generated in advance for each of the n registered individuals; and (d) a process to determine which of the n registered individuals the animal to be identified corresponds to using the N1 × N2 p-th type distances for each of the n registered individuals. The process (d) includes (d1) a process of determining an integrated determination distance for each of the n registered individuals by integrating N1 × N2 first type distances and N1 × N2 second type distances; (d2) a process of determining that the registered individual corresponding to the minimum value of the integrated determination distance is an individual of the target animal if the minimum value of the integrated determination distance is smaller than a preset integration threshold; and (d3) a process of determining that the target animal is an unregistered individual that does not fall under any of the n registered individuals if it satisfies the unregistered condition, which includes the minimum value of the integrated determination distance being larger than the integration threshold. [Brief explanation of the drawing]

[0007] [Figure 1] A block diagram showing the configuration of the individual identification system. [Figure 2] A flowchart illustrating the steps for training a deep metric learning model and creating registration data. [Figure 3] A flowchart showing the detailed steps for step S20. [Figure 4] This diagram illustrates the process of generating the target image in step S20. [Figure 5] An explanatory diagram showing an example of registered data. [Figure 6] A flowchart illustrating the procedure for individual identification processing in the first embodiment. [Figure 7] An explanatory diagram showing an example of an embedding vector calculated during individual identification processing. [Figure 8] An explanatory diagram showing an example of the distance between an embedding vector and a registered embedding vector. [Figure 9] A flowchart showing the detailed procedure for step S140 in the first embodiment. [Figure 10] An explanatory diagram showing examples of various detection distances. [Figure 11] A flowchart showing the procedure for registration and renewal processing. [Figure 12] A flowchart showing the detailed procedure for step S140 in the second embodiment. [Figure 13] A flowchart showing the detailed procedure for step S140 in the third embodiment. [Modes for carrying out the invention]

[0008] A. First Embodiment: Figure 1 is an explanatory diagram showing the configuration of the individual identification system in the first embodiment. This individual identification system comprises an information processing device 300 and a plurality of image sensors 400(p). In this embodiment, the target of individual identification is a cow (CW). However, other animals such as pigs, dogs, or cats may be used as the target of processing instead of cows.

[0009] The image sensor 400(p) is a camera that captures images of the cow CW that are the target of individual identification processing. Preferably, each image sensor 400(p) is installed to capture the back of the cow CW from above. The image sensor 400(p) may be a video camera that captures video, or a still image camera that captures still images. In addition, one or more of the various sensors exemplified below can be used as the image sensor 400(p). (1) Depth sensor By using depth images captured by a depth sensor, it is possible to identify individual cows based on the contours of their backs. (2) RGB sensor (color image sensor) Using color images captured by an RGB sensor, it is possible to identify individual cows based on the black and white markings on their backs. However, if there is dirt on the cow's back, the color image will change, so using only an RGB sensor for individual identification may lead to misidentification. On the other hand, depth images captured by a depth sensor are not affected by dirt on the cow's back, so they have the advantage of being less likely to cause misidentification due to dirt. (3) Spectroscopic sensor By using spectral images captured with a spectroscopic sensor, it is possible to identify individual cows based on the black and white patterns on their backs, which are the wavelength-specific markings. (4) Thermosensor By using thermal images captured with a thermal sensor, it is possible to identify individual cows based on the infrared intensity distribution on their backs. The infrared intensity distribution on the cow's back reflects how the degree of infrared scattering changes depending on the length of the hair, and how the degree of infrared diffusion changes depending on the distance due to unevenness.

[0010] Multiple image sensors 400(p) can be combinations of different image sensors or combinations of the same type. When using combinations of the same type of image sensors, it is preferable to set at least one attribute of each image sensor, such as the installation angle or field of view, to be different from each other. It is preferable that the relative positions of each sensor coordinate system of the multiple image sensors 400(p) are known, and that the coordinate transformation matrices for any two sensor coordinate systems are known.

[0011] The number P of image sensors 400(p) is an integer of 2 or greater. In the first embodiment, P=2, and two image sensors 400, a depth sensor and an RGB sensor, are used to capture a video of the back of a cow CW. It is preferable that the depth sensor and the RGB sensor have substantially the same imaging area and are configured to be capable of capturing images at the same imaging timing. For example, one RGBD sensor including a depth sensor and an RGB sensor can be used as the image sensor 400(p).

[0012] The information processing apparatus 300 performs individual identification of a cow CW using images relating to the back of the cow CW captured by the image sensor 400(p). The information processing apparatus 300 includes a processor 310, a memory 320, an interface circuit 330, and an input device 340 and a display device 350 connected to the interface circuit 330. The image sensor 400(p) is also connected to the interface circuit 330. The processor 310 not only has a function of executing the processing described in detail below, but also has a function of displaying, on the display device 350, data obtained by the processing and data generated in the process of the processing. The information processing apparatus 300 can be implemented by a computer such as a personal computer.

[0013] The processor 310 has the functions of a target image acquisition unit 510, a learning unit 520, and an individual identification unit 530. The target image acquisition unit 510 acquires a plurality of target images from a back image captured by an image sensor 400(p). The learning unit 520 performs distance learning on P pieces of deep distance learning models 620(p). The individual identification unit 530 performs individual identification of a cow CW using the trained deep distance learning model 620(p). The functions of each of these units are respectively realized by the processor 310 executing a computer program stored in the memory 320. However, some of these functions may be implemented by hardware circuits. The term "processor" in the present disclosure includes such hardware circuits. Furthermore, the one or more processors that execute various processes may be processors included in one or more remote computers connected via a network.

[0014] As the distance learned by the deep distance learning model 620(p), similarity may be used. A high similarity corresponds to a small distance. That is, "a small distance" is equivalent to "high similarity". In the following description, the term "distance" is used as a term that includes "similarity".

[0015] The memory 320 stores an object recognition model 610, P deep distance learning models 620(p), and P registered data 630(p). The object recognition model 610 is a machine learning model that receives an image of a cow CW as input and outputs a plurality of keypoints, which are geometric feature points of the cow CW, as output. In the present embodiment, it is assumed that the object recognition model 610 has been trained. The object recognition model 610 can also be referred to as a "feature recognition model".

[0016] The deep distance learning model 620(p) is a machine learning model that receives an image of a cow CW as input and outputs an embedding vector. p is an ordinal number from 1 to P representing the order of the deep distance learning models 620(p). The embedding vector is also referred to as a "feature vector". The deep distance learning model 620(p) can be configured using, for example, FaceNet.

[0017] In the following explanation, the sign followed by (p) indicates that it corresponds to the ordinal number p of the deep metric learning model 620. Similarly, the prefix "p-th kind" also indicates that it corresponds to the ordinal number p of the deep metric learning model 620.

[0018] The registration data 630(p) is a database in which, for each of several registered individuals, the individual ID and multiple p-th type embedding vectors obtained using the p-th deep distance learning model 620(p) are registered. The individual ID is, for example, the individual identification number displayed on the ear tag of a cow CW. The p-th type embedding vector registered in the p-th registration data 630(p) is called the "p-th type registration embedding vector". In addition to the p-th type registration embedding vector, the registration data 630(p) may also include images of each registered individual.

[0019] Figure 2 is a flowchart showing the procedure for training the deep distance learning model 620(p) and creating the registration data 630(p). In the following explanation, we assume that n is an integer greater than or equal to 2, and that n target cows are registered in the registration data 630(p). In this embodiment, the n target cows are of the same breed.

[0020] In step S10, the target image acquisition unit 510 uses two image sensors 400(p) to capture multiple dorsal images of each of the n target cows to be registered and the k target cows to be trained. Here, n and k are integers of 2 or greater. This capture can be performed, for example, as each cow sequentially passes below the image sensor 400(p). If a video is captured, multiple frame images are selected from the video as dorsal images. This selection can be performed, for example, by automatically selecting frame images that include the rear portion of the target cow to be registered using an annotation tool. Alternatively, the operator may make the selection manually.

[0021] In step S20, the target image acquisition unit 510 acquires multiple target images of the backs of n registered target cows and k training target cows from the back images captured by the p-th image sensor 400(p). "Training target cows" refers to cows used to create training data for the deep distance learning model 620(p). In deep distance learning, from the viewpoint of robustness, it is desirable to train the neural network using training data different from the registered data 630(p). It is also desirable to create training data that contains more data than the registered data 630(p). Therefore, it is preferable that k be set to a number greater than n. However, some of the training target cows may be registered cows. Also, n=k may be used, and the same cow may be used as both the training target cows and the registered target cows. The number of target images acquired from each cow may be set to a constant value N(p), or it may be set to a different number for each individual. Here, N(p) is an integer of 1 or more defined for the ordinal number p, but it is preferable that it be 2 or more. Furthermore, N(p) may be a constant value independent of the ordinal number p.

[0022] Figure 3 is a flowchart showing the detailed procedure for step S20, and Figure 4 is an explanatory diagram illustrating the process. In Figure 4, a depth sensor is used as the image sensor 400(p). Part of the step numbers are included in Figure 4.

[0023] The dorsal image BG shown in Figure 4 includes the posterior portion of the cattle coww, including the lumbar horns and the base of the tail. Since individual cattle cowws have distinctive features in the shape of their posterior portion, it is preferable that the dorsal image BG includes the posterior portion of the cattle coww, including the lumbar horns and the base of the tail.

[0024] In step S21, the target image acquisition unit 510 selects one cow to be processed from n registered cows and k training cows. In step S22, the target image acquisition unit 510 selects one back image BG to be processed from multiple back images BG. In step S23, the target image acquisition unit 510 creates a feature detection image CG by performing a grayscale conversion process on the back image BG. This grayscale conversion process extracts a grayscale range from the total grayscale of the back image BG that makes it easy to detect the contour of the cow, and converts it into an image with a predetermined number of grayscales. For example, if the back image BG, which is a depth image, has 2400 grayscales, the feature detection image CG may be created by extracting 512 intermediate grayscales and compressing them to 256 grayscales. However, step S23 is optional.

[0025] In step S24, the target image acquisition unit 510 uses the object recognition model 610 to detect multiple keypoints from the feature detection image CG. In the example in Figure 4, two keypoints KP1 and KP2 are detected at the hip horns of the cow CW, and one keypoint KP3 is detected at the base of the tail. These keypoints KP1 to KP3 are feature points present on the outline of the cow CW. The object recognition model 610 is a machine learning model that takes the feature detection image CG as input and outputs multiple keypoints KP1 to KP3. However, the object recognition model 610 may be configured to recognize keypoints indicating other locations. The object recognition model 610 can be configured using, for example, a deep learning model such as DeepLabCut or ResNet. The object recognition model 610 may also be configured using other machine learning models.

[0026] In step S25, the target image acquisition unit 510 determines whether a predetermined number of keypoints KP1 to KP3 have been detected. If the predetermined number of keypoints KP1 to KP3 have not been detected, the process returns to step S22, a new back image BG is selected as the processing target, and the processing from step S23 onwards is executed again. If the predetermined number of keypoints KP1 to KP3 have been detected, the process proceeds to step S26.

[0027] In step S26, the target image acquisition unit 510 extracts the target image TG from the back image BG based on a plurality of key points KP1 to KP3. The target image TG is an image that includes the characteristic parts of the cow's back. In the example in Figure 4, the back image BG is shown with a cropping frame CF set based on key points KP1 to KP3, and the target image TG extracted according to the cropping frame CF is shown. At this time, it is preferable to rotate and resize the target image TG so that the direction and size of the cow CW's hip horns within the target image TG become the desired values. Alternatively, the pixel values ​​of the target image TG may be normalized. This normalization is, for example, normalizing the pixel value of the minimum depth value to 1.0 and the pixel value of the maximum depth value to 0, and further converting the pixel values ​​in the range of 0 to 1.0 to a grayscale of 0 to 255. Steps S23 to S26 may be omitted, and the back image BG may be used as the target image TG as is.

[0028] In step S27, the target image acquisition unit 510 determines whether N(p) target images TG have already been created for a single cow. If N(p) target images TG have not been created, the process returns to step S22, a new back image BG is selected as the processing target, and the processing from step S22 onward is executed again. If N(p) target images TG have been created, the process proceeds to step S28.

[0029] In step S28, the target image acquisition unit 510 determines whether processing has been completed for all n registered target cows and k learning target cows. If processing is not completed for all cows, the process returns to step S21, a new cow is selected as the target for processing, and the processing from step S22 onwards is executed again. If processing is completed for all cows, the processing in step S20 for the p-th image sensor 400(p) is terminated.

[0030] Furthermore, in the processing shown in Figure 3 for images captured by the RGB sensor, it is possible to detect multiple keypoints KP1 to KP3 from the back image BG captured by the RGB sensor, similar to the processing for images captured by the depth sensor, and set the cropping frame CF based on these keypoints. Alternatively, a coordinate transformation matrix between the depth sensor and the RGB sensor may be used to transform the multiple keypoints KP1 to KP3 detected from the back image BG of the depth sensor into coordinates in the sensor coordinate system of the RGB sensor, and then set the cropping frame CF from the transformed keypoints KP1 to KP3. The latter method is particularly useful when the depth sensor and RGB sensor are configured as a single RGBD sensor.

[0031] In step S30 of Figure 2, the learning unit 520 creates distance learning data for deep distance learning by associating N(p) target images TG acquired for each target cow with individual IDs. In step S40, the learning unit 520 performs distance learning of the p-th deep distance learning model 620(p) using the distance learning data. This distance learning is a process of adjusting the internal parameters of the deep distance learning model 620(p) so that the distance of the p-th kind embedding vector output from the deep distance learning model 620(p) is small for the same individual and large for different individuals. The distance of the p-th kind embedding vector may be, for example, the Euclidean distance between vectors or the angle between vectors.

[0032] In step S50, the learning unit 520 sequentially inputs N(p) target images TG for each registered cow into the trained p-th deep distance learning model 620(p), and obtains a p-th type embedding vector for each target image TG. As a result, N(p) p-th type embedding vectors are obtained for each registered cow. In step S60, the learning unit 520 creates the p-th registration data 630(p) for each of the n registered cows by associating the N(p) p-th type embedding vectors with individual IDs.

[0033] In step S70, the learning unit 520 determines whether processing has been completed for all cases where the ordinal number p is 1 to P. If processing is complete, the process shown in Figure 2 is terminated. If processing is not complete, the process returns to step S20, and steps S20 to S70 are executed again using the next value of the ordinal number p.

[0034] By performing the learning process shown in Figure 2 above, we obtain P trained deep distance learning models 620(p) and P registration data 630(p) for n target cows. In the following explanation, target cows will be referred to as "registered individuals." Also, the p-th type embedding vector registered in the registration data 630(p) will be referred to as the "p-th type registration embedding vector."

[0035] Figure 5 is an explanatory diagram showing an example of registration data 630(p). In this example, two image sensors, a depth sensor and an RGB sensor, are used as P image sensors 400(p). The number of target cows n is 4, and the number of target images TG N(p) is N(1)=N(2)=3. Registration data 630(p) includes a first registration data 630(1) created using the depth sensor and a second registration data 630(2) created using the RGB sensor. The first registration data 630(1) contains registration embedding vectors Vr(1) for each of the four registered individuals, corresponding to three target images TG. In this example, the registration embedding vector Vr(1) is a 5-dimensional vector with 5 elements. The individual IDs of the four registered individuals are ID1 to ID4. Also, the three target images TG obtained for each individual are assigned different image IDs. However, image IDs do not necessarily have to be registered. The second registration data 630(2) also contains registration embedding vectors Vr(2) for three target images TG for each of the same four individuals as the first registration data 630(1). Normally, a larger number of registration embedding vectors Vr(p) are registered for each registered individual, but in Figure 5, the number of registration embedding vectors Vr(p) has been reduced for illustrative purposes.

[0036] Figure 6 is a flowchart showing the procedure for individual identification. It is preferable that the individual identification process be performed periodically, for example, on all cattle raised in the same farm, with the cattle being identified as the target animals.

[0037] In step S110, the individual identification unit 530 uses two image sensors 500(p) to acquire N1 target images TG of the back of one target cow. Here, N1 is an integer greater than or equal to 1. The value of N1 may be set to a value equal to the number of p-th type registration embedding vectors for each registered individual registered in the registration data 630(p), or it may be set to a different value. Also, N1 may not be a pre-set value, but the number of target images TG that have been acquired for the target cow at that time. In the following description, the number of p-th type registration embedding vectors for each registered individual will be referred to as "N2" and distinguished from the number of target images TG N1 of the target cow. The integer N2 is the same as the integer N(p) used in Figures 2 and 3, and is an integer greater than or equal to 2. In the example in Figure 5, the number of p-th type registration embedding vectors N2 is 3. It is preferable that each of the integers N1 and N2 be a constant value independent of the ordinal number p. The specific processing steps in step S110 are the same as the process for creating target image TGs for registered cattle, as explained in Figures 3 and 4.

[0038] In step S120, the individual identification unit 530 uses two pre-trained deep distance learning models 620(p) to obtain N1 p-th kind embedding vectors for N1 target images TG.

[0039] Figure 7 is an explanatory diagram showing an example of an embedding vector calculated in the individual identification process. In this example, the number of target images TG N1 is 3. That is, the first type embedding vector Vt(1) and the second type embedding vector Vt(2) are calculated for the three target images obtained for the cow to be identified. The p-th type embedding vector Vt(p) is a vector of the same dimension as the p-th type registration embedding vector Vr(p) shown in Figure 5.

[0040] In step S130, the individual identification unit 530 calculates the p-th species distance, which is the distance between N1 p-th species embedding vectors and N2 p-th species registration embedding vectors for each registered individual. As a result, N1 × N2 p-th species distances are calculated for each registered individual. Furthermore, since there are n registered individuals, for each cow to be identified, n × N1 × N2 p-th species distances are calculated using the p-th deep distance learning model 620(p). In other words, n × N1 × N2 p-th species distances are calculated from the images captured by each image sensor 400(p).

[0041] Figure 8 is an explanatory diagram illustrating an example of the distance between the embedding vector and the registered embedding vector. For the registered embedding vector Vr(p) shown in Figure 5, n=4 and N2=3, and for the embedding vector Vt(p) shown in Figure 7, N1=3. Therefore, for one target cow, 36 p-th distances L(p) are calculated from the images captured by each individual image sensor 400(p).

[0042] In step S140, the individual identification unit 530 determines the individual ID of the cow to be identified from the p-th distance calculated in step S130.

[0043] Figure 9 is a flowchart showing the detailed procedure for step S140 in the first embodiment. In step S141, the individual identification unit 530 determines an integrated determination distance Ljt(ID) for each of the n registered individuals by integrating N1 × N2 p-th type distances. The integrated determination distance Ljt(ID) can be determined by, for example, one of the following methods.

[0044] <Method for determining the integrated judgment distance Ljt(ID) DM1> For each registered individual, the integrated determination distance Ljt(ID) is determined to be a value proportional to the sum of the following: the average value of the first type distance obtained by averaging M first type distances L(1) selected in ascending order from N1 × N2 first type distances L(1), and the average value of the second type distance obtained by averaging M second type distances L(2) selected in ascending order from N1 × N2 second type distances L(2). M is an integer of 2 or greater and less than or equal to N1 × N2, and preferably less than N1 × N2. It is also preferable that integers N1 and N2 are set such that N1 × N2 is 3 or greater. The "value proportional to the sum" may be the sum itself, or it may be a value obtained by multiplying that value by a positive coefficient.

[0045] <Method for determining the integrated judgment distance Ljt(ID) DM2> For each registered individual, N1 × N2 first-type distances L(1) and corresponding N1 × N2 second-type distances L(2) are added together to obtain N1 × N2 summation results. From these N1 × N2 summation results, M summation results are selected in ascending order, and a value proportional to the average value of the selected M summation results is determined as the integrated determination distance Ljt(ID). The term "corresponding" means that the results are obtained using images taken at substantially the same shooting timing using two image sensors 400(p). Specifically, the first-type distance L(1) and second-type distance L(2) obtained using images with the same frame number taken by the depth sensor and RGB sensor, respectively, correspond to the distances that "correspond" to each other.

[0046] Figure 10 is an explanatory diagram showing examples of various judgment distances. At the top of Figure 10, the integrated judgment distance Ljt(ID) calculated using M=3 by applying the above-described determination method DM1 to the p-th type distance shown in Figure 8 is displayed.

[0047] In step S142 of Figure 9, the individual identification unit 530 determines whether the minimum value of the integrated determination distance Ljt(ID) is smaller than the pre-set integrated threshold Tht. If the minimum value of the integrated determination distance Ljt(ID) is smaller than the integrated threshold Tht, the process proceeds to step S151, where the registered individual corresponding to the minimum value of the integrated determination distance Ljt(ID) is identified as the individual of the cattle to be identified. In the example in Figure 10, the minimum value of the integrated determination distance Ljt(ID) corresponds to the registered individual whose individual ID is ID3, so this registered individual is identified as the individual of the cattle to be identified.

[0048] On the other hand, if the minimum value of the integrated determination distance Ljt(ID) is greater than the integrated threshold Tht, the process proceeds to step S143. The minimum value of the integrated determination distance Ljt(ID) is greater than the integrated threshold Tht when the cow to be identified is an unregistered individual, or when there is dirt on the back of the cow to be identified. In this case, the identification determination is performed using a determination value different from the integrated determination distance Ljt(ID).

[0049] Furthermore, if the minimum value of the integrated determination distance Ljt(ID) is equal to the integrated threshold Tht, the process proceeds to one of the two pre-selected branch destinations from step S142. This is also true for other determination steps that use thresholds.

[0050] In step S143, the individual identification unit 530 determines a first-class determination distance Lj1(ID) that represents N1 × N2 first-class distances L(1) for each of the n registered individuals. The first-class determination distance Lj1(ID) is determined by, for example, one of the following methods.

[0051] <Method for determining the Type 1 classification distance Lj1 (ID) EM1> For each registered individual, M1 values ​​are selected in ascending order from N1 × N2 first-type distances L(1), and the first-type determination distance Lj1(ID) is determined to be a value proportional to the average value of the M1 first-type distances L(1). Here, M1 is an integer greater than or equal to 2 and less than or equal to N1 × N2, and preferably less than N1 × N2. Furthermore, it is preferable that the integers N1 and N2 are set such that N1 × N2 is 3 or greater.

[0052] <Method for determining the Type 1 classification distance Lj1 (ID) EM2> For each registered individual, the Class 1 determination distance Lj1(ID) is determined to be a value proportional to the minimum value among N1 × N2 Class 1 distances L(1). The "value proportional to the minimum value" may be the minimum value itself, or the minimum value multiplied by a positive coefficient.

[0053] In the first embodiment, the determination method EM1 described above is applied to the p-th type distance shown in Figure 8, and the first type determination distance Lj1(ID) is determined using M1=3. The first type determination distance Lj1(ID) thus determined is shown at the bottom of Figure 10.

[0054] In step S144, the individual identification unit 530 determines whether the minimum value of the first type determination distance Lj1(ID) is smaller than a preset first threshold Th1. If the minimum value of the first type determination distance Lj1(ID) is smaller than the first threshold Th1, the process proceeds to step S152, where the registered individual corresponding to the minimum value of the first type determination distance Lj1(ID) is identified as the individual of the cattle to be identified.

[0055] On the other hand, if the minimum value of the first type determination distance Lj1(ID) is greater than the first threshold Th1, the process proceeds to step S145. In step S145, the individual identification unit 530 determines a second type determination distance Lj2(ID) that represents N1 × N2 second type distances L(2) for each of the n registered individuals. The second type determination distance Lj2(ID) is determined in the same way as the first type determination distance Lj1(ID) determination methods EM1, EM1 described above. The lower part of Figure 10 shows the second type determination distance Lj2(ID) calculated according to the same method as the determination method EM1 described above.

[0056] In step S146, the individual identification unit 530 determines whether the minimum value of the second type determination distance Lj2(ID) is smaller than a preset second threshold Th2. If the minimum value of the second type determination distance Lj2(ID) is smaller than the second threshold Th2, the process proceeds to step S153, where the registered individual corresponding to the minimum value of the second type determination distance Lj2(ID) is identified as the individual of the cattle to be identified.

[0057] On the other hand, if the minimum value of the second type determination distance Lj2(ID) is greater than the second threshold Th2, the process proceeds to step S154. In step S154, the individual identification unit 530 determines that the cow to be identified is an unregistered individual. An unregistered individual means an individual that does not correspond to any of the multiple registered individuals registered in the registration data 630(p).

[0058] According to the processing shown in Figures 6 to 10 above, the individual cow to be identified can be determined from N1 × N2 p-th type distances for each registered individual. In particular, in the first embodiment, individuals can be identified using various determination distances, including the integrated determination distance Ljt(ID). Furthermore, if the N1 × N2 p-th type distances satisfy the pre-set unregistered condition, the cow to be identified can be determined to be an unregistered individual.

[0059] Figure 11 is a flowchart showing the procedure for the registration update process. This registration update process is preferably performed periodically, for example, once a day.

[0060] In step S210, the individual identification unit 530 updates the registration data 630(p) using the embedding vector Vt obtained for the cattle to be identified in the process shown in Figure 9. For example, if N1 = N2, that is, if the number of embedding vectors Vt obtained for the cattle to be identified N1 is equal to the number of registration embedding vectors Vr for each registered individual in the registration data 630(p), the registration data 630 may be updated to replace N(p) registration embedding vectors Vr with new N(p) embedding vectors Vt. Alternatively, the registration data 630 may be updated to add N1 embedding vectors Vt obtained for the cattle to be identified without discarding the old registration embedding vectors Vr.

[0061] Furthermore, when q is an integer greater than or equal to 2, if the number of registered embedding vectors Vr N2 is equal to q × N1, the oldest N1 registered embedding vectors Vr may be discarded and the newly obtained N1 embedding vectors Vt for the identified cow may be added to update the registration data 630. In this way, the registration data 630(p) can always be updated to include the latest q × N1 registered embedding vectors Vr. Alternatively, instead of updating the registration data 630(p) by adding the same number of registered embedding vectors Vr each time, the registration data 630(p) may be updated by adding a different number of registered embedding vectors Vr each time. That is, the number of registrations may be counted, and the registration data 630(p) may always be updated to include q registered embedding vectors Vr.

[0062] In step S220, the individual identification unit 530 determines whether or not a new registered individual exists. A new registered individual is an individual not yet registered in the registration data 630(p). For example, in the process shown in Figure 9, if the cow to be identified is determined to be an unregistered individual, that cow becomes a new registered individual. Alternatively, the user may specify a new registered individual. If no new registered individuals exist, the process proceeds to step S240, which will be described later. On the other hand, if a new registered individual exists, the process proceeds to step S230, where the individual identification unit 530 obtains N(p) embedding vectors for the new registered individual and registers them in the registration data 630(p) in association with the individual ID. Here, N(p) may be equal to the number N1 of embedding vectors Vt for the cow to be identified used in step S210, or it may be equal to the number N2 of registration embedding vectors Vr for each registered individual in the registration data 630(p). The process in step S230 is almost the same as the processes in steps S10, S20, S50, and S60 in Figure 2 described above.

[0063] In step S240, the individual identification unit 530 determines whether or not there are individuals to be deleted from the registered data 630(p). Whether or not there are individuals to be deleted is specified by the user. If there are no individuals to be deleted, the process shown in Figure 10 is terminated. On the other hand, if there are individuals to be deleted, the process proceeds to step S250, where the individual identification unit 530 deletes the data of those individuals from the registered data 630(p). By executing the process shown in Figure 10, it is possible to maintain the registered data 630(p) with the latest content.

[0064] According to the first embodiment described above, two types of embedding vectors Vt can be obtained from a target image TG relating to the back of the target cow using two deep distance learning models 620(p), and individuals can be identified using various judgment distances determined from two types of distances between these two types of embedding vectors Vt and two types of registered embedding vectors Vr. Furthermore, since the target cow is identified using two types of target images TG captured using two types of image sensors 400(p), the identification accuracy can be improved compared to when only one type of image sensor is used. Moreover, in the first embodiment, if the minimum value of each of the three judgment distances Ljt(ID), Lj1(ID), and Lj2(ID) is greater than the respective thresholds Tht, Th1, and Th2, it can be determined that the target cow is an unregistered individual.

[0065] B. Second Embodiment: Figure 12 is a flowchart showing the detailed procedure for step S140 in the second embodiment. The configuration of the apparatus in the second embodiment and the processing content in Figures 2, 3, and 6 are the same as in the first embodiment. The processing procedure in Figure 12 omits steps S145, S146, and S153 in Figure 9, and the other steps are the same as in the first embodiment.

[0066] In step S144 of the second embodiment, if the minimum value of the first type determination distance Lj1(ID) is greater than the first threshold Th1, the process proceeds to step S154, and it is determined that the cow to be identified is an unregistered individual.

[0067] The second embodiment also produces almost the same effects as the first embodiment. Furthermore, according to the process shown in Figure 12 above, if the minimum values ​​of the integrated determination distance Ljt(ID) and the first type determination distance Lj1(ID) are greater than their respective thresholds Tht and Th1, it can be determined that the cow to be identified is an unregistered individual.

[0068] C. Third Embodiment: Figure 13 is a flowchart detailing the procedure for step S140 in the third embodiment. The configuration of the apparatus in the third embodiment and the processing content in Figures 2, 3, and 6 are the same as in the first and second embodiments. The processing procedure in Figure 13 further omits steps S143, S144, and S152 in Figure 12, and the other steps are the same as in the second embodiment.

[0069] In step S142 of the third embodiment, if the minimum value of the integrated determination distance Ljt(ID) is greater than the integrated threshold Tht, the process proceeds to step S154, where it is determined that the cow to be identified is an unregistered individual.

[0070] The third embodiment also produces substantially the same effects as the first and second embodiments. Furthermore, according to the process shown in Figure 13 described above, if the minimum value of the integrated determination distance Ljt(ID) is greater than the threshold Tht, it can be determined that the cow to be identified is an unregistered individual.

[0071] Furthermore, the individual identification process shown in Figure 9 in the first embodiment, the individual identification process shown in Figure 12 in the second embodiment, and the individual identification process shown in Figure 13 in the third embodiment are the same in that they determine that the target cow is an unregistered individual when it satisfies the unregistered condition, which includes the minimum value of the integrated determination distance Ljt(ID) being greater than the integrated threshold Tht.

[0072] Other forms: This disclosure is not limited to the embodiments described above, and can be implemented in various forms without departing from its spirit. For example, this disclosure can also be implemented in the following forms (aspects). The technical features in the embodiments described above that correspond to the technical features in each of the forms described below can be replaced or combined as appropriate in order to solve some or all of the problems of this disclosure, or to achieve some or all of the effects of this disclosure. Furthermore, if such technical features are not described as essential in this specification, they can be deleted as appropriate.

[0073] (1) According to a first embodiment of the present disclosure, a method for individual identification of an animal to be identified is provided. This method includes: (a) when p is an ordinal number from 1 to 2 and N1 is an integer of 1 or more, acquiring N1 p-th type target images with respect to the back of the animal to be identified using a p-th type image sensor; (b) obtaining N1 p-th type embedding vectors from the N1 p-th type target images using a p-th deep distance learning model; (c) when n and N2 are integers of 2 or more, calculating N1 × N2 p-th type distances between the N1 p-th type embedding vectors and the N2 p-th type registration embedding vectors using registration data that includes N2 p-th type registration embedding vectors that have been generated in advance for each of the n registered individuals; and (d) determining which of the n registered individuals the animal to be identified corresponds to using the N1 × N2 p-th type distances for each of the n registered individuals. The step (d) includes (d1) determining an integrated determination distance for each of the n registered individuals by integrating N1 × N2 first-type distances and N1 × N2 second-type distances; (d2) determining that the registered individual corresponding to the minimum value of the integrated determination distance is an individual of the target animal if the minimum value of the integrated determination distance is smaller than a preset integration threshold; and (d3) determining that the target animal is an unregistered individual that does not correspond to any of the n registered individuals if it satisfies an unregistered condition including that the minimum value of the integrated determination distance is larger than the integration threshold. This method uses two deep distance learning models to obtain two types of embedding vectors from two types of target images of the back of the target animal. The integrated decision distance, determined from the distance between these two embedding vectors and two registered embedding vectors, can then be used to identify the individual. Furthermore, if the p-th species distance satisfies the pre-set unregistered condition, the target animal can be determined to be an unregistered individual.

[0074] (2) In the above method, when M is an integer of 2 or more and less than or equal to N1 × N2, the combined determination distance for each of the n registered individuals may be a value proportional to the sum of the first type distance average value obtained by averaging the M first type distances selected in ascending order from the N1 × N2 first type distances and the second type distance average value obtained by averaging the M second type distances selected in ascending order from the N1 × N2 second type distances. This method allows for the calculation of an appropriate overall judgment distance.

[0075] (3) In the above method, step (d3) may include (d3-1) a step of determining a first type determination distance that represents the N1 × N2 first type distances for each of the n registered individuals, and (d3-2) a step of determining that the registered individual corresponding to the minimum value of the n first type determination distance is an individual of the animal to be identified, when the minimum value of the n first type determination distances is smaller than a preset first type threshold.

[0076] (4) In the above method, step (d3) may further include: (d3-3) when the minimum value of the first type determination distance is greater than the first type threshold, a step of determining a second type determination distance that represents the N1 × N2 second type distances for each of the n registered individuals; (d3-4) when the minimum value of the n second type determination distances is less than a preset second type threshold, a step of determining that the registered individual corresponding to the minimum value of the second type determination distance is an individual of the animal to be identified; and (d3-5) when the minimum value of the second type determination distance is greater than the second type threshold, a step of determining that the animal to be identified is an unregistered individual that does not fall under any of the plurality of registered individuals. This method allows for the determination of whether the animal being identified is a registered individual or an unregistered individual, based on the Type 1 and Type 2 determination distances.

[0077] (5) In the above method, step (d3) may further include (d3-3) a step of determining that the animal to be identified is an unregistered individual that does not belong to any of the plurality of registered individuals when the minimum value of the first type determination distance is greater than the first type threshold. According to this method, it is possible to determine whether the animal being identified is a registered individual or an unregistered individual, depending on the Type 1 determination distance.

[0078] (6) In the above method, when M is an integer of 2 or more and less than or equal to N1 × N2, the first species determination distance for each of the n registered individuals may be a value proportional to the average first species distance obtained by averaging the M first species distances selected in ascending order from the N1 × N2 first species distances. This method allows for the calculation of an appropriate Class 1 classification distance.

[0079] (7) In the above method, the first type image sensor may be a depth sensor and the second type image sensor may be an RGB sensor. This method allows for individual identification using depth sensors and RGB sensors.

[0080] (8) In the above method, step (a) may include (a1) a step of acquiring a depth image and a color image of the dorsal side of the animal to be identified using the depth sensor and the RGB sensor; (a2) a step of detecting a plurality of key points from at least one of the depth image and the color image using an object recognition model; (a3) ​​a step of generating a first type target image by cutting out the characteristic portion of the dorsal side from the depth image based on the positions of the plurality of key points; and (a4) a step of generating a second type target image by cutting out the characteristic portion of the dorsal side from the color image based on the positions of the plurality of key points. This method allows us to obtain a target image, including the back surface features, from depth images and color images.

[0081] (9) According to a second embodiment of the present disclosure, a computer program is provided that causes a processor to perform a process for individual identification of an animal to be identified. The computer program causes the processor to perform the following: (a) when p is an ordinal number from 1 to 2 and N1 is an integer of 1 or more, a process to acquire N1 p-th type target images with respect to the back of the animal to be identified using a p-th type image sensor; (b) a process to obtain N1 p-th type embedding vectors from the N1 p-th type target images using a p-th deep distance learning model; (c) when n and N2 are integers of 2 or more, a process to calculate N1 × N2 p-th type distances between the N1 p-th type embedding vectors and the N2 p-th type registration embedding vectors using registration data that includes N2 p-th type registration embedding vectors that have been generated in advance for each of the n registered individuals; and (d) a process to determine which of the n registered individuals the animal to be identified corresponds to using the N1 × N2 p-th type distances for each of the n registered individuals. The process (d) includes (d1) a process of determining an integrated determination distance for each of the n registered individuals by integrating N1 × N2 first type distances and N1 × N2 second type distances; (d2) a process of determining that the registered individual corresponding to the minimum value of the integrated determination distance is an individual of the target animal if the minimum value of the integrated determination distance is smaller than a preset integration threshold; and (d3) a process of determining that the target animal is an unregistered individual that does not fall under any of the n registered individuals if it satisfies the unregistered condition, which includes the minimum value of the integrated determination distance being larger than the integration threshold.

[0082] This disclosure can also be implemented in various forms other than those described above. For example, it can be implemented in the form of a device that performs individual identification processing, or a non-transitory storage medium on which a computer program is recorded. [Explanation of symbols]

[0083] 300... Information processing device, 310... Processor, 320... Memory, 330... Interface circuit, 340... Input device, 350... Display device, 400... Image sensor, 510... Target image acquisition unit, 520... Learning unit, 530... Individual identification unit, 610... Object recognition model, 620... Deep distance learning model, 630... Registered data

Claims

1. A method for identifying individual animals, (a) When p is an ordinal number from 1 to 2 and N1 is an integer of 1 or greater, the process of acquiring N1 type p target images of the dorsal side of the animal to be identified using a type p image sensor, (b) A step of obtaining N1 p-th type embedding vectors from the N1 p-th type target images using a p-th deep distance learning model, (c) When n and N2 are integers of 2 or more, the process of calculating N1 × N2 p-th type distances between the N1 p-th type embedding vectors and the N2 p-th type registration embedding vectors using registration data that includes N2 p-th type registration embedding vectors that have been generated in advance for each of the n registered individuals, (d) A step of determining which of the n registered individuals the target animal corresponds to, using the N1 × N2 p-species distances for each of the n registered individuals, Includes, The aforementioned step (d) is, (d1) A step of determining an integrated determination distance for each of the n registered individuals by integrating N1 × N2 first type distances and N1 × N2 second type distances, (d2) When the minimum value of the integrated determination distance is smaller than a preset integrated threshold, the step of determining that the registered individual corresponding to the minimum value of the integrated determination distance is an individual of the animal to be identified, (d3) A step of determining that the animal to be identified is an unregistered individual that does not correspond to any of the n registered individuals when the unregistered condition is met, which includes the minimum value of the integrated determination distance being greater than the integrated threshold, Methods that include...

2. The method according to claim 1, A method in which, when M is an integer of 2 or more and less than or equal to N1 × N2, the integrated determination distance for each of the n registered individuals is a value proportional to the sum of the first type distance mean value obtained by averaging M first type distances selected in ascending order from the N1 × N2 first type distances and the second type distance mean value obtained by averaging M second type distances selected in ascending order from the N1 × N2 second type distances.

3. The method according to claim 1, The aforementioned step (d3) is, (d3-1) A step of determining a first type determination distance that represents the N1 × N2 first type distances for each of the n registered individuals, (d3-2) A step in which, if the minimum value of the n first type determination distances is smaller than a preset first type threshold, the registered individual corresponding to the minimum value of the first type determination distance is determined to be an individual of the animal to be identified. Methods that include...

4. The method according to claim 3, The above step (d3) further, (d3-3) When the minimum value of the first type determination distance is greater than the first type threshold, a step is taken to determine a second type determination distance that represents the N1 × N2 second type distances for each of the n registered individuals, (d3-4) A step in which, if the minimum value of the n second type determination distances is smaller than a preset second type threshold, the registered individual corresponding to the minimum value of the second type determination distance is determined to be an individual of the animal to be identified. (d3-5) A step in which, if the minimum value of the second type determination distance is greater than the second type threshold, the animal to be identified is determined to be an unregistered individual that does not correspond to any of the multiple registered individuals, Methods that include...

5. The method according to claim 3, The above step (d3) further, (d3-3) A step in which, if the minimum value of the first species determination distance is greater than the first species threshold, the animal to be identified is an unregistered individual that does not correspond to any of the multiple registered individuals, Methods that include...

6. The method according to claim 3, A method in which, when M is an integer between 2 and N1 × N2, the first species determination distance for each of the n registered individuals is a value proportional to the average first species distance obtained by averaging the M first species distances selected in ascending order from the N1 × N2 first species distances.

7. The method according to claim 1, The first type of image sensor is a depth sensor, and the second type of image sensor is an RGB sensor, in this method.

8. The method according to claim 7, The above step (a) is, (a1) A step of acquiring a depth image and a color image of the dorsal side of the animal to be identified using the depth sensor and the RGB sensor, (a2) A step of detecting a plurality of key points from at least one of the depth image and the color image using an object recognition model, (a3) A step of generating a first type target image by extracting the back surface feature portion from the depth image based on the positions of the plurality of key points, (a4) A step of generating a second type target image by cutting out the characteristic portion of the back surface from the color image based on the positions of the plurality of key points, Methods that include...

9. A computer program that causes a processor to perform a process for identifying individual animals, (a) When p is an ordinal number from 1 to 2 and N1 is an integer of 1 or greater, the process of acquiring N1 type p target images of the dorsal side of the animal to be identified using a type p image sensor, (b) A process to obtain N1 p-th type embedding vectors from the N1 p-th type target images using a p-th deep distance learning model, (c) When n and N2 are integers of 2 or more, a process to calculate N1 × N2 p-th type distances between the N1 p-th type embedding vectors and the N2 p-th type registration embedding vectors using registration data that includes N2 p-th type registration embedding vectors that have been generated in advance for each of the n registered individuals, (d) A process to determine which of the n registered individuals the target animal corresponds to, using the N1 × N2 p-species distances for each of the n registered individuals, The processor is made to execute the above, The process (d) is, (d1) For each of the n registered individuals, a process to calculate an integrated determination distance by integrating N1 × N2 first-type distances and N1 × N2 second-type distances, (d2) When the minimum value of the integrated determination distance is smaller than a preset integrated threshold, the process of determining that the registered individual corresponding to the minimum value of the integrated determination distance is an individual of the animal to be identified, (d3) A process to determine that the animal to be identified is an unregistered individual that does not correspond to any of the n registered individuals when the unregistered condition is met, which includes the minimum value of the integrated determination distance being greater than the integrated threshold, A computer program that includes [this].

Images