Fish individual identification method, device, and program

Abstract

To provide a fish individual identification method, a fish individual identification device, and a fish individual identification program for identifying a fish for each individual by using a method different from an identification method of a human.SOLUTION: A fish individual identification device of the present invention is a device that identifies a fish including a pattern on a body surface, for each individual. The fish individual identification device includes: a fish image acquisition unit 1 that acquires an image of the body surface of the fish; and a fish individual identification unit 23 that identifies a fish appearing in the image acquired by the fish image acquisition unit 1 for each individual by using a machine learning model that identifies a fish for each individual on the basis of an image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a fish individual identification method, a fish individual identification device, and a fish individual identification program for individually identifying fish having patterns on their body surfaces. [Background technology]

[0002] Organisms have traditionally been used in medical research, biological research, and the like. Zebrafish (scientific name: Danio rerio) are sometimes used as such organisms because they are favorable for such research, such as being easy to raise, being prolific, having a short generation cycle, and being easy to observe embryos. Since such research requires the observation of individual individuals, individual identification is necessary. Since manual individual identification is time-consuming and laborious, automation is desirable, and one such technology is the individual identification method disclosed in Patent Document 1, for example.

[0003] The method for identifying individual zebrafish disclosed in Patent Document 1 is a method for identifying multiple zebrafish (Danio rerio) swimming in an aquarium, which involves photographing a side image of each individual zebrafish from the side of the aquarium to obtain lateral image data, determining characteristics of the stripe patterns on the anal and caudal fins of each individual from the obtained lateral image data of each individual to obtain characteristic data for each individual, and identifying each individual from the multiple zebrafish based on the obtained characteristic data. In this method for identifying individual zebrafish, the characteristic data on the stripe patterns on the anal and caudal fins is at least one piece of data selected from the group consisting of whether or not stripes branch, whether or not stripes are separated, the number of stripes, whether or not stripes are curved, and whether or not stripe width changes. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2016-253724 A (Patent No. 6679474 A) Summary of the Invention [Problem to be solved by the invention]

[0005] The method for identifying zebrafish individuals disclosed in Patent Document 1 automates the conventional method of identifying stripe patterns by hand by generating images and processing the images, but individual identification may not be possible based solely on the presence or absence of branching or division of stripes, the number of stripes, the presence or absence of curvature of stripes, and the presence or absence of changes in stripe width. In particular, genetically close individuals have similar stripe patterns, making individual identification difficult and increasing the risk of misidentification.

[0006] The present invention has been made in consideration of the above-mentioned circumstances, and its purpose is to provide a fish individual identification method, a fish individual identification device, and a fish individual identification program that identify individual fish using a technique different from human identification methods. [Means for solving the problem]

[0007] After extensive investigation, the inventors have found that the above object can be achieved by the present invention described below. Specifically, a fish individual identification method according to one aspect of the present invention is a method for individually identifying fish having a pattern on its body surface, comprising: a fish image acquisition step of acquiring an image of the body surface of the fish; and a fish individual identification step of individually identifying fish appearing in the image acquired in the fish image acquisition step by using a machine learning model that individually identifies fish based on the image. A fish individual identification device according to another aspect of the present invention is a device for individually identifying fish having a pattern on its body surface, comprising: a fish image acquisition unit that acquires an image of the body surface of the fish; and a fish individual identification unit that individually identifies fish appearing in the image acquired by the fish image acquisition unit by using a machine learning model that individually identifies fish based on the image. Preferably, in the above-described individual fish identification method and individual fish identification device, the pattern includes a striped pattern. Preferably, in the above-described individual fish identification method and individual fish identification device, the pattern includes vertical stripes. Preferably, in the above-described individual fish identification method and individual fish identification device, the pattern includes horizontal stripes. Preferably, in the above-described individual fish identification method and individual fish identification device, the fish is a zebrafish.

[0008] This individual fish identification method and individual fish identification device uses a machine learning model that identifies individual fish based on images to identify each individual fish, thereby providing an individual fish identification method and individual fish identification device that identify each individual fish using a technique different from human identification methods.

[0009] In another aspect, the above-mentioned individual fish identification method further comprises a machine learning step of machine learning the machine learning model using training data, and the machine learning model used in the individual fish identification step is a machine learning model that has been machine learned in the machine learning step.In another aspect, the above-mentioned individual fish identification device further comprises a machine learning unit that machine learns the machine learning model using training data, and the machine learning model used in the individual fish identification unit is a machine learning model that has been machine learned in the machine learning unit.Preferably, in each of the above-mentioned individual fish identification method and individual fish identification device, the training data is supervised training data that includes an image containing a fish and an identifier (e.g., an individual name) for specifying and identifying the fish appearing in the image as a supervised data.

[0010] This provides a fish individual identification method and a fish individual identification device that generate a machine-learned model through machine learning.

[0011] In another aspect, in the above-mentioned fish individual identification method, the training data includes images of the fish with a monochromatic background. In another aspect, in the above-mentioned fish individual identification device, the training data includes images of the fish with a monochromatic background.

[0012] Such a method and device for identifying individual fish trains the machine learning model using training data that includes images of the fish with a monochrome background, thereby removing the noisy background and training only on the fish, enabling the machine learning model to be trained to identify individuals more accurately.

[0013] In another aspect, in the above-mentioned individual fish identification methods, the learning data includes an image of one side of the fish and an image of the other side of the fish. In another aspect, in the above-mentioned individual fish identification devices, the learning data includes an image of one side of the fish and an image of the other side of the fish. Preferably, in the above-mentioned individual fish identification methods and individual fish identification devices, the image of one side of the fish and the image of the other side of the fish are formed as a single image. Preferably, in the above-mentioned individual fish identification methods and individual fish identification devices, the image of one side of the fish and the image of the other side of the fish are formed as separate images.

[0014] Such a method and device for identifying individual fish trains the machine learning model using training data including an image of one side of the fish and an image of the other side of the fish, so that the machine learning model can be trained using patterns on both sides, allowing for more accurate individual identification.

[0015] In another aspect, in these above-mentioned individual fish identification methods, the individual fish identification step further includes an individual name correction step of outputting the individual name of the identified individual and correcting the individual name output in the individual fish identification step.In another aspect, in these above-mentioned individual fish identification devices, the individual fish identification unit further includes an individual name correction unit of outputting the individual name of the identified individual and correcting the individual name output in the individual fish identification unit.

[0016] Such a fish individual identification method and fish individual identification device can correct the individual name even if an error is made in the individual identification.

[0017] In another aspect, the above-mentioned fish individual identification method further comprises a classification step of classifying the plurality of fish into groups by using the pseudo-space of the machine learning model. In another aspect, the above-mentioned fish individual identification device further comprises a classification processing unit of classifying the plurality of fish into groups by using the pseudo-space of the machine learning model.

[0018] Such a fish individual identification method and fish individual identification device classifies multiple fish into groups, so that by selecting fish from each group, individual identification becomes easier and it becomes possible to reduce erroneous individual identification decisions.

[0019] In another aspect, the above-mentioned fish individual identification methods further include a machine learning step of machine-learning the machine learning model using training data, and a transmission step of transmitting at least one of the training data and the machine learning model to a management server device that manages at least one of the training data and the machine learning model in a shareable manner.In another aspect, the above-mentioned fish individual identification devices further include a machine learning unit that machine-learns the machine learning model using training data, and a transmission processing unit that transmits at least one of the training data and the machine learning model to a management server device that manages at least one of the training data and the machine learning model in a shareable manner.

[0020] Such an individual fish identification method and individual fish identification device can utilize at least one of the training data and the machine learning model, which can be shared, thereby enabling the accuracy of the machine learning model to be improved.

[0021] A fish individual identification program according to another aspect of the present invention is a program for causing a computer to function as any one of the above-described fish individual identification devices.

[0022] This makes it possible to provide an individual fish identification program, which has the same effects as the individual fish identification devices described above. [Effects of the Invention]

[0023] According to the present invention, it is possible to provide a fish individual identification method, a fish individual identification device, and a fish individual identification program that identify individual fish using a technique different from human identification methods. [Brief explanation of the drawings]

[0024] [Figure 1] 1 is a block diagram showing the configuration of a fish individual identification device according to an embodiment. [Figure 2] FIG. 2 is a schematic diagram showing the configuration of a drug administration experiment system using the fish individual identification device. [Figure 3] FIG. 2 is a diagram showing, as an example, fish images of learning data in a first mode using the fish individual identification device. [Figure 4] FIG. 10 is a diagram showing, as an example, fish images of learning data in a second mode using the fish individual identification device. [Figure 5] FIG. 10 is a diagram showing, as an example, fish images of learning data in a third mode using the fish individual identification device. [Figure 6] FIG. 10 is a diagram showing, as an example, fish images of learning data in a fourth mode using the fish individual identification device. [Figure 7] FIG. 10 is a diagram showing, as an example, fish images of learning data in a fifth aspect using the fish individual identification device. [Figure 8] FIG. 10 is a diagram showing a fish information table as an example. [Figure 9] 4 is a flowchart showing the operation of the fish individual identification device regarding machine learning. [Figure 10] FIG. 3 is a diagram showing, as an example, a first screen displayed on the fish individual identification device. [Figure 11] FIG. 10 is a diagram showing, as an example, a second screen displayed on the fish individual identification device. [Figure 12] FIG. 10 is a diagram showing, as an example, a third screen displayed on the fish individual identification device. [Figure 13] FIG. 10 is a diagram showing, as an example, a fourth screen displayed on the fish individual identification device. [Figure 14] FIG. 10 is a diagram showing, as an example, a fifth screen displayed on the fish individual identification device. [Figure 15] FIG. 10 is a diagram showing a sixth screen displayed on the fish individual identification device, as an example. [Figure 16] FIG. 10 is a diagram showing, as an example, a seventh screen displayed on the fish individual identification device. [Figure 17]FIG. 10 is a diagram showing an eighth screen displayed on the fish individual identification device, as an example. [Figure 18] FIG. 10 is a diagram showing, as an example, a ninth screen displayed on the fish individual identification device. [Figure 19] 4 is a flowchart showing the operation of the fish individual identification device regarding individual identification. [Figure 20] FIG. 13 is a diagram showing, as an example, a tenth screen displayed on the fish individual identification device. [Figure 21] FIG. 11 is a diagram showing an eleventh screen displayed on the fish individual identification device as an example. [Figure 22] FIG. 13 is a diagram showing, as an example, a twelfth screen displayed on the fish individual identification device. [Figure 23] FIG. 13 is a diagram showing, as an example, a thirteenth screen displayed on the fish individual identification device. [Figure 24] FIG. 13 is a diagram showing, as an example, a fourteenth screen displayed on the fish individual identification device. [Figure 25] FIG. 10 is a diagram for explaining a fish individual identification device in a first modified embodiment. [Figure 26] FIG. 10 is a diagram for explaining a fish individual identification device in a second modified embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0025] An embodiment of the present invention will be described below with reference to the drawings. In each drawing, components with the same reference numerals are designated as identical components, and their description will be omitted where appropriate. In this specification, generic reference numerals are used without subscripts, and individual components are designated with subscripts.

[0026] The fish individual identification device in this embodiment is a device that individually identifies fish having patterns on their body surfaces. This fish individual identification device includes a fish image acquisition unit and a fish individual identification unit. The fish image acquisition unit acquires an image of the body surface of the fish. The fish individual identification unit identifies each individual fish that appears in an image acquired by the fish image acquisition unit by using a machine learning model that individually identifies each fish based on the image. Below, such a fish individual identification device, as well as the fish individual identification method and fish individual identification program implemented therein, will be described in more detail.

[0027] FIG. 1 is a block diagram showing the configuration of an individual fish identification device according to an embodiment. FIG. 2 is a schematic diagram showing the configuration of a drug administration experiment system using the individual fish identification device. FIG. 3 is a diagram showing, as an example, fish images of training data in a first mode using the individual fish identification device. FIG. 4 is a diagram showing, as an example, fish images of training data in a second mode using the individual fish identification device. FIG. 5 is a diagram showing, as an example, fish images of training data in a third mode using the individual fish identification device. FIG. 6 is a diagram showing, as an example, fish images of training data in a fourth mode using the individual fish identification device. In FIGS. 2 to 6, FIG. A shows an example of a fish image, and FIG. B shows a schematic diagram of a fish image obtained by tracing the outline of a fish captured in the fish image shown in FIG. A. FIG. 7 is a diagram showing, as an example, fish images of training data in a fifth mode using the individual fish identification device. In Figure 7, Figure A shows an example of a fish image, Figure B shows a schematic diagram of a fish image obtained by tracing the outline of a fish reflected in the fish image shown in Figure A, Figure C shows an example of a fish image, and Figure D shows a schematic diagram of a fish image obtained by tracing the outline of a fish reflected in the fish image shown in Figure C. Figure A shows one side of the fish, and Figure C shows the other side of the fish. Figure 8 shows an example of a fish information table.

[0028] The fish individual identification device 1000 in the embodiment includes, for example, a fish image acquisition unit 1, a control processing unit 2, an input unit 3, an output unit 4, an interface unit (IF unit) 5, and a memory unit 6, as shown in FIG.

[0029] The fish individual identification device 1000 is a device that individually identifies fish Ob that have patterns on their body surfaces. The fish Ob to be identified may be of any type as long as they have patterns on their body surfaces. The pattern may include, for example, stripes. Alternatively, the pattern may include, for example, vertical stripes. Alternatively, the pattern may include, for example, horizontal stripes. Here, in the case of fish, the vertical direction is the direction of the body length. That is, when the fish is held head-up and tail-fin-down, the direction of the stripes is determined, with the up-down direction being the vertical direction and the left-right direction being the horizontal direction. From the perspective of being favorable for the research, the fish is a zebrafish (scientific name: Danio rerio).

[0030] Such a fish individual identification device 1000 is used, for example, in a drug administration experiment, and a drug administration experiment system SY for performing this drug administration experiment includes, for example, a pre-administration tank TB for containing the fish Ob before drug administration, an imaging tank TS for containing the fish Ob to generate fish images, a holding device HD for holding the fish Ob to administer a drug, a post-administration tank TA for containing the fish Ob to raise after drug administration, and the fish individual identification device 1000, as shown in Fig. 2. The imaging tank TS has a hollow rectangular parallelepiped shape that is relatively long in one direction, and its top surface is provided with an introduction port EN for introducing the fish Ob and an extraction port EX for removing the fish Ob. When imaging the fish Ob, the imaging tank TS is filled with water all the way to the top to prevent reflections from the water surface. 2, reflecting mirrors ML and MR are arranged on each side of the imaging tank TS at an angle of 45 degrees so that the imaging unit 1, which is an example of the fish image acquisition unit 1, can capture images of the back (top) of the fish Ob, one side, and the other side in a single image. The reflecting mirrors ML and MR are arranged so that the normals to the reflective surfaces of the reflecting mirrors ML and MR intersect at 90 degrees with the optical axis of the imaging unit 1. This makes the one side and the other side of the fish Ob visible from the optical axis direction of the imaging unit 1, and the imaging unit 1 can capture images of the back, one side, and the other side of the fish Ob in a single image.

[0031] Returning to FIG. 1 , the fish image acquisition unit 1 is connected to the control processing unit 2 and acquires an image (fish image) of the body surface of the fish Ob under the control of the control processing unit 2. In the example shown in FIG. 2 , the fish image acquisition unit 1 is an example of the image acquisition unit 1 that generates images, as described above. The image acquisition unit 1 may be configured with, for example, a monochrome digital camera, but in this embodiment, it is configured with a color digital camera. Note that the fish image acquisition unit 1 may also be configured with, for example, an interface circuit that inputs and outputs data to and from an external device. The external device may be a storage medium, such as a USB (Universal Serial Bus) memory or an SD card (registered trademark), that stores the fish image. Alternatively, the external device may be a drive device that reads data from a storage medium, such as a CD-ROM (Compact Disc Read Only Memory), a CD-R (Compact Disc Recordable), a DVD-ROM (Digital Versatile Disc Read Only Memory), or a DVD-R (Digital Versatile Disc Recordable) that stores the fish image. Alternatively, the fish image acquisition unit 1 may be, for example, a communication interface circuit that transmits and receives communication signals to and from an external device, and the external device may be a server device that manages fish images and is connected to the communication interface circuit via a network (WAN (Wide Area Network, including a public communication network)) or LAN (Local Area Network). Here, when the fish image acquisition unit 1 is an interface circuit or a communication interface circuit, the fish image acquisition unit 1 may also serve as the IF unit 5 (i.e., the IF unit 5 may be used as the fish image acquisition unit 1).

[0032] In the above description, one imaging unit 1 captures images of the back, one side, and the other side of the fish Ob to generate one fish image, but each fish image may be generated individually by an imaging unit (first imaging unit) that generates an image of the back of the fish Ob, an imaging unit (second imaging unit) that generates an image of one side of the fish Ob, and an imaging unit (third imaging unit) that generates an image of the other side of the fish Ob.

[0033] The input unit 3 is connected to the control processing unit 2 and is a device that inputs various commands, such as a command to start machine learning or a command to start individual identification, as well as various data required to operate the fish individual identification device 1000, such as the experiment name and the name of the individual fish, to the fish individual identification device 1000, and is, for example, a keyboard, a mouse, and multiple input switches to which predetermined functions are assigned. The output unit 4 is connected to the control processing unit 2 and is a device that outputs the commands and data, fish images, individual identification results, etc. input from the input unit 3 under the control of the control processing unit 2, and is, for example, a display device such as a CRT display, LCD (liquid crystal display), or organic EL display, or a printing device such as a printer.

[0034] The input unit 3 and the output unit 4 may be configured as a touch panel. In this touch panel configuration, the input unit 3 is a position input device, such as a resistive or capacitive type, that detects and inputs an operation position, and the output unit 4 is a display device. In this touch panel, a position input device is provided on the display surface of the display device, and one or more input content candidates that can be input are displayed on the display device. When a user touches the display position showing the input content they want to input, the position is detected by the position input device, and the display content displayed at the detected position is input to the fish individual identification device 1000 as the user's operation input content. Such a touch panel makes it easy for users to intuitively understand input operations, providing a fish individual identification device 1000 that is easy for users to use.

[0035] The IF unit 5 is connected to the control processing unit 2 and is a circuit that inputs and outputs data to and from, for example, an external device under the control of the control processing unit 2, and is, for example, an interface circuit for RS-232C, which is a serial communication method, an interface circuit using the Bluetooth (registered trademark) standard, an interface circuit using the USB standard, etc. The IF unit 5 may also be, for example, a communication interface circuit that transmits and receives communication signals to and from an external device, such as a data communication card or a communication interface circuit conforming to the IEEE802.11 standard, etc.

[0036] The memory unit 6 is connected to the control processing unit 2 and is a circuit that stores various predetermined programs and various predetermined data under the control of the control processing unit 2. The various predetermined programs include, for example, a control processing program, which includes, for example, a control program, a machine learning program, a fish individual identification program, and a recording processing program. The control program is a program that controls each of the units 1, 3 to 6 of the fish individual identification device 1000 according to the function of each unit. The machine learning program is a program that uses training data to train a machine learning model (fish individual identification model) that identifies individual fish based on images. The fish individual identification program is a program that uses the machine learning model to identify individual fish captured in images acquired by the fish image acquisition unit 1. The recording processing program is a program that stores predetermined information (fish information) about the fish Ob in the memory unit 6. The various predetermined data include, for example, data necessary for executing each of these programs, such as training data, fish images acquired by the fish image acquisition unit 1, fish information, and each screen.

[0037] Such storage unit 6 includes, for example, a ROM (Read Only Memory) which is a nonvolatile storage element, an EEPROM (Electrically Erasable Programmable Read Only Memory) which is a rewritable nonvolatile storage element, etc. The storage unit 6 also includes a RAM (Random Access Memory) which serves as a so-called working memory of the control processing unit 2 and stores data generated during execution of the predetermined program, etc. The storage unit 6 may also be configured to include a hard disk device with a relatively large storage capacity.

[0038] The storage unit 6 functionally includes a learning data storage unit 61 and a fish information storage unit 62 for storing learning data and fish information, respectively.

[0039] The training data storage unit 61 stores a training data set including a plurality of training data. The training data is supervised training data including a fish image of a fish Ob and an identifier of the fish appearing in the fish image. The fish identifier (ID) is a code (a symbol, one or more characters, etc.) for specifying and identifying the fish Ob, such as a fish name assigned to each fish Ob. A first-mode fish image PCa in the training data is, for example, a color image generated by the imaging unit 1, as shown in FIG. 3, and the back (top) of the fish Ob, one side, and the other side are captured in the fish image PCa. Alternatively, a second-mode fish image PCb in the training data is, for example, a color image generated by the imaging unit 1, as shown in FIG. 4, and the one side and the other side of the fish Ob are captured in the fish image PCb. In other words, the fish image PCb is an image obtained by removing a rectangular image area that includes the back of the fish Ob from the fish image PCa. Alternatively, for example, a fish image PCc of a third mode in the training data is a fish image with a monochromatic background (e.g., black or white) as shown in FIG. 5, and the back (top) of the fish Ob, one side, and the other side are captured in the fish image PCc. Such a fish image PCc may be created, for example, by image processing that fills the background of the fish image generated by the imaging unit 1 with a monochromatic color, or by placing a monochromatic background member against the fish Ob and generating the fish image with the imaging unit 1. Alternatively, for example, a fish image PCd of a fourth mode in the training data is a fish image with a monochromatic background as shown in FIG. 6, and one side and the other side of the fish Ob are captured in the fish image PCd. As in these first to fourth modes, the image of one side of the fish Ob and the image of the other side of the fish Ob may be composed of a single fish image PCa to PCd. Alternatively, for example, the fifth type of fish image PCe in the learning data is a pair of fish images PCe1 and PCe2 with a monochrome background, as shown in Figure 7, where one side of the fish Ob is reflected in one fish image PCe1, and the other fish image PCe2 is reflected in the other side of the fish Ob.As in this fifth embodiment, the image of one side of the fish Ob and the image of the other side of the fish Ob may be configured as separate images PCe1 and PCe2.

[0040] In this embodiment, learning data is prepared for at least each fish Ob to be identified, and multiple learning data are prepared for one individual fish Ob.

[0041] The fish information storage unit 62 stores fish information. The fish information is predetermined information about the fish Ob, and is configured with appropriate items (attributes) depending on the intended use of the fish Ob, etc. For example, in follow-up observation of drug administration, the fish information is configured with the ID, weight, body length, and measurement date of the fish Ob, etc.

[0042] In this embodiment, this fish information is stored in table form in the fish information storage unit 62. The fish information table LT for registering this fish information includes, for example, an individual ID field 621 for registering the ID of the fish Ob, a date field 622 for registering the measurement date on which the weight and body length of the fish Ob having the ID registered in the individual ID field 621 were measured, a weight field 623 for registering the weight measured on the measurement date registered in the date field 622 of the fish Ob having the ID registered in the individual ID field 621, and a body length field 624 for registering the body length measured on the measurement date registered in the date field 622 of the fish Ob having the ID registered in the individual ID field 621, and has a record for each ID and measurement date, as shown in FIG.

[0043] The control processing unit 2 is a circuit that controls each of the units 1, 3 to 6 of the fish individual identification device 1000 according to the function of each unit, trains the machine learning model by machine learning, and identifies each individual fish Ob. The control processing unit 2 is configured, for example, with a CPU (Central Processing Unit) and its peripheral circuits. By executing the control processing program, the control processing unit 2 is functionally configured to include a control unit 21, a machine learning unit 22, a fish individual identification unit 23, and a recording processing unit 24.

[0044] The control unit 21 controls each of the units 1, 3 to 6 of the fish individual identification device 1000 according to the function of each unit, and is in charge of overall control of the fish individual identification device 1000.

[0045] The machine learning unit 22 uses training data to train an untrained machine learning model that identifies individual fish based on an image. In this embodiment, the machine learning unit 22 trains the machine learning model using a training dataset stored in the training data storage unit 61. In this embodiment, when a fish image PC is input, the machine learning model identifies each individual fish that appears in the fish image PC and outputs the ID and confidence level [%] of the fish Ob. For example, a deep learning convolutional neural network (CNN) is used as the machine learning model. More specifically, in this embodiment, the well-known VGG16, which is a type of CNN and has relatively high object recognition accuracy, is used. The VGG16 includes 13 convolutional layers (first to thirteenth convolutional layers) and three connection layers (first to third connection layers). More specifically, between the first and second convolutional layers and the third and fourth convolutional layers, a first pooling layer (max The neural network includes a second pooling layer for the third and fourth convolutional layers between the third and fourth convolutional layers and the fifth to seventh convolutional layers, a third pooling layer for the fifth to seventh convolutional layers between the fifth to seventh convolutional layers and the eighth to tenth convolutional layers, and a fourth pooling layer for the eighth to tenth convolutional layers between the eighth to tenth convolutional layers and the eleventh to thirteenth convolutional layers. A softmax algorithm is used in the final layer.

[0046] In this embodiment, the machine learning unit 22 further creates a training data set before the machine learning and stores the created training data set in the training data storage unit 61. More specifically, in this embodiment, for each fish Ob to be identified, the machine learning unit 22 captures a plurality of images of the fish Ob using the imaging unit 1, generates, for example, a plurality of fifth-state fish images PCe (PCe1, PCe2) from each of the captured fish images PC, and associates the ID of the fish Ob with each of the generated fifth-state fish images PCe to create a plurality of training data for the fish Ob, and stores the created plurality of training data for each of the fish Ob to be identified as a training data set in the training data storage unit 61. Here, fish images PC generated by the imaging unit 1 that are inappropriate as fish images PC of the training data, for example, because the fish Ob is captured with a bent body part, are discarded and excluded. It should be noted that a plurality of fish images PCe may be generated and padded from the fish image PC generated by the imaging unit 1 by, for example, changing the position of the image area in which the fish Ob appears.

[0047] The individual fish identification unit 23 uses a machine learning model that identifies individual fish based on an image, thereby identifying each individual fish Ob appearing in an image acquired by the fish image acquisition unit 1. The machine learning model used by the individual fish identification unit 23 is a machine learning model trained by the machine learning unit 22. More specifically, an individual fish Ob appearing in the fish image PC may be identified using a machine learning model trained based on a single fish image PC acquired by the imaging unit 1. However, in this embodiment, in order to more accurately identify individuals, the individual fish identification unit 23 identifies each individual fish Ob appearing in the fish image PC using a machine learning model trained based on multiple fish images PC acquired by the imaging unit 1. More specifically, the individual fish identification unit 23 includes a VGG16 (not shown) as a machine learning model trained by machine learning, and a confidence factor calculation unit (not shown) that calculates a final confidence factor for each ID based on the ID and each confidence factor of the fish Ob identified in each of the multiple fish images PC. The individual fish identification unit 23 first generates and acquires multiple fish images PC for one individual fish using the imaging unit 1. Next, the individual fish identification unit 23 calculates the ID of the fish Ob and its certainty factor as output for each of the multiple fish images PC using the VGG 16. Then, the individual fish identification unit 23 calculates, using the certainty factor calculation unit, the average value of the certainty factors corresponding to the ID of each of the fish Ob calculated as the final certainty factor for each of the calculated IDs of the fish Ob. Therefore, in this embodiment, the individual fish identification unit 23 outputs the ID of the fish Ob calculated in this manner and its final certainty factor.

[0048] The recording processing unit 24 stores the fish information in the fish information storage unit 62 of the storage unit 6. In this embodiment, the recording processing unit 24 functions as an individual name correction unit that corrects the individual name output from the individual fish identification unit 23, which in this embodiment is the ID of the fish Ob. More specifically, when the machine learning unit 22 creates a learning dataset as described above, the recording processing unit 24 assigns an ID to each individual fish Ob, calculates the weight and body length of the fish Ob, creates a new record in the fish information table LT stored in the fish information storage unit 62, and registers the assigned ID, the calculated weight, the calculated body length, and the measurement date as fish information for the fish Ob in the individual ID field 621, weight field 623, body length field 624, and date field 622 of the newly created record, respectively. As a result, for example, an initial value for starting follow-up observation of drug administration is stored (recorded) for each individual fish in the fish information storage unit 62. To register the measurement date, in this embodiment, the recording processing unit 24 has a clock function that keeps track of the date. The weight and body length measurements may be obtained manually by an operator (user). However, in this embodiment, the weight and body length of the fish Ob are obtained based on the fish image PC. For example, the recording processing unit 24 first determines the body part by removing the image regions of the dorsal fin, pelvic fin, and caudal fin using machine-learned semantic segmentation and extracting the image region of the body part of the fish Ob. Next, the recording processing unit 24 obtains a rectangular region surrounding the dorsal body part and obtains the elongated length of the obtained rectangular region as the body length of the fish Ob. A model (fish weight estimation model) for obtaining the weight of the fish Ob is then generated in advance based on the extracted image of the body part of the fish Ob. The recording processing unit 24 then obtains the weight of the fish Ob based on the extracted image of the body part of the fish Ob using this fish weight estimation model. The fish weight estimation model may be, for example, a machine learning model, and was obtained by linear multiple regression analysis in which the objective variable was the weight of the fish Ob and the explanatory variables were the area of ​​the back of the body part, the area of ​​one side and the other side of the body part, and the inclination of the body relative to the up and down direction of the fish image PC when viewed from the back (top).Then, when the machine learning of the machine learning model is completed and individual identification is performed, the recording processing unit 24 calculates the weight and body length of the fish Ob based on the fish image PC used for the individual identification. Next, the recording processing unit 24 creates a new record in the fish information table LT stored in the fish information storage unit 62, and registers the individual ID field 621, weight field 623, body length field 624, and date field 622 of the newly created record as fish information for the fish Ob captured in the fish image PC used for the individual identification. For the individual name, if the individual name output from the fish individual identification unit 23 (in this example, the ID of the fish Ob) is corrected, the corrected individual name is used. On the other hand, if the individual name output from the fish individual identification unit 23 is not corrected, the individual name output from the fish individual identification unit 23 is used. In this way, fish information during follow-up observation of, for example, drug administration, is stored (recorded) in the fish information storage unit 62.

[0049] The control processing unit 2, input unit 3, output unit 4, IF unit 5, and memory unit 6 can be configured, for example, by a desktop or notebook computer. As described above, if the fish image acquisition unit 1 is an interface circuit or a communication interface circuit, the fish image acquisition unit 1 can also serve as the IF unit 5, and therefore the fish image acquisition unit 1 and the entire unit can be configured by a computer.

[0050] Next, the operation of this embodiment will be described. Fig. 9 is a flowchart showing the operation of the fish individual identification device related to machine learning. Figs. 10 to 18 are diagrams showing, as an example, first to ninth screens displayed on the fish individual identification device. Fig. 19 is a flowchart showing the operation of the fish individual identification device related to individual identification. Figs. 20 to 24 are diagrams showing, as an example, tenth to fourteenth screens displayed on the fish individual identification device.

[0051] When the power is turned on, the fish individual identification device 1000 configured as described above initializes each necessary part and starts operation. The control processing unit 2 functionally configures a control unit 21, a machine learning unit 22, a fish individual identification unit 23, and a recording processing unit 24 by executing the control processing program.

[0052] First, the operation of the fish individual identification device 1000 regarding machine learning will be described. In machine learning, first, to create a learning dataset, as shown in FIG. 9 , the fish individual identification device 1000 acquires fish images PC using the imaging unit 1 via the machine learning unit 22 of the control processing unit 2 (S11, fish image acquisition step). More specifically, the control unit 21 of the control processing unit 2 displays, for example, a first screen (fish individual identification top screen) SC1 shown in FIG. 10 on the output unit 4. The first screen SC1 includes an "imaging" button 41 that instructs the fish individual identification device 1000 to capture an image using the imaging unit 1, and a "view data" button 42 that instructs the fish individual identification device 1000 to view fish information. When the view data button 42 is operated by an operator, the recording processing unit 24 of the control processing unit 2 reads fish information registered in the fish information table LT from the fish information storage unit 62 and outputs the read fish information to the output unit 4, for example, in the form of a list. Here, the "imaging" button 41 is operated by the operator. When the "Image Capture" button 41 is operated, the control unit 21 displays, for example, the second screen (learning individual identification selection screen) SC2 shown in FIG. 11 on the output unit 4. The second screen SC2 includes a "First Time (Learning)" button 43 that instructs the fish individual identification device 1000 to create a learning dataset and perform machine learning on an unmachine-learned machine learning model, a "Second Time or Later (Discrimination)" button 44 that instructs the fish individual identification device 1000 to perform individual identification and acquire fish information, and a "Return to Top" button 45 that instructs the fish individual identification device 1000 to display the first screen SC. When the "Return to Top" button 45 is operated by an operator, the first screen SC1 is displayed on the output unit 4. Here, the "First Time (Learning)" button 43 is operated by the operator. When the "First Time (Learning)" button 43 is operated by an operator, the control unit 21 displays, for example, the third screen (case name input screen) SC3 shown in FIG. 12 on the output unit 4. The third screen SC3 includes a case name display area 46 that displays the case name entered in the input unit 3, and a "Next" button 47 that instructs the fish individual identification device 1000 to transition to the next screen. Here, the case name "XX Case" is entered by the operator, and the "Next" button 47 is operated by the operator.When the "Next" button 47 is operated, the control unit 21 displays, for example, a fourth screen (first imaging start instruction screen) SC4 shown in FIG. 13 on the output unit 4. The fourth screen SC3 includes an ID display area 48 that displays the ID of the fish Ob input via the input unit 3, and a "Start Imaging" button 49 that instructs the fish individual identification device 1000 to start imaging. The ID of the fish Ob may be automatically generated by the control processing unit 2, but in this embodiment, it is generated and input by the operator. In the example shown in FIG. 13, the video (monitor video) being captured by the imaging unit 1 is also displayed on the fourth screen SC4. Here, the "Start Imaging" button 49 is operated by the operator. When the "Start Imaging" button 49 is operated, the machine learning unit 22 acquires a fish image PC using the imaging unit 1. While this fish image PC is being acquired, in this embodiment, the control unit 21 displays, for example, a fifth screen (imaging in progress notification screen) SC5 shown in FIG. 14 on the output unit 4. The fifth screen SC5 includes a "Stop Imaging" button 50 that instructs the fish individual identification device 1000 to stop capturing images. When the "Stop Imaging" button 50 is pressed, capturing images is stopped and the first screen SC1 is displayed on the output unit 4. In the example shown in FIG. 13, the fifth screen SC5 displays the progress of the capturing process and displays the fish image PC being captured.

[0053] Next, the fish individual identification device 1000 uses the machine learning unit 22 to determine whether the acquired fish image PC is in a proper orientation to determine whether it can be used as learning data (S12). If the result of this determination is that the orientation is not proper (No), the machine learning unit 22 discards the fish image PC acquired in step S12 (S14) and returns the process to step S11. On the other hand, if the result of the determination is that the orientation is proper (Yes), the machine learning unit 22 then stores the fish image PC acquired in step S12 in the memory unit 6 (S13).

[0054] The determination in process S12 may be made manually by an operator, but in this embodiment, for example, a pattern of appropriate posture is prepared in advance and the determination is made by pattern matching, or by a machine learning model (appropriate posture determination model) that has been trained to determine whether or not the posture is appropriate based on an image, for example.

[0055] Following process S13, the fish individual identification device 1000 uses the recording processing unit 24 of the control processing unit 2 to perform body part determination on the fish image PC acquired in process S12 to determine its weight and body length (S15). Next, the fish individual identification device 1000 uses the recording processing unit 24 to determine the weight and body length of the fish Ob captured in the fish image PC acquired in process S12, and stores the fish information in the fish information storage unit 62 (S16). In this embodiment, the fish information is stored in association with a case name. More specifically, the case name is associated with a fish information table LT in which the fish information is registered.

[0056] Next, the fish individual identification device 1000 uses the machine learning unit 22 to process the fish image PC acquired in process S12 for use as learning data, associates it with the ID of the fish Ob, and stores it in the memory unit 6 (S17). The machine learning unit 22 processes the fish image PC acquired in process S12 into, for example, one of the fish images PCb to PCe of the second to fifth modes, for example, into a fish image PCe of the fifth mode, and stores the processed fish image PCe of the fifth mode in the memory unit 6, associates it with the ID of the fish Ob.

[0057] Next, the fish individual identification device 1000 determines whether a predetermined number (a first number, for example, 300 or 400) of fish images PC have been acquired by the machine learning unit 22 (S18). If the result of this determination is that the predetermined number of fish images PC have not been acquired (No), the fish individual identification device 1000 returns the process to step S11. If the result of the determination is that the predetermined number of fish images PC have been acquired (Yes), the fish individual identification device 1000 stores (saves) the fish images PC processed for the learning data and their IDs that were temporarily stored (saved) in the storage unit 6 in step S17 in the learning data storage unit 61 as learning data for the learning dataset (S19).

[0058] Following step S19, the fish individual identification device 1000 determines, via the machine learning unit 22, whether or not learning data for the learning dataset has been created for all individuals of the fish Ob (S20). This determination is made in response to an instruction input by the operator. The machine learning unit 22 displays, for example, a sixth screen (first determination instruction screen) SC6 shown in FIG. 15 on the output unit 4. The sixth screen SC6 includes a "Take an image of the next individual" button 51 that instructs the fish individual identification device 1000 to create a learning dataset for the next individual, and a "Complete all individuals" button 52 that instructs the fish individual identification device 1000 to complete creation of learning datasets for all individuals. When the "Take an image of the next individual" button 51 is input by the operator, the fish individual identification device 1000 determines that the learning data has not been created for all individuals (No), and returns the process to step S11. On the other hand, when the "All Individuals Completed" button 52 is input by the operator, it is determined that the learning data has been created for all individuals (Yes), and the fish individual identification device 1000 uses the machine learning unit 22 to machine-learn the untrained machine learning model using the learning dataset stored in the training data storage unit 61 (S21, machine learning process), and stores this machine-learned machine learning model in the storage unit 6 as the machine learning model to be used by the fish individual identification unit 23 (S22), and terminates this processing.

[0059] In the above description, process S21 is started automatically, but it may also be started by an operator's instruction. In this case, the machine learning unit 22 displays, for example, a seventh screen (learning instruction screen) SC7 shown in FIG. 16 on the output unit 4. The seventh screen SC7 includes a "Start learning" button 53 that instructs the fish individual identification device 1000 to start machine learning, and a "Return to TOP" button 54 that instructs the individual identification device 1000 to display the first screen SC1.

[0060] Furthermore, during execution of process S22, for example, an eighth screen (learning in progress notification screen) SC8 shown in FIG. 17 may be displayed on the output unit 4, and when the machine learning is completed, for example, a ninth screen (learning completion notification screen) SC9 shown in FIG. 18 may be displayed on the output unit 4. The eighth screen SC8 includes a "Stop Learning" button 55 that instructs the fish individual identification device 1000 to stop learning, and a progress status display bar 56 that displays the progress of the machine learning. When the "Stop Learning" button 55 is operated by an operator, the seventh screen SC7 is displayed on the output unit 4. The ninth screen SC9 includes a "Return to TOP" button 57 that instructs the individual identification device 1000 to display the first screen SC1, and a learning completion display area 58 that displays the completion of the machine learning.

[0061] By operating in this manner, a training dataset is created, which is used to train a machine learning model.

[0062] Next, the operation of the fish individual identification device 1000 regarding individual identification will be described. For individual identification, in FIG. 19 , first, the fish individual identification device 1000 causes the fish individual identification unit 23 of the control processing unit 2 to acquire a fish image PC using the imaging unit 1 (S31, fish image acquisition step). More specifically, the control unit 21 of the control processing unit 2 displays a first screen SC1 on the output unit 4. Here, the "Image Capture" button 41 is operated by the operator. When the "Image Capture" button 41 is operated, the control unit 21 displays a second screen SC2 on the output unit 4. Here, the "Second Time or Later (Discrimination)" button 44 is operated by the operator. When the "Second Time or Later (Discrimination)" button 44 is operated, the control unit 21 displays, for example, a tenth screen (case selection screen) SC10 shown in FIG. 20 on the output unit 4. The tenth screen SC10 is a screen for selecting the case name of the fish information stored in the fish information storage unit 62. The tenth screen SC in the example shown in FIG. 20 displays the following items for each project name in a list format: project name, number of individuals for that project name, and measurement date. The number of individuals for that project name is calculated by counting the number of unique IDs registered in the fish information table LT corresponding to the project name. The measurement date is calculated by extracting the measurement dates registered in the fish information table LT corresponding to the project name for each different date. The tenth screen SC10 includes, for each project name displayed in list format, a "Select" button 71 (71-1, 71-2, 71-3) that instructs the fish individual identification device 1000 to specify that project name; a vertical scroll slider 72-1 that instructs the fish individual identification device 1000 to scroll through the project names displayed in list format; a horizontal scroll slider 72-2 that instructs the fish individual identification device 1000 to scroll through each item displayed in list format; and a "Back" button 73 that instructs the fish individual identification device 1000 to display the previous screen SC. Here, the operator operates the "Select" button 71 (for example, the "Select" button 71-1). When the "Select" button 71 is operated, the control unit 21 displays, for example, an eleventh screen (second imaging start instruction screen) SC11 shown in FIG. 21 on the output unit 4. The eleventh screen SC11 includes a "Start imaging" button 74 that instructs the fish individual identification device 1000 to start imaging.In the example shown in FIG. 21, the eleventh screen SC11 displays a message saying, "When the subject to be imaged is ready, press the image capture start button," as well as a video (monitor video) being captured by the imaging unit 1. When the "image capture start" button 74 is pressed, the fish individual identification unit 23 acquires a fish image PC using the imaging unit 1. While this fish image PC is being acquired, in this embodiment, the control unit 21 displays, for example, a twelfth screen (image capture in progress notification screen) SC12 shown in FIG. 22 on the output unit 4. The twelfth screen SC12 includes a "image capture stop" button 75 that instructs the fish individual identification device 1000 to stop capturing images. In the example shown in FIG. 22, the twelfth screen SC12 displays the progress of the image capture process and displays the fish image PC being captured.

[0063] Next, the fish individual identification device 1000 executes processes S32 to S38 which are similar to the processes S12 to S18 described above.

[0064] In process S39 (fish individual identification process), which is executed when a predetermined number (a second number, for example, 100 or 200) of fish images PC have been acquired as a result of the judgment in process S38, the fish individual identification device 1000 uses the multiple fish images PC acquired in each of the above-mentioned processes in a machine learning model that has been trained by machine learning to identify individual fish Ob captured in the fish images PC using the fish individual identification unit 23, and stores the ID of the identified fish Ob and the final confidence level in the memory unit 6.

[0065] Following step S39, the fish individual identification device 1000 determines whether individual identification has been performed for all of the fish Ob using the fish individual identification unit 23 (S40). This determination is made based on an instruction input by the operator. The fish individual identification unit 23 displays, for example, a thirteenth screen (second determination instruction screen) SC13 shown in FIG. 23 on the output unit 4. The thirteenth screen SC13 includes a "Take image of next individual" button 76 that instructs the fish individual identification device 1000 to perform individual identification for the next individual, and a "Complete all individuals" button 77 that instructs the fish individual identification device 1000 to complete individual identification for all individuals. When the "Take image of next individual" button 76 is input by the operator, the fish individual identification device 1000 determines that the individual identification has not been performed for all of the individuals (No), and returns the process to step S31. On the other hand, when the "All Individuals Completed" button 77 is input by the operator, it is determined that the individual identification has been performed on all individuals (Yes), and the fish individual identification device 1000 executes an individual name correction process to correct the individual name (ID) of the identification result using the recording processing unit 24 (S41, individual name correction process).

[0066] More specifically, in the individual name correction process, the recording processing unit 24 outputs, for example, a fourteenth screen (individual name correction instruction screen) SC14 shown in FIG. 24 to the output unit 4. The fourteenth screen SC14 displays the identification result and is a screen for instructing the fish individual identification device 1000 to correct the individual name (ID) of the identification result. The fourteenth screen SC14 in the example shown in FIG. 24 includes an identification result display area 78 that displays the individual name (ID) of the identification result and its confidence level in a list format for each individual to be identified. The individual name (ID) of the identification result and its confidence level displayed in this identification result display area 78 are displayed for each individual identified by the machine learning model. For example, if the results of individual identification using a machine learning model based on multiple fish images PC for one individual are individual name "A", individual name "B", and individual name "C", the identification results for this one individual to be identified are displayed in the identification result display area 78 as individual name "A" and its certainty, individual name "B" and its certainty, and individual name "C" and its certainty. The fourteenth screen SC14 includes a vertical scroll slider 79-1 that instructs the fish individual identification device 1000 to scroll through the individual fish identification target individuals displayed in a list format, a horizontal scroll slider 79-2 that instructs the fish individual identification device 1000 to scroll through pairs of individual fish names and their confidence levels, a correction content display area 80 that displays correction content entered by the operator using the input unit 3, a "Correction" button 81 that instructs the fish individual identification device 1000 to make corrections using the correction content displayed in the correction content display area 80, and a "Register result" button 82 that instructs the fish individual identification device 1000 to store (save) the fish information as the identification result of the individual identification performed in process S39 or to store (save) the fish information as the identification result after the correction. In the example shown in Figure 24, the correction content display area 80 includes a correction target display area 80-1 that displays the individual fish identification target individuals whose individual names (IDs) are to be corrected, and a corrected individual name display area 80-2 that displays the corrected individual fish name (ID) entered by the operator using the input unit 3.

[0067] When the "Register result" button 82 is operated by the operator, the fish individual identification device 1000 causes the recording processing unit 24 to store (save) the fish information in the fish information storage unit 62 (S42). This process ends. When storing this fish information, if the "Register result" button 82 is operated when the correction content display area 80 is blank, the recording processing unit 24 stores the fish information based on the identification result obtained by the individual identification in process S39 in the fish information storage unit 62. On the other hand, if the "Register result" button 82 is operated with the correction content displayed in the correction content display area 80, the recording processing unit 24 stores the fish information based on the identification result after the correction in the fish information storage unit 62. Note that if the identification results are duplicated (for example, if the identification results for multiple individuals are all "A"), the "Register result" button 82 cannot be operated when the correction content display area 80 is blank.

[0068] By operating in this manner, the fish Ob to be identified is individually identified, and the fish information is stored (saved) in the fish information storage unit 62.

[0069] In the above description, if the individual name (ID) is corrected, the fish image PC with the corrected name and the corrected ID may be added to the training data set as training data, and the machine learning model may be re-trained by the machine learning unit 22 using the training data set to which this training data has been added. In this case, the fourteenth screen SC14 may further be provided with an "update model" button 83, as shown by the dashed line in Figure 24, which instructs the fish individual identification device 1000 to perform the re-training. This allows the confidence level to be restored even if the confidence level decreases as the fish Ob grows over time during follow-up observation.

[0070] As described above, the fish individual identification device 1000 and the fish individual identification method and program implemented therein in this embodiment identify individual fish Ob by using a machine learning model that identifies individual fish based on images. Therefore, this makes it possible to provide the fish individual identification device 1000, fish individual identification method, and fish individual identification program that identify individual fish using a method different from human identification methods.

[0071] According to this embodiment, it is possible to provide a fish individual identification device 1000, a fish individual identification method, and a fish individual identification program that generate a machine-learned machine learning model through machine learning.

[0072] When the above-mentioned fish individual identification device 1000, fish individual identification method, and fish individual identification program train the machine learning model using training data including images of the fish with a monochrome background, the noisy background can be removed and machine learning can be performed on just the fish, allowing the machine learning model to be trained to more accurately identify individuals.

[0073] When the above-mentioned fish individual identification device 1000, fish individual identification method, and fish individual identification program machine-learn the machine learning model using training data including an image of one side of the fish and an image of the other side of the fish, the machine learning model can be machine-learned using patterns on both sides, allowing for more accurate individual identification.

[0074] The fish individual identification device 1000, the fish individual identification method, and the fish individual identification program can correct the individual name even if an error is made in the individual identification.

[0075] In the above-described embodiment, the fish individual identification device 1000 may further include a classification unit 25 that classifies multiple fish Ob into groups by using the pseudospace of the machine learning model, as indicated by the dashed line in FIG. 1 (first modified embodiment). In the example shown in FIG. 1, the storage unit 6 further stores, as a program included in the control processing program, a classification program that classifies multiple fish Ob into groups by using the pseudospace of the machine learning model, and the classification unit 25 is functionally configured in the control processing unit 2. Such fish individual identification device 1000, fish individual identification method, and fish individual identification program classify multiple fish Ob into groups, making it easier to identify individual fish by selecting fish Ob from each group and reducing erroneous individual identification decisions.

[0076] Fig. 25 is a diagram illustrating a fish individual identification device in a first modified embodiment. Fig. 25A shows classification results in a pseudo-space consisting of the nodes in the first combining layer of machine-learned VGG16 as an example of the machine learning model, and Fig. 25B shows classification results in a pseudo-space consisting of the nodes in the first pooling layer of machine-learned VGG16 as an example of the machine learning model. For convenience of drawing, in Figs. 25A and 25B, the n-dimensional pseudo-space is flattened to two dimensions.

[0077] 25A and 25B show the results obtained by using 288 fish images PC (e.g., unprocessed fish images PC of one side of a fish, approximately 30 fish images PC per fish) of nine fish (fish) in a machine-learned model. Comparing FIGS. 25A and 25B reveals that near the first stage of VGG16 (near the input side, in this example, the first pooling layer), the nine fish cannot be classified in the pseudospace. However, near the final stage of VGG16 (near the output side, in this example, the first combined layer), the nine fish are roughly grouped into nine groups Gr1 to Gr9, allowing the nine fish to be classified. Therefore, for example, by using the fish image PC of a tenth fish in the machine-learned model to determine which of the nine groups Gr1 to Gr9 it belongs to, the group Gr to which the tenth fish belongs can be determined. Of course, there may be cases where the tenth fish does not belong to any of the nine groups Gr1 to Gr9. The determination can be performed using, for example, a known clustering method. For example, the classification unit 25 determines the center position of each group Gr, and calculates the distance between the result obtained by using the fish image PC of the tenth individual in a machine-learned model and each center position of each group Gr. If any of these distances is equal to or less than a predetermined threshold (group determination threshold), the classification unit 25 determines that the tenth individual belongs to the group Gr corresponding to that distance. Note that if multiple distances are equal to or less than the group determination threshold, the classification unit 25 may determine that the tenth individual belongs to the group Gr corresponding to the shortest distance. The operator selects a fish Ob to be identified (observed) by referring to the classification results of the classification unit 25. More specifically, the pseudo-space is divided into multiple sections of a predetermined size, and the fish Ob to be identified are selected so that the classification results of the classification unit 25 are distributed among the multiple sections. In addition, if the classification result of the fish Ob to be selected is located on the periphery of the section, and the classification result of another fish Ob that has already been selected is located at a distance less than a predetermined threshold (selection judgment threshold), the fish Ob to be selected will not be selected. By adjusting the size of the section, the ease of individual identification can be adjusted.

[0078] In the above-described embodiment (including the first variant), the fish individual identification device 1000 may further include a transmission processing unit 26 that transmits at least one of the training data and the machine learning model to a management server device that manages at least one of the training data and the machine learning model in a shareable manner, as indicated by the dashed line in FIG. 1 (second variant). In the example shown in FIG. 1, the storage unit 6 further stores, as a program included in the control processing program, a transmission processing program that transmits at least one of the training data and the machine learning model to a management server device that manages at least one of the training data and the machine learning model in a shareable manner, and the transmission processing unit 26 is functionally configured in the control processing unit 2. Such fish individual identification device 1000, fish individual identification method, and fish individual identification program can utilize at least one of the training data and the machine learning model that can be shared, thereby enabling the machine learning model to have high accuracy.

[0079] Fig. 26 is a diagram for explaining a fish individual identification device in a second modified embodiment. In this second modified embodiment, for example, as shown in Fig. 26, a data model sharing system MS is configured including a management server device SV that manages at least one of the training data and the machine learning model in a shareable manner, and a plurality of fish individual identification devices 1000 (1000-1, 1000-2, 1000-3, ..., 1000-n) in the second modified embodiment that are communicatively connected to the management server device SV via a communication network NW.

[0080] For example, in the fish individual identification device 1000-1, when an operator issues an instruction to transmit training data via the input unit 3, the transmission processing unit 26 generates a communication signal (training data transmission signal) containing the training data set stored in the training data storage unit 61 and transmits the generated training data transmission signal to the management server device VS via the IF unit 5 and the communication network NW. When the management server device SV receives the training data transmission signal, it stores (preserves) the training data set contained in the received training data transmission signal and manages it so that it can be shared. When the management server device SV receives a communication signal (training data request signal) requesting the transmission of a training data set from, for example, the fish individual identification device 1000-2, it generates and returns a communication signal (training data reply signal) containing the training data set it manages. Upon receiving this training data reply signal, the fish individual identification device 1000-2 retrains its own machine learning model, which has already been trained by machine learning, using the received training data set. This allows the machine learning model of the own device to perform more accurate individual identification. Alternatively, the fish individual identification device 1000-2 performs transfer learning on another machine learning model that has undergone machine learning to identify individuals other than zebrafish using the received training dataset. This allows the other machine learning model to be repurposed as a machine learning model for identifying individual zebrafish, thereby generating a machine learning model for identifying individual zebrafish in a shorter time. Alternatively, the fish individual identification device 1000-2 performs machine learning on multiple fish images PCs generated by itself using the received training dataset to create an unmachined machine learning model. In this case, as long as the received training dataset contains supervised learning data (not all of the training data necessarily needs to be supervised learning data), the unmachined machine learning model can be trained by semi-supervised learning on the multiple fish images PCs generated by itself, even if supervised data is not necessarily attached to the multiple fish images PCs generated by itself.

[0081] Furthermore, for example, federated learning may be performed by mutually transmitting and receiving a machine-learned machine learning model between the management server device SV and the fish individual identification device 1000. First, the management server device SV distributes (transmits) a machine-learned machine learning model to multiple fish individual identification devices 1000. This distributed machine-learned machine learning model may be newly generated by the management server device SV, or may be generated by the fish individual identification device 1000 and transmitted to the management server device SV. Upon receiving this machine-learned machine learning model, the fish individual identification device 1000 performs machine learning using its own learning dataset and updates the distributed machine-learned machine learning model. The fish individual identification device 1000 returns this updated machine-learned machine learning model to the management server device SV. At this time, the updated machine-learned machine learning model itself may be returned, or only the changes made by the update may be returned. Upon receiving this reply, the management server device SV updates the distributed machine-learned machine-learning model with the difference between the distributed machine-learned machine-learning model and the returned machine-learned machine-learning model (i.e., the changes). At this time, replies can be received from multiple fish individual identification devices 1000, making it possible to update with multiple changes. The management server device SV then re-distributes (transmits) this updated machine-learned machine-learning model to multiple fish individual identification devices 1000. This is repeated as necessary. In this way, the fish individual identification device 1000 can obtain a machine-learned machine-learning model that can more accurately identify individuals, even if the training data set has a relatively small amount of training data.

[0082] In order to express the present invention, the present invention has been properly and sufficiently described above through the embodiments with reference to the drawings, but it should be recognized that those skilled in the art can easily change and / or improve the above-mentioned embodiments. Therefore, unless the changes or improvements made by those skilled in the art are at a level that causes departure from the scope of the claims described in the claims, such changes or improvements are interpreted as being included in the scope of the claims. [Explanation of symbols]

[0083] 1000 Fish individual identification device SV management server device 1 Fish image acquisition unit 2. Control processing section 3 Input section 4 Output section 5 Interface section (IF section) 6 Memory section 21 Control Unit 22 Machine Learning Department 23 Fish Identification Department 24 Recording processing unit (including correction unit) 25 Classification Department 26 Transmission processing unit

Claims

1. A method for identifying individual fish having a pattern on a body surface, comprising: a fish image acquisition step of acquiring an image of a body surface of the fish; and a fish individual identification process for identifying individual fish captured in the image acquired in the fish image acquisition process by using a machine learning model that identifies individual fish based on the image. Methods for identifying individual fish.

2. Further comprising a machine learning step of machine learning the machine learning model using training data, The machine learning model used in the fish individual identification process is a machine learning model that has been machine-learned in the machine learning process. The method for identifying individual fish according to claim 1.

3. The training data includes images of the fish with a solid color background. The method for identifying individual fish according to claim 2.

4. The training data includes an image of one side of the fish and an image of the other side of the fish. The method for identifying individual fish according to claim 2.

5. The fish individual identification step further includes outputting an individual name of the identified individual, The method further includes a step of correcting the individual name output in the fish individual identification step. The method for identifying individual fish according to claim 1.

6. The method further comprises the step of classifying the plurality of fish into groups by using the pseudospace of the machine learning model. The method for identifying individual fish according to claim 1.

7. A machine learning process of machine learning the machine learning model using training data; A transmission process for transmitting at least one of the learning data and the machine learning model to a management server device that manages at least one of the learning data and the machine learning model so that they can be shared. The method for identifying individual fish according to claim 1.

8. A fish individual identification device that individually identifies fish having a pattern on a body surface, a fish image acquisition unit for acquiring an image of a body surface of the fish; and a fish individual identification unit that identifies individual fishes captured in the image acquired by the fish image acquisition unit by using a machine learning model that identifies individual fishes based on the image. Fish individual identification device.

9. A machine learning unit that machine-learns the machine learning model using learning data, The machine learning model used in the fish individual identification unit is a machine learning model that has been machine-learned by the machine learning unit. The fish individual identification device according to claim 8.

10. The training data includes images of the fish with a solid color background. The fish individual identification device according to claim 9.

11. The training data includes an image of one side of the fish and an image of the other side of the fish. The fish individual identification device according to claim 9.

12. The fish individual identification unit further outputs an individual name of the identified individual, Further, an individual name correction unit is provided for correcting the individual name output by the individual fish identification unit. The fish individual identification device according to claim 8.

13. A classification processing unit is further provided for classifying the plurality of fish into groups by using the pseudo space of the machine learning model. The fish individual identification device according to claim 8.

14. A machine learning unit that machine-learns the machine learning model using training data; A transmission processing unit that transmits at least one of the learning data and the machine learning model to a management server device that manages at least one of the learning data and the machine learning model in a shareable manner. The fish individual identification device according to claim 8.

15. A fish individual identification program for causing a computer to function as the fish individual identification device according to any one of claims 8 to 14.

Images