Fish individual identification method, device and program

Abstract

To provide a fish individual identification method, a fish individual identification device, and a fish individual identification program for identifying a fish for each individual by using a method different from an identification method of a human.SOLUTION: A fish individual identification device of the present invention is a device that identifies a fish including a pattern on a body surface, for each individual. The fish individual identification device includes: a fish image acquisition unit 1 that acquires an image of the body surface of the fish; and a fish individual identification unit 23 that identifies a fish appearing in the image acquired by the fish image acquisition unit 1 for each individual by using a machine learning model that identifies a fish for each individual on the basis of an image.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a fish individual identification method, a fish individual identification device, and a fish individual identification program for identifying individual fish having a pattern on the body surface. [Background technology]

[0002] Conventionally, living organisms have been used in medical research, biological research, and the like. Zebrafish (scientific name: Danio rerio) are sometimes used as the living organisms from the viewpoints favorable for the research, such as ease of rearing, fecundity, short generation cycle, and ease of embryo observation. The research requires observation of individual individuals, and therefore individual identification is necessary. Manual individual identification is time-consuming and laborious, and therefore automation is desirable. One such technique is the individual identification method disclosed in Patent Document 1, for example.

[0003] The method for identifying individual zebrafish disclosed in Patent Document 1 is a method for identifying multiple zebrafish (Danio rerio) swimming in an aquarium, which involves taking a side image of each individual zebrafish from the side of the aquarium to obtain side image data, determining characteristics of the stripe patterns on the anal and caudal fins of each individual from the obtained side image data of each individual to obtain characteristic data for each individual, and identifying each individual from the multiple zebrafish based on the obtained characteristic data. In this method for identifying individual zebrafish, the characteristic data on the stripe patterns on the anal and caudal fins is at least one data item selected from the group consisting of whether or not stripes branch, whether or not stripes are separated, the number of stripes, whether or not stripes are curved, and whether or not there is a change in stripe width. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2016-253724 A (Patent No. 6679474 A) Summary of the Invention [Problem to be solved by the invention]

[0005] The method for identifying individual zebrafish disclosed in Patent Document 1 is an automation of the conventional method for identifying stripe patterns by humans through image generation and image processing of the images, and individual identification may not be possible based only on the presence or absence of branching of stripes, the presence or absence of division of stripes, the number of stripes, the presence or absence of curvature of stripes, and the presence or absence of change in stripe width. In particular, since the stripe patterns of genetically close individuals are similar, it is not easy to identify individuals, and there is a risk of erroneous identification.

[0006] The present invention has been made in consideration of the above-mentioned circumstances, and its purpose is to provide a fish individual identification method, a fish individual identification device, and an individual fish identification program that identify individual fish using a technique different from human identification methods. [Means for solving the problem]

[0007] As a result of various studies, the inventors of the present invention have found that the above object can be achieved by the present invention described below. That is, a fish individual identification method according to one aspect of the present invention is a method for identifying fish having a pattern on its body surface, and includes a fish image acquisition step of acquiring an image of the body surface of the fish, and a fish individual identification step of identifying fish appearing in the image acquired in the fish image acquisition step by using a machine learning model that identifies fish individually based on the image. A fish individual identification device according to another aspect of the present invention is a device for identifying fish having a pattern on its body surface, and includes a fish image acquisition unit that acquires an image of the body surface of the fish, and a fish individual identification unit that identifies fish appearing in the image acquired by the fish image acquisition unit by using a machine learning model that identifies fish individually based on the image. Preferably, in each of the above-mentioned fish individual identification method and fish individual identification device, the pattern includes a striped pattern. Preferably, in each of the above-mentioned fish individual identification method and fish individual identification device, the pattern includes a vertical striped pattern. Preferably, in the above-mentioned individual fish identification method and individual fish identification device, the pattern includes horizontal stripes. Preferably, in the above-mentioned individual fish identification method and individual fish identification device, the fish is a zebrafish.

[0008] Such an individual fish identification method and individual fish identification device uses a machine learning model that identifies individual fish based on images to identify each individual fish, thereby providing an individual fish identification method and individual fish identification device that identify each individual fish using a method different from human identification methods.

[0009] In another aspect, the above-mentioned individual fish identification method further includes a machine learning step of machine learning the machine learning model with training data, and the machine learning model used in the individual fish identification step is a machine learning model that has been machine-learned in the machine learning step.In another aspect, the above-mentioned individual fish identification device further includes a machine learning unit that machine-learns the machine learning model with training data, and the machine learning model used in the individual fish identification unit is a machine learning model that has been machine-learned in the machine learning unit.Preferably, in each of the above-mentioned individual fish identification method and individual fish identification device, the training data is supervised training data that includes an image in which a fish is captured and an identifier (e.g., an individual name, etc.) for specifying and identifying the fish captured in the image as a supervised data.

[0010] This makes it possible to provide a fish individual identification method and a fish individual identification device that generate a machine-learned machine learning model through machine learning.

[0011] In another aspect, in the above-mentioned fish individual identification method, the training data includes images of the fish with a monochromatic background. In another aspect, in the above-mentioned fish individual identification device, the training data includes images of the fish with a monochromatic background.

[0012] Such an individual fish identification method and individual fish identification device trains the machine learning model using learning data including images of the fish with a monochrome background, so that the noisy background can be removed and only the fish can be trained, allowing the machine learning model to be trained to more accurately identify individuals.

[0013] In another aspect, in these individual fish identification methods, the learning data includes an image of one side of the fish and an image of the other side of the fish. In another aspect, in these individual fish identification devices, the learning data includes an image of one side of the fish and an image of the other side of the fish. Preferably, in each of the individual fish identification methods and individual fish identification devices, the image of one side of the fish and the image of the other side of the fish are composed of a single image. Preferably, in each of the individual fish identification methods and individual fish identification devices, the image of one side of the fish and the image of the other side of the fish are composed of separate images.

[0014] Such an individual fish identification method and individual fish identification device trains the machine learning model using learning data including an image of one side of the fish and an image of the other side of the fish, so that machine learning can be performed on patterns on both sides, allowing the machine learning model to be trained to more accurately identify individuals.

[0015] In another aspect, in these above-mentioned individual fish identification methods, the fish individual identification step further includes an individual name correction step of outputting an individual name of the identified individual and correcting the individual name output in the fish individual identification step.In another aspect, in these above-mentioned individual fish identification devices, the fish individual identification unit further includes an individual name correction unit of outputting an individual name of the identified individual and correcting the individual name output in the fish individual identification unit.

[0016] Such a method and device for identifying individual fish can correct the name of the individual fish even if an error is made in the individual identification.

[0017] In another aspect, the above-mentioned fish individual identification method further includes a classification step of classifying the plurality of fish into groups by using the pseudo space of the machine learning model. In another aspect, the above-mentioned fish individual identification device further includes a classification processing unit that classifies the plurality of fish into groups by using the pseudo space of the machine learning model.

[0018] Such an individual fish identification method and individual fish identification device classify multiple fish into groups, so that by selecting fish from each group, individual identification becomes easier and it is possible to reduce erroneous individual identification decisions.

[0019] In another aspect, the above-mentioned fish individual identification methods further include a machine learning step of machine learning the machine learning model using training data, and a transmission step of transmitting at least one of the training data and the machine learning model to a management server device that manages at least one of the training data and the machine learning model in a shareable manner.In another aspect, the above-mentioned fish individual identification devices further include a machine learning unit that machine learns the machine learning model using training data, and a transmission processing unit that transmits at least one of the training data and the machine learning model to a management server device that manages at least one of the training data and the machine learning model in a shareable manner.

[0020] Such an individual fish identification method and individual fish identification device can utilize at least one of the training data and the machine learning model, which are shareable, thereby enabling the accuracy of the machine learning model to be improved.

[0021] A fish individual identification program according to another aspect of the present invention is a program for causing a computer to function as any one of the above-mentioned fish individual identification devices.

[0022] This makes it possible to provide an individual fish identification program, which has the same operational effects as the above-mentioned individual fish identification devices. Effect of the Invention

[0023] According to the present invention, it is possible to provide an individual fish identification method, an individual fish identification device, and an individual fish identification program that identify individual fish using a technique different from human identification methods. [Brief description of the drawings]

[0024] [Figure 1] 1 is a block diagram showing a configuration of a fish individual identification device according to an embodiment. [Diagram 2] FIG. 2 is a schematic diagram showing the configuration of a drug administration experiment system using the fish individual identification device. [Diagram 3] FIG. 2 is a diagram showing, as an example, fish images of learning data in a first aspect using the fish individual identification device. [Figure 4] FIG. 11 is a diagram showing, as an example, fish images of learning data in a second aspect using the fish individual identification device. [Diagram 5] FIG. 11 is a diagram showing, as an example, fish images of learning data in a third aspect using the fish individual identification device. [Figure 6] FIG. 11 is a diagram showing, as an example, fish images of learning data in a fourth aspect using the fish individual identification device. [Figure 7] FIG. 13 is a diagram showing, as an example, fish images of learning data in a fifth aspect using the fish individual identification device. [Figure 8] FIG. 13 is a diagram showing a fish information table as an example. [Figure 9] 4 is a flowchart showing the operation of the fish individual identification device regarding machine learning. [Figure 10] FIG. 4 is a diagram showing, as an example, a first screen displayed on the fish individual identification device. [Figure 11] FIG. 11 is a diagram showing, as an example, a second screen displayed on the fish individual identification device. [Figure 12] FIG. 11 is a diagram showing, as an example, a third screen displayed on the fish individual identification device. [Figure 13] FIG. 11 is a diagram showing a fourth screen displayed on the fish individual identification device as an example. [Figure 14] FIG. 11 is a diagram showing, as an example, a fifth screen displayed on the fish individual identification device. [Figure 15] FIG. 13 is a diagram showing a sixth screen displayed on the fish individual identification device, as an example. [Figure 16] FIG. 13 is a diagram showing, as an example, a seventh screen displayed on the fish individual identification device. [Figure 17]FIG. 13 is a diagram showing an eighth screen displayed on the fish individual identification device, as an example. [Figure 18] FIG. 13 is a diagram showing, as an example, a ninth screen displayed on the fish individual identification device. [Figure 19] 4 is a flowchart showing the operation of the fish individual identification device regarding individual identification. [Figure 20] FIG. 13 is a diagram showing, as an example, a tenth screen displayed on the fish individual identification device. [Figure 21] FIG. 13 is a diagram showing an eleventh screen displayed on the fish individual identification device as an example. [Figure 22] FIG. 15 is a diagram showing, as an example, a twelfth screen displayed on the fish individual identification device. [Figure 23] FIG. 13 is a diagram showing, as an example, a thirteenth screen displayed on the fish individual identification device. [Figure 24] FIG. 17 is a diagram showing, as an example, a fourteenth screen displayed on the fish individual identification device. [Diagram 25] FIG. 2 is a diagram for explaining a fish individual identification device in a first modified embodiment. [Figure 26] FIG. 11 is a diagram for explaining a fish individual identification device in a second modified embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0025] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In each drawing, the same reference numerals are used to denote the same configuration, and the description thereof will be omitted as appropriate. In this specification, when referring to a general term, a reference numeral without a subscript is used, and when referring to an individual configuration, a reference numeral with a subscript is used.

[0026] The fish individual identification device in the embodiment is a device that identifies individual fish having a pattern on the body surface. This fish individual identification device includes a fish image acquisition unit and a fish individual identification unit. The fish image acquisition unit acquires an image of the body surface of the fish. The fish individual identification unit identifies individual fishes captured in an image acquired by the fish image acquisition unit by using a machine learning model that identifies individual fishes based on the image. Such a fish individual identification device, as well as a fish individual identification method and a fish individual identification program implemented therein, will be described in more detail below.

[0027] FIG. 1 is a block diagram showing the configuration of a fish individual identification device in an embodiment. FIG. 2 is a schematic diagram showing the configuration of a drug administration experiment system using the fish individual identification device. FIG. 3 is a diagram showing, as an example, a fish image of learning data in a first aspect in which the fish individual identification device is used. FIG. 4 is a diagram showing, as an example, a fish image of learning data in a second aspect in which the fish individual identification device is used. FIG. 5 is a diagram showing, as an example, a fish image of learning data in a third aspect in which the fish individual identification device is used. FIG. 6 is a diagram showing, as an example, a fish image of learning data in a fourth aspect in which the fish individual identification device is used. In FIG. 2 to FIG. 6, FIG. A shows an example of a fish image, and FIG. B shows a schematic diagram of a fish image obtained by tracing the outline of a fish captured in the fish image shown in FIG. A. FIG. 7 is a diagram showing, as an example, a fish image of learning data in a fifth aspect in which the fish individual identification device is used. In Fig. 7, Fig. A shows an example of a fish image, Fig. B shows a schematic diagram of a fish image obtained by tracing the contour of a fish captured in the fish image shown in Fig. A, Fig. C shows an example of a fish image, and Fig. D shows a schematic diagram of a fish image obtained by tracing the contour of a fish captured in the fish image shown in Fig. C. Fig. A shows one side of the fish, and Fig. C shows the other side of the fish. Fig. 8 shows an example of a fish information table.

[0028] The fish individual identification device 1000 in the embodiment includes a fish image acquisition unit 1, a control processing unit 2, an input unit 3, an output unit 4, an interface unit (IF unit) 5, and a storage unit 6, as shown in FIG.

[0029] The fish individual identification device 1000 is a device that identifies individual fish Ob having a pattern on the body surface. The fish Ob to be identified may be any type of fish as long as it has a pattern on the body surface. The pattern includes, for example, a striped pattern. Alternatively, for example, the pattern includes a vertical striped pattern. Alternatively, for example, the pattern includes a horizontal striped pattern. Here, in the case of fish, the vertical direction is the direction of the body length. That is, when the fish is placed with its head up and its tail fin down, the direction of the striped pattern is determined, with the up-down direction being the vertical direction and the left-right direction being the horizontal direction. The fish is a zebrafish (scientific name: Danio rerio) from the viewpoint of being favorable for the research.

[0030] Such a fish individual identification device 1000 is used, for example, in a drug administration experiment, and a drug administration experiment system SY for carrying out this drug administration experiment includes, for example, a pre-administration tank TB for accommodating the fish Ob before drug administration, an imaging tank TS for accommodating the fish Ob to generate a fish image, a holding device HD for holding the fish Ob to administer a drug, a post-administration tank TA for accommodating the fish Ob to raise the fish Ob after drug administration, and the fish individual identification device 1000, as shown in Fig. 2. The imaging tank TS is a hollow rectangular parallelepiped that is relatively long in one direction, and on its top surface is provided an input port EN for inputting the fish Ob and an output port EX for removing the fish Ob, and when imaging the fish Ob, the imaging tank TS is filled with water up to the top surface in order to prevent reflection from the water surface. 2, reflecting mirrors ML and MR inclined at 45 degrees are arranged on both sides of the imaging tank TS so that the imaging unit 1, which is an example of the fish image acquisition unit 1, can capture images of the back (top) of the fish Ob, one side thereof, and the other side thereof in a single capture. The reflecting mirrors ML and MR are arranged so that the normals of the reflecting surfaces of the reflecting mirrors ML and MR intersect at 90 degrees with the optical axis of the imaging unit 1. This makes the one side and the other side of the fish Ob visible from the optical axis direction along the optical axis of the imaging unit 1, and the imaging unit 1 can capture images of the back, one side, and the other side of the fish Ob in a single capture.

[0031] Returning to FIG. 1, the fish image acquisition unit 1 is connected to the control processing unit 2 and acquires an image (fish image) of the body surface of the fish Ob under the control of the control processing unit 2. In the example shown in FIG. 2, the fish image acquisition unit 1 is an example of the image acquisition unit 1 that generates an image as described above. The image acquisition unit 1 may be configured to include, for example, a monochrome digital camera, but in this embodiment, it is configured to include a color digital camera. Note that the fish image acquisition unit 1 may be configured to include, for example, an interface circuit that inputs and outputs data to and from an external device. The external device is a storage medium, such as a USB (Universal Serial Bus) memory and an SD card (registered trademark), in which the fish image is stored. Alternatively, the external device is a drive device that reads data from a storage medium, such as a CD-ROM (Compact Disc Read Only Memory), a CD-R (Compact Disc Recordable), a DVD-ROM (Digital Versatile Disc Read Only Memory), and a DVD-R (Digital Versatile Disc Recordable), in which the fish image is recorded. Alternatively, the fish image acquisition unit 1 may be, for example, a communication interface circuit that transmits and receives communication signals to and from an external device, and the external device may be a server device that is connected to the communication interface circuit via a network (WAN (Wide Area Network, including a public communication network)) or LAN (Local Area Network) and manages fish images. Here, when the fish image acquisition unit 1 is an interface circuit or a communication interface circuit, the fish image acquisition unit 1 may also serve as the IF unit 5 (i.e., the IF unit 5 may be used as the fish image acquisition unit 1).

[0032] In the above description, one imaging unit 1 captures an image of the back, one side, and the other side of the fish Ob to generate one fish image, but each fish image may be generated individually by an imaging unit (first imaging unit) that generates an image of the back of the fish Ob, an imaging unit (second imaging unit) that generates an image of one side of the fish Ob, and an imaging unit (third imaging unit) that generates an image of the other side of the fish Ob.

[0033] The input unit 3 is connected to the control processing unit 2 and is a device that inputs various commands, such as a command to start machine learning or a command to start individual identification, and various data required to operate the fish individual identification device 1000, such as an experiment name and an individual fish name, to the fish individual identification device 1000, and is, for example, a keyboard, a mouse, and a plurality of input switches to which predetermined functions are assigned. The output unit 4 is connected to the control processing unit 2 and is a device that outputs the commands and data input from the input unit 3, fish images, individual identification results, etc., under the control of the control processing unit 2, and is, for example, a display device such as a CRT display, LCD (liquid crystal display device), or organic EL display, or a printing device such as a printer.

[0034] The input unit 3 and the output unit 4 may be configured as a touch panel. In the case of configuring this touch panel, the input unit 3 is a position input device that detects and inputs an operation position, for example, a resistive film type or a capacitive type, and the output unit 4 is a display device. In this touch panel, a position input device is provided on the display surface of the display device, and one or more input content candidates that can be input to the display device are displayed. When a user touches a display position that displays the input content that the user wants to input, the position is detected by the position input device, and the display content displayed at the detected position is input to the fish individual identification device 1000 as the user's operation input content. With such a touch panel, the user can easily intuitively understand the input operation, and therefore a fish individual identification device 1000 that is easy for the user to handle is provided.

[0035] The IF unit 5 is connected to the control processing unit 2 and is a circuit that inputs and outputs data to and from, for example, an external device under the control of the control processing unit 2, and is, for example, an interface circuit of RS-232C, which is a serial communication method, an interface circuit using the Bluetooth (registered trademark) standard, an interface circuit using the USB standard, etc. The IF unit 5 may also be, for example, a communication interface circuit that transmits and receives communication signals to and from an external device, such as a data communication card or a communication interface circuit conforming to the IEEE802.11 standard, etc.

[0036] The storage unit 6 is connected to the control processing unit 2 and is a circuit that stores various predetermined programs and various predetermined data according to the control of the control processing unit 2. The various predetermined programs include, for example, a control processing program, and the control processing program includes, for example, a control program, a machine learning program, a fish individual identification program, and a recording processing program. The control program is a program that controls each unit 1, 3 to 6 of the fish individual identification device 1000 according to the function of each unit. The machine learning program is a program that uses learning data to machine-learn a machine learning model (fish individual identification model) that identifies individual fish based on an image. The fish individual identification program is a program that uses the machine learning model to identify individual fish captured in an image acquired by the fish image acquisition unit 1. The recording processing program is a program that stores predetermined information (fish information) about the fish Ob in the storage unit 6. The various predetermined data includes, for example, data necessary for executing each of these programs, such as learning data, fish images acquired by the fish image acquisition unit 1, fish information, and each screen.

[0037] Such a storage unit 6 includes, for example, a ROM (Read Only Memory) which is a nonvolatile storage element, an EEPROM (Electrically Erasable Programmable Read Only Memory) which is a rewritable nonvolatile storage element, etc. The storage unit 6 also includes a RAM (Random Access Memory) which serves as a so-called working memory of the control processing unit 2 for storing data generated during execution of the predetermined program, etc. The storage unit 6 may also be configured to include a hard disk device with a relatively large storage capacity.

[0038] The memory unit 6 functionally comprises a learning data memory unit 61 and a fish information memory unit 62 for storing the learning data and the fish information, respectively.

[0039] The learning data storage unit 61 stores a learning data set including a plurality of learning data. The learning data is supervised learning data including a fish image including a fish Ob and an identifier of the fish included in the fish image. The fish identifier (ID) is a code (symbol, one or more characters, etc.) for specifying and identifying the fish Ob, such as a fish name given to each fish Ob. The fish image PCa of the first mode in the learning data is, for example, a color image generated by the imaging unit 1 as shown in FIG. 3, and the back (top) of the fish Ob, one side surface thereof, and the other side surface thereof are included in the fish image PCa. Alternatively, for example, the fish image PCb of the second mode in the learning data is, for example, a color image generated by the imaging unit 1 as shown in FIG. 4, and the fish image PCb is, for example, an image obtained by removing an image area including the back surface of the fish Ob from the fish image PCa in a rectangular shape. Alternatively, for example, a fish image PCc of a third mode in the learning data is a fish image with a monochromatic background (e.g., black or white, etc.) as shown in FIG. 5, and the back (top) of the fish Ob, its one side, and its other side are reflected in the fish image PCc. Such a fish image PCc may be created, for example, by image processing that fills the background of the fish image generated by the imaging unit 1 with a monochromatic color, or may be created, for example, by placing a monochromatic background member against the fish Ob and generating the fish image by the imaging unit 1. Alternatively, for example, a fish image PCd of a fourth mode in the learning data is a fish image with a monochromatic background as shown in FIG. 6, and the fish image PCd is a fish image with a monochromatic background and its one side and its other side are reflected in the fish image PCd. As in these first to fourth modes, the image of one side of the fish Ob and the image of the other side of the fish Ob may be composed of one fish image PCa to PCd. Alternatively, for example, the fifth aspect of the fish image PCe in the learning data is a pair of fish images PCe1 and PCe2 with a monochrome background, as shown in Figure 7, where one fish image PCe1 reflects one side of the fish Ob, and the other fish image PCe2 reflects the other side of the fish Ob.As in the fifth embodiment, the image of one side of the fish Ob and the image of the other side of the fish Ob may be composed of separate images PCe1 and PCe2.

[0040] In this embodiment, learning data is prepared for at least each fish Ob to be classified, and multiple learning data are prepared for one individual fish Ob.

[0041] The fish information storage unit 62 stores fish information. The fish information is predetermined information related to the fish Ob, and is configured with appropriate items (attributes) according to the intended use of the fish Ob. For example, in the case of a follow-up observation of drug administration, the fish information is configured with the ID, weight, body length, and the date of measurement of the fish Ob.

[0042] In this embodiment, this fish information is stored in table form in the fish information storage unit 62. The fish information table LT in which this fish information is registered includes, for example, an individual ID field 621 in which the ID of a fish Ob is registered, a date field 622 in which the measurement date on which the weight and body length of the fish Ob having the ID registered in the individual ID field 621 was measured, a weight field 623 in which the weight of the fish Ob having the ID registered in the individual ID field 621, measured on the measurement date registered in the date field 622, and a body length field 624 in which the body length of the fish Ob having the ID registered in the individual ID field 621, measured on the measurement date registered in the date field 622, has a record for each ID and measurement date, as shown in FIG.

[0043] The control processing unit 2 is a circuit for controlling each of the units 1, 3 to 6 of the fish individual identification device 1000 according to the function of each unit, for machine learning the machine learning model, and for identifying each individual fish Ob. The control processing unit 2 is configured, for example, with a CPU (Central Processing Unit) and its peripheral circuits. In the control processing unit 2, a control unit 21, a machine learning unit 22, an individual fish identification unit 23, and a record processing unit 24 are functionally configured by executing the control processing program.

[0044] The control unit 21 controls each of the units 1, 3 to 6 of the fish individual identification device 1000 according to the function of each unit, and is responsible for controlling the fish individual identification device 1000 as a whole.

[0045] The machine learning unit 22 uses training data to train an untrained machine learning model that identifies individual fish based on an image. In this embodiment, the machine learning unit 22 trains the machine learning model using a training data set stored in the training data storage unit 61. In this embodiment, when a fish image PC is input, the machine learning model identifies individual fishes captured in the fish image PC and outputs the ID and confidence level [%] of the fish Ob. For example, a deep learning Convolutional Neural Network (CNN) is used as the machine learning model. More specifically, in this embodiment, the well-known VGG16, which is a type of CNN and has relatively high object recognition accuracy, is used. The VGG16 includes 13 layers of first to 13th convolution layers and three layers of first to third connection layers. More specifically, between the first and second convolution layers and the third and fourth convolution layers, a first pooling layer (max The neural network has a second pooling layer for the third and fourth convolution layers between the third and fourth convolution layers and the fifth to seventh convolution layers, a third pooling layer for the fifth to seventh convolution layers between the fifth to seventh convolution layers and the eighth to tenth convolution layers, and a fourth pooling layer for the eighth to tenth convolution layers between the eighth to tenth convolution layers and the eleventh to thirteenth convolution layers. A softmax is used in the final layer.

[0046] In this embodiment, the machine learning unit 22 further creates a learning data set before the machine learning, and stores the created learning data set in the learning data storage unit 61. More specifically, in this embodiment, the machine learning unit 22 captures a plurality of images of each fish Ob to be identified by the imaging unit 1, generates, for example, a plurality of fish images PCe (PCe1, PCe2) of a fifth mode from each of the captured fish images PC, and associates the ID of the fish Ob with each of the generated fish images PCe of the fifth mode, thereby creating a plurality of learning data for the fish Ob, and stores the created plurality of learning data for each of the fish Ob to be identified in the learning data storage unit 61 as a learning data set. Here, fish images that are inappropriate as the fish images PC of the learning data, such as fish images PC generated by the imaging unit 1 in which the fish Ob is captured with a bent body part, are discarded and excluded. Note that a plurality of fish images PCe may be generated and padded from the fish image PC generated by the imaging unit 1 by, for example, changing the position of the image area in which the fish Ob appears.

[0047] The individual fish identification unit 23 uses a machine learning model that identifies individual fish based on an image, thereby identifying individual fish Ob captured in an image acquired by the fish image acquisition unit 1. The machine learning model used by the individual fish identification unit 23 is a machine learning model that has been machine-learned by the machine learning unit 22. More specifically, an individual fish Ob captured in the fish image PC may be identified by a machine learning model that has been machine-learned based on one fish image PC acquired by the imaging unit 1, but in this embodiment, in order to identify the individual fish more accurately, the individual fish identification unit 23 identifies the individual fish Ob captured in the fish image PC by a machine learning model that has been machine-learned based on multiple fish images PC acquired by the imaging unit 1. More specifically, the individual fish identification unit 23 includes a VGG16 (not shown) as a machine learning model that has been machine-learned, and a confidence calculation unit (not shown) that calculates a final confidence level for each ID based on each ID and each confidence level of the fish Ob identified in each of the multiple fish images PC. First, the individual fish identification unit 23 causes the imaging unit 1 to generate and acquire multiple fish images PC for one individual fish. Next, the individual fish identification unit 23 obtains the ID of the fish Ob and its certainty as an output for each of the multiple fish images PC using the VGG 16. Then, the individual fish identification unit 23 obtains, for each of the obtained IDs of the fish Ob, the average value of the certainty corresponding to the ID of the fish Ob as the final certainty using the certainty calculation unit. For this reason, in this embodiment, the ID of the fish Ob obtained in this manner and its final certainty are output from the individual fish identification unit 23.

[0048] The recording processing unit 24 stores the fish information in the fish information storage unit 62 of the storage unit 6. In this embodiment, the recording processing unit 24 functions as an individual name correction unit that corrects the individual name output from the individual fish identification unit 23, which in this embodiment is the ID of the fish Ob. More specifically, when the machine learning unit 22 creates a learning dataset as described above, the recording processing unit 24 assigns an ID to each individual fish Ob, obtains the weight and body length of the fish Ob, creates a new record in the fish information table LT stored in the fish information storage unit 62, and registers the assigned ID, the obtained weight, the obtained body length, and the measurement date as fish information of the fish Ob in the individual ID field 621, weight field 623, body length field 624, and date field 622 of the newly created record. As a result, for example, an initial value for starting follow-up of drug administration is stored (recorded) in the fish information storage unit 62 for each individual. In order to register the date of measurement, in this embodiment, the recording processing unit 24 has a clock function for counting the date. The weight and body length may be measured manually by an operator (user), but in this embodiment, the weight and body length of the fish Ob are obtained based on the fish image PC. For example, the recording processing unit 24 first removes the image areas of the dorsal fin, pelvic fin, and caudal fin by machine learning-prepared semantic segmentation, and extracts an image area of ​​the body part of the fish Ob to determine the body part. Next, the recording processing unit 24 obtains a rectangular area surrounding the body part on the back side, and obtains the long length of the obtained rectangular area as the body length of the fish Ob. Then, a model (fish weight estimation model) for obtaining the weight of the fish Ob based on the extracted image of the body part of the fish Ob is generated in advance, and the recording processing unit 24 obtains the weight of the fish Ob based on the extracted image of the body part of the fish Ob using this fish weight estimation model. The fish weight estimation model may be, for example, a machine learning model, and was obtained by linear multiple regression analysis in which the objective variable was the weight of the fish Ob and the explanatory variables were the area of ​​the back of the body part, the area of ​​one side and the other side of the body part, and the inclination of the body relative to the up-down direction of the fish image PC when viewed from the back (top).Then, when the machine learning of the machine learning model is completed and individual identification is performed, the recording processing unit 24 obtains the weight and body length of the fish Ob based on the fish image PC used for the individual identification. Next, the recording processing unit 24 creates a new record in the fish information table LT stored in the fish information storage unit 62, and registers the ID as the individual name, the obtained weight, the obtained body length, and the measurement date as fish information of the fish Ob captured in the fish image PC used for the individual identification in the individual ID field 621, the weight field 623, the body length field 624, and the date field 622 of the newly created record. When the individual name output from the fish individual identification unit 23 (in this example, the ID of the fish Ob) is corrected, the corrected individual name is used as the individual name, while when the individual name output from the fish individual identification unit 23 is not corrected, the individual name output from the fish individual identification unit 23 is used. As a result, for example, fish information during follow-up observation of drug administration is stored (recorded) in the fish information storage unit 62.

[0049] The control processing unit 2, input unit 3, output unit 4, IF unit 5 and memory unit 6 can be configured, for example, by a desktop or notebook computer. As described above, if the fish image acquisition unit 1 is an interface circuit or a communication interface circuit, the fish image acquisition unit 1 can also serve as the IF unit 5, and therefore the fish image acquisition unit 1 as well can be configured by a computer.

[0050] Next, the operation of this embodiment will be described. Fig. 9 is a flowchart showing the operation of the fish individual identification device regarding machine learning. Figs. 10 to 18 are diagrams showing, as an example, first to ninth screens displayed on the fish individual identification device. Fig. 19 is a flowchart showing the operation of the fish individual identification device regarding individual identification. Figs. 20 to 24 are diagrams showing, as an example, tenth to fourteenth screens displayed on the fish individual identification device.

[0051] When the power is turned on, the fish individual identification device 1000 configured as described above initializes each required part and starts its operation. The control processing unit 2 functionally configures a control unit 21, a machine learning unit 22, an individual fish identification unit 23, and a record processing unit 24 by executing the control processing program.

[0052] First, the operation of the fish individual identification device 1000 regarding machine learning will be described. In machine learning, first, in order to create a learning dataset, in FIG. 9, the fish individual identification device 1000 acquires a fish image PC with the imaging unit 1 by the machine learning unit 22 of the control processing unit 2 (S11, fish image acquisition step). More specifically, the control unit 21 of the control processing unit 2 displays, for example, a first screen (fish individual identification top screen) SC1 shown in FIG. 10 on the output unit 4. The first screen SC1 includes an "imaging" button 41 that instructs the fish individual identification device 1000 to capture an image with the imaging unit 1, and a "data browsing" button 42 that instructs the fish individual identification device 1000 to browse fish information. When the data browsing button 42 is operated by an operator, the recording processing unit 24 of the control processing unit 2 reads fish information registered in the fish information table LT from the fish information storage unit 62, and outputs the read fish information to the output unit 4, for example, in a list format. Here, the "imaging" button 41 is operated by the operator. When the "imaging" button 41 is operated, the control unit 21 displays, for example, the second screen (learning individual identification selection screen) SC2 shown in FIG. 11 on the output unit 4. The second screen SC2 includes a "first time (learning)" button 43 that instructs the fish individual identification device 1000 to create a learning dataset and to perform machine learning of an unmachine-learned machine learning model, a "second time or later (discrimination)" button 44 that instructs the fish individual identification device 1000 to perform individual identification and obtain fish information, and a "return to TOP" button 45 that instructs the fish individual identification device 1000 to display the first screen SC. When the "return to TOP" button 45 is operated by an operator, the first screen SC1 is displayed on the output unit 4. Here, the "first time (learning)" button 43 is operated by the operator. When the "first time (learning)" button 43 is operated by an operator, the control unit 21 displays, for example, the third screen (case name input screen) SC3 shown in FIG. 12 on the output unit 4. The third screen SC3 includes a case name display area 46 that displays the case name inputted by the input unit 3, and a "Next" button 47 that instructs the fish individual identification device 1000 to transition to the next screen. Here, the case name "XX case" is inputted by the operator, and the "Next" button 47 is operated by the operator.When the "Next" button 47 is operated, the control unit 21 displays, for example, a fourth screen (first imaging start instruction screen) SC4 shown in FIG. 13 on the output unit 4. The fourth screen SC3 includes an ID display area 48 that displays the ID of the fish Ob inputted by the input unit 3, and a "imaging start" button 49 that instructs the fish individual identification device 1000 to start imaging. The ID of the fish Ob may be automatically generated by the control processing unit 2, but in this embodiment, it is generated and input by the operator. In the example shown in FIG. 13, a video (monitor video) captured by the imaging unit 1 is also displayed on the fourth screen SC4. Here, the "imaging start" button 49 is operated by the operator. When the "imaging start" button 49 is operated, the machine learning unit 22 acquires a fish image PC by the imaging unit 1. During the acquisition of this fish image PC, in this embodiment, the control unit 21 displays, for example, a fifth screen (imaging in progress notification screen) SC5 shown in FIG. 14 on the output unit 4. The fifth screen SC5 includes a "Stop Imaging" button 50 for instructing the fish individual identification device 1000 to stop capturing images. When the "Stop Imaging" button 50 is operated, capturing images is stopped, and the first screen SC1 is displayed on the output unit 4. In the example shown in Fig. 13, the fifth screen SC5 displays the progress of the imaging process, and displays the fish image PC being captured.

[0053] Next, the fish individual identification device 1000 uses the machine learning unit 22 to determine whether the acquired fish image PC is in a proper posture in order to determine whether the acquired fish image PC can be used as learning data (S12). If the result of this determination is that the posture is not proper (No), the machine learning unit 22 discards the fish image PC acquired in process S12 (S14) and returns the process to process S11. On the other hand, if the result of the determination is that the posture is proper (Yes), the machine learning unit 22 next stores the fish image PC acquired in process S12 in the memory unit 6 (S13).

[0054] The judgment in process S12 may be made manually by an operator, but in this embodiment, for example, a pattern of proper posture is prepared in advance and the judgment is made by pattern matching, or by a machine learning model (proper posture discrimination model) that has been trained by machine learning and that judges whether or not the posture is proper based on an image, for example.

[0055] Following process S13, the fish individual identification device 1000 performs part determination on the fish image PC acquired in process S12 to obtain weight and body length using the recording processing unit 24 of the control processing unit 2 (S15), and then the fish individual identification device 1000, using the recording processing unit 24, obtains the weight and body length of the fish Ob captured in the fish image PC acquired in process S12 and stores the fish information in the fish information storage unit 62 (S16). In this embodiment, the fish information is stored in association with a case name. More specifically, the case name is associated with the fish information table LT in which the fish information is registered.

[0056] Next, the fish individual identification device 1000 uses the machine learning unit 22 to process the fish image PC acquired in process S12 for use as learning data, and stores the data in the memory unit 6 in association with the ID of the fish Ob (S17). The machine learning unit 22 processes the fish image PC acquired in process S12 into, for example, one of the fish images PCb to PCe of second to fifth modes, for example, into a fish image PCe of a fifth mode, and stores the processed fish image PCe of the fifth mode in association with the ID of the fish Ob in the memory unit 6.

[0057] Next, the fish individual identification device 1000 determines whether or not a predetermined number (a first number, for example, 300 or 400) of fish images PC set in advance has been acquired by the machine learning unit 22 (S18). If the result of this determination is that the predetermined number of fish images PC has not been acquired (No), the fish individual identification device 1000 returns the process to step S11. If the result of the determination is that the predetermined number of fish images PC has been acquired (Yes), the fish individual identification device 1000 stores (stores) the fish images PC processed for the learning data and their IDs that were temporarily stored (saved) in the storage unit 6 in step S17 in the learning data storage unit 61 as learning data of the learning dataset (S19).

[0058] Following the process S19, the fish individual identification device 1000 determines whether or not the learning data of the learning data set has been created for all the individuals of the fish Ob by the machine learning unit 22 (S20). This determination is made by an instruction input by the operator. The machine learning unit 22 displays, for example, a sixth screen (first determination instruction screen) SC6 shown in FIG. 15 on the output unit 4. The sixth screen SC6 includes a "Take an image of the next individual" button 51 that instructs the fish individual identification device 1000 to create a learning data set for the next individual, and a "Complete all individuals" button 52 that instructs the fish individual identification device 1000 to complete the creation of the learning data set for all the individuals. When the "Take an image of the next individual" button 51 is input by the operator, the fish individual identification device 1000 determines that the learning data has not been created for all the individuals (No), and returns the process to process S11. On the other hand, when the "All individuals completed" button 52 is input by the operator, it is determined that the learning data has been created for all individuals (Yes), and the fish individual identification device 1000 causes the machine learning unit 22 to machine-learn the untrained machine learning model using the learning data set stored in the learning data memory unit 61 (S21, machine learning process), and stores this machine-learned machine learning model in the memory unit 6 as the machine learning model to be used by the fish individual identification unit 23 (S22), and terminates this process.

[0059] In the above description, the process S21 is started automatically, but it may be started by an instruction from an operator. In this case, the machine learning unit 22 displays, for example, a seventh screen (learning instruction screen) SC7 shown in Fig. 16 on the output unit 4. The seventh screen SC7 includes a "Start learning" button 53 for instructing the fish individual identification device 1000 to start machine learning, and a "Return to TOP" button 54 for instructing the individual identification device 1000 to display the first screen SC1.

[0060] During the execution of the process S22, for example, an eighth screen (learning in progress notification screen) SC8 shown in FIG. 17 may be displayed on the output unit 4, and when the machine learning is completed, for example, a ninth screen (learning end notification screen) SC9 shown in FIG. 18 may be displayed on the output unit 4. The eighth screen SC8 includes a "Stop learning" button 55 that instructs the fish individual identification device 1000 to stop learning, and a progress status display bar 56 that displays the progress status of the machine learning. When the "Stop learning" button 55 is input by the operator, the seventh screen SC7 is displayed on the output unit 4. The ninth screen SC9 includes a "Return to TOP" button 57 that instructs the individual identification device 1000 to display the first screen SC1, and a learning end display area 58 that displays the end of the machine learning.

[0061] By operating in this manner, a training dataset is created, which is then used to train the machine learning model.

[0062] Next, the operation of the fish individual identification device 1000 regarding individual identification will be described. In individual identification, in FIG. 19, first, the fish individual identification device 1000 acquires a fish image PC with the imaging unit 1 by the fish individual identification unit 23 of the control processing unit 2 (S31, fish image acquisition step). More specifically, the control unit 21 of the control processing unit 2 displays a first screen SC1 on the output unit 4. Here, the "imaging" button 41 is operated by the operator. When the "imaging" button 41 is operated, the control unit 21 displays a second screen SC2 on the output unit 4. Here, the "second time or later (discrimination)" button 44 is operated by the operator. When the "second time or later (discrimination)" button 44 is operated, the control unit 21 displays, for example, a tenth screen (case selection screen) SC10 shown in FIG. 20 on the output unit 4. The tenth screen SC10 is a screen for selecting a case name of fish information stored in the fish information storage unit 62. In the tenth screen SC of the example shown in FIG. 20, the items of the case name, the number of individuals for the case name, and the measurement date are displayed in a list format for each case name. The number of individuals for the case name is obtained by counting the number of different IDs registered in the fish information table LT corresponding to the case name. The measurement date is obtained by extracting the measurement dates registered in the fish information table LT corresponding to the case name for each different date. The tenth screen SC10 includes a "Select" button 71 (71-1, 71-2, 71-3) for each case name displayed in the list format that instructs the fish individual identification device 1000 to specify the case name, a vertical scroll slider 72-1 that instructs the fish individual identification device 1000 to scroll the case names displayed in the list format, a horizontal scroll slider 72-2 that instructs the fish individual identification device 1000 to scroll each item displayed in the list format, and a "Back" button 73 that instructs the fish individual identification device 1000 to display the previous screen SC. Here, a "Select" button 71 (e.g., "Select" button 71-1) is operated by the operator. When the "Select" button 71 is operated, the control unit 21 displays, for example, an eleventh screen (second imaging start instruction screen) SC11 shown in FIG. 21 on the output unit 4. The eleventh screen SC11 includes a "Start imaging" button 74 that instructs the fish individual identification device 1000 to start imaging.In the example shown in FIG. 21, a message saying "When the subject to be imaged is ready, press the image capture start button" and a video (monitor video) being captured by the imaging unit 1 are also displayed on the eleventh screen SC11. When the "image capture start" button 74 is operated, the fish individual identification unit 23 acquires a fish image PC by the imaging unit 1. During acquisition of this fish image PC, in this embodiment, the control unit 21 displays, for example, a twelfth screen (image capture in progress notification screen) SC12 shown in FIG. 22 on the output unit 4. The twelfth screen SC12 includes a "image capture stop" button 75 that instructs the fish individual identification device 1000 to stop capturing images. In the example shown in FIG. 22, the twelfth screen SC12 displays the progress of the image capture process, and displays the fish image PC being captured.

[0063] Next, the fish individual identification device 1000 executes steps S32 through S38 which are similar to steps S12 through S18 described above.

[0064] In process S39 (fish individual identification process), which is executed when a predetermined number (a second number, for example, 100 or 200) of fish images PC have been acquired as a result of the judgment in process S38, the fish individual identification device 1000 uses the multiple fish images PC acquired in each of the above-mentioned processes in a machine learning model that has been trained by machine learning to identify individual fish Ob captured in the fish images PC by the fish individual identification unit 23, and stores the ID of the identified fish Ob and the final confidence level in the memory unit 6.

[0065] Following step S39, the fish individual identification device 1000 judges whether or not individual identification has been performed for all the fish Ob by the fish individual identification unit 23 (S40). This judgment is performed by an instruction input by the operator. The fish individual identification unit 23 displays, for example, a thirteenth screen (second judgment instruction screen) SC13 shown in FIG. 23 on the output unit 4. The thirteenth screen SC13 includes a "Take an image of the next individual" button 76 that instructs the fish individual identification device 1000 to perform individual identification for the next individual, and a "Complete all individuals" button 77 that instructs the fish individual identification device 1000 to complete the individual identification for all the individuals. When the "Take an image of the next individual" button 76 is input by the operator, the fish individual identification device 1000 judges that the individual identification has not been performed for all the individuals (No), and returns the process to step S31. On the other hand, when the "All individuals completed" button 77 is operated by the operator, it is determined that the individual identification has been performed on all individuals (Yes), and the fish individual identification device 1000 executes an individual name correction process, using the recording processing unit 24, to correct the individual name (ID) of the identification result (S41, individual name correction process).

[0066] More specifically, in the individual name correction process, the recording processing unit 24 outputs, for example, a fourteenth screen (individual name correction instruction screen) SC14 shown in Fig. 24 to the output unit 4. The fourteenth screen SC14 is a screen for displaying the identification result and instructing the fish individual identification device 1000 to correct the individual name (ID) of the identification result. The fourteenth screen SC14 in the example shown in Fig. 24 includes an identification result display area 78 that displays the individual name (ID) of the identification result and its confidence level in a list format for each individual to be identified. The individual name (ID) of the identification result and its confidence level displayed in this identification result display area 78 are displayed for each individual identified by the machine learning model. For example, if the result of individual identification using a machine learning model based on multiple fish images PC for one individual is individual name "A", individual name "B", and individual name "C", the identification results for this one individual to be identified are displayed in the identification result display area 78 as individual name "A" and its certainty, individual name "B" and its certainty, and individual name "C" and its certainty. The fourteenth screen SC14 includes a vertical scroll slider 79-1 for instructing the fish individual identification device 1000 to scroll through the individual fish to be identified that are displayed in a list format, a horizontal scroll slider 79-2 for instructing the fish individual identification device 1000 to scroll through pairs of individual fish names and their confidence levels, a correction content display area 80 for displaying correction content input by the operator via the input unit 3, a "Correction" button 81 for instructing the fish individual identification device 1000 to make corrections using the correction content displayed in the correction content display area 80, and a "Result registration" button 82 for instructing the fish individual identification device 1000 to store (save) the fish information as the identification result of the individual fish identified in process S39 or to store (save) the fish information as the identification result after the correction. In the example shown in FIG. 24, the correction content display area 80 includes a correction target display area 80-1 for displaying the individual fish to be identified whose individual fish name (ID) is to be corrected, and a corrected individual fish name display area 80-2 for displaying the individual fish name (ID) after the correction input by the operator via the input unit 3.

[0067] When the "Register result" button 82 is operated by the operator, the fish individual identification device 1000 causes the recording processing unit 24 to store (save) the fish information in the fish information storage unit 62 (S42). This process ends. In storing this fish information, if the correction content display area 80 is blank and the "Register result" button 82 is operated, the recording processing unit 24 stores the fish information as the identification result obtained by the individual identification in process S39 in the fish information storage unit 62. On the other hand, if the "Register result" button 82 is operated with the correction content displayed in the correction content display area 80, the recording processing unit 24 stores the fish information as the identification result after the correction in the fish information storage unit 62. Note that if the identification results are duplicated (for example, if the identification results for multiple individuals are all "A"), the "Register result" button 82 cannot be operated if the correction content display area 80 is blank.

[0068] By operating in this manner, the fish Ob to be identified is individually identified, and the fish information is stored (saved) in the fish information storage unit 62.

[0069] In the above, when the individual name (ID) is corrected, the fish image PC with the corrected name and the corrected ID may be added as learning data to the learning data set, and the machine learning model may be re-trained by the machine learning unit 22 using the learning data set to which this learning data has been added. In this case, the fourteenth screen SC14 may further be provided with a "model update" button 83 for instructing the fish individual identification device 1000 to perform the re-training, as shown by a dashed line in Fig. 24. This allows the certainty to be restored even if the fish Ob grows over time as observation progresses and the certainty decreases.

[0070] As described above, the fish individual identification device 1000 and the fish individual identification method and program implemented therein in this embodiment identify each individual fish Ob by using a machine learning model that identifies each individual fish based on an image. Therefore, this makes it possible to provide the fish individual identification device 1000, the fish individual identification method, and the fish individual identification program that identify each individual fish using a method different from the identification method used by humans.

[0071] According to this embodiment, it is possible to provide a fish individual identification device 1000 that generates a machine-learned machine learning model through machine learning, a fish individual identification method, and a fish individual identification program.

[0072] When the above-mentioned fish individual identification device 1000, fish individual identification method, and fish individual identification program train the machine learning model using training data including images of the fish with a monochrome background, the noisy background can be removed and only the fish can be trained, thereby enabling the machine learning model to be trained to identify individuals more accurately.

[0073] When the above-mentioned fish individual identification device 1000, fish individual identification method, and fish individual identification program train the machine learning model using learning data including an image of one side of the fish and an image of the other side of the fish, the machine learning model can be trained using patterns on both sides, thereby enabling more accurate individual identification.

[0074] The above-mentioned fish individual identification device 1000, fish individual identification method, and fish individual identification program are capable of correcting the individual name even if an error is made in the individual identification.

[0075] In the above-described embodiment, the fish individual identification device 1000 may further include a classification unit 25 that classifies a plurality of fish Ob into groups by using the pseudo space of the machine learning model, as indicated by the dashed line in Fig. 1 (first modified embodiment). In the example shown in Fig. 1, the storage unit 6 further stores a classification program that classifies a plurality of fish Ob into groups by using the pseudo space of the machine learning model as a program included in the control processing program, and the classification unit 25 is functionally configured in the control processing unit 2. Such fish individual identification device 1000, fish individual identification method, and fish individual identification program classify a plurality of fish Ob into groups, so that by selecting a fish Ob from each group, individual identification becomes easier and it becomes possible to reduce erroneous determinations in individual identification.

[0076] Fig. 25 is a diagram for explaining a fish individual identification device in a first modified embodiment. Fig. 25A shows a classification result in a pseudo space consisting of each node of a first combining layer in machine-learned VGG16 as an example of the machine learning model, and Fig. 25B shows a classification result in a pseudo space consisting of each node of a first pooling layer in machine-learned VGG16 as an example of the machine learning model. In Fig. 25A and Fig. 25B, the n-dimensional pseudo space is flattened to two dimensions for convenience of drawing.

[0077] FIG. 25A and FIG. 25B show the results obtained by using 288 fish images PC (for example, unprocessed fish images PC with one side imaged, approximately 30 fish images PC per individual) of nine (fish) individuals in a machine-learned machine-learning model. As can be seen by comparing FIG. 25A and FIG. 25B, near the first stage of VGG16 (near the input side, the first pooling layer in this example), the nine individuals cannot be classified in the pseudospace, but near the final stage of VGG16 (near the output side, the first combined layer in this example), the nine individuals are roughly grouped into nine groups Gr1 to Gr9, so that the nine individuals can be classified. Therefore, for example, by using the fish image PC of the tenth individual in the machine-learned machine-learning model to determine which of the nine groups Gr1 to Gr9 it belongs to, the group Gr to which the tenth individual belongs can be determined. Of course, there may be cases where the tenth individual does not belong to any of the nine groups Gr1 to Gr9. The determination may be performed, for example, by using a known clustering method. For example, the classification unit 25 determines the center position of each group Gr, and determines the distance between the result obtained by using the fish image PC of the 10th individual in the machine learning model that has been machine-learned and each center position of each group Gr. If any of these distances is equal to or less than a preset threshold (group determination threshold), the classification unit 25 determines that the 10th individual belongs to the group Gr corresponding to that distance. Note that, if a plurality of distances are equal to or less than the group determination threshold, the classification unit 25 may determine that the 10th individual belongs to the group Gr corresponding to the shortest distance. The operator selects the fish Ob to be identified (observed) by referring to the classification result of the classification unit 25. More specifically, the pseudo space is divided into a plurality of sections of a predetermined size, and the fish Ob to be identified is selected so that the classification result of the classification unit 25 is distributed among the plurality of sections. In addition, when the classification result of the fish Ob to be selected is located on the periphery of the section, if the classification result of another fish Ob that has already been selected is located at a distance equal to or less than a preset threshold (selection judgment threshold), the fish Ob to be selected is not selected. By adjusting the size of the section, the ease of individual identification can be adjusted.

[0078] In the above-described embodiment (including the first modified embodiment), the fish individual identification device 1000 may further include a transmission processing unit 26 that transmits at least one of the learning data and the machine learning model to a management server device that manages at least one of the learning data and the machine learning model in a shareable manner, as shown by a dashed line in FIG. 1 (second modified embodiment). In the example shown in FIG. 1, the storage unit 6 further stores, as a program included in the control processing program, a transmission processing program that transmits at least one of the learning data and the machine learning model to a management server device that manages at least one of the learning data and the machine learning model in a shareable manner, and the transmission processing unit 26 is functionally configured in the control processing unit 2. Such fish individual identification device 1000, fish individual identification method, and fish individual identification program can utilize at least one of the learning data and the machine learning model that are shareable, thereby enabling the machine learning model to have high accuracy.

[0079] Fig. 26 is a diagram for explaining a fish individual identification device in the second modified embodiment. In this second modified embodiment, for example, as shown in Fig. 26, a data model sharing system MS is configured including a management server device SV that manages at least one of the learning data and the machine learning model in a shareable manner, and a plurality of fish individual identification devices 1000 (1000-1, 1000-2, 1000-3, ..., 1000-n) in the second modified embodiment that are communicatively connected to the management server device SV via a communication network NW.

[0080] For example, in the fish individual identification device 1000-1, when an operator instructs transmission of learning data from the input unit 3, the transmission processing unit 26 generates a communication signal (learning data transmission signal) containing the learning data set stored in the learning data storage unit 61, and transmits the generated learning data transmission signal to the management server device VS via the IF unit 5 and the communication network NW. When the management server device SV receives the learning data transmission signal, it stores (saves) the learning data set contained in the received learning data transmission signal and manages it so that it can be shared. When the management server device SV receives a communication signal (learning data request signal) requesting transmission of the learning data set from, for example, the fish individual identification device 1000-2, it generates and returns a communication signal (learning data reply signal) containing the managed learning data set. When the fish individual identification device 1000-2 receives this learning data reply signal, it re-trains its own machine learning model that has already been trained by machine learning with the received learning data set. This allows the machine learning model of the own device to perform machine learning so that individuals can be identified more accurately. Alternatively, the fish individual identification device 1000-2 transfers learning to another machine learning model that has been machine-learned to identify other individuals other than zebrafish by the own device using the received learning dataset. This allows the other machine learning model to be diverted to a machine learning model that identifies individual zebrafish, and a machine learning model that identifies individual zebrafish can be generated in a shorter time. Alternatively, the fish individual identification device 1000-2 trains a machine learning model that has not been machine-learned using the received learning dataset on multiple fish images PCs generated by the own device. At this time, if the received learning dataset contains supervised learning data (not all of the learning data is necessarily supervised learning data), the machine learning model that has not been machine-learned can be trained by semi-supervised learning even if supervised data is not necessarily assigned to the multiple fish images PCs generated by the own device.

[0081] Also, for example, federated learning may be performed by mutually transmitting and receiving a machine-learned machine learning model between the management server device SV and the individual fish identification device 1000. First, the management server device SV distributes (transmits) a machine-learned machine learning model to a plurality of individual fish identification devices 1000. This distributed machine-learned machine learning model may be newly generated by the management server device SV, or may be generated by the individual fish identification device 1000 and transmitted to the management server device SV. Upon receiving the distribution of this machine-learned machine learning model, the individual fish identification device 1000 performs machine learning using its own learning dataset, and updates the distributed machine-learned machine learning model. The individual fish identification device 1000 returns this updated machine-learned machine learning model to the management server device SV. At this time, the updated machine-learned machine learning model itself may be returned, or only the changes made by the update may be returned. The management server device SV, which receives this reply, updates the distributed machine-learned machine learning model with the difference between the distributed machine-learned machine learning model and the returned machine-learned machine learning model (i.e., the changes). At this time, replies can be received from each of the multiple fish individual identification devices 1000, making it possible to update with multiple changes. The management server device SV then distributes (sends) this updated machine-learned machine learning model again to the multiple fish individual identification devices 1000. This is repeated as necessary. In this way, the fish individual identification device 1000 can obtain a machine-learned machine learning model that can more accurately identify individuals even if the learning data in the learning dataset is relatively small.

[0082] In order to express the present invention, the present invention has been described adequately and sufficiently through the embodiments with reference to the drawings in the above description, but it should be recognized that those skilled in the art can easily change and / or improve the above-mentioned embodiments. Therefore, unless the changes or improvements made by those skilled in the art are at a level that departs from the scope of the claims described in the claims, the changes or improvements are interpreted as being included in the scope of the claims. [Explanation of symbols]

[0083] 1000 Fish individual identification device SV management server device 1. Fish image acquisition unit 2. Control processing section 3. Input section 4 Output section 5 Interface section (IF section) 6 Memory section 21 Control section 22 Machine Learning Department 23 Fish Identification Division 24 Recording processing section (including correction section) 25 Classification Department 26 Transmission processing unit

Claims

1. A method for identifying individual fish having a pattern on a body surface, comprising: a first fish image acquiring step of acquiring an image of a body surface of the fish as a first fish image; a classification step of classifying the plurality of first fish images acquired in the first fish image acquisition step into groups of individual fish by using a pseudospace of a machine learning model that identifies individual fish based on images; a second fish image acquisition step of acquiring an image of a body surface of any one of the fish selected from each group classified in the classification step as a second fish image; and a fish individual identification process for identifying each individual fish captured in the second fish image acquired in the second fish image acquisition process by using the machine learning model. The pattern includes a striped pattern. Methods for identifying individual fish.

2. Further comprising a machine learning step of machine learning the machine learning model using training data, The machine learning model used in the fish individual identification process is a machine learning model that has been machine-learned in the machine learning process. The method for identifying individual fish according to claim 1.

3. The training data includes images of the fish with a solid color background. The method for identifying individual fish according to claim 2.

4. The learning data includes an image of one side of the fish and an image of the other side of the fish generated by one camera in one image capture. The method for identifying individual fish according to claim 2.

5. The fish individual identification step further includes outputting an individual name of the identified individual, The method further includes a step of correcting the individual name output in the fish individual identification step. The method for identifying individual fish according to claim 1.

6. A machine learning process of machine learning the machine learning model using training data; A transmission process for transmitting at least one of the learning data and the machine learning model to a management server device that manages at least one of the learning data and the machine learning model so that they can be shared. The method for identifying individual fish according to claim 1.

7. A fish individual identification device that individually identifies fish having a pattern on a body surface, a first fish image acquisition unit that acquires an image of a body surface of the fish as a first fish image; A classification unit that classifies the first fish images acquired by the first fish image acquisition unit into groups of individual fish by using a pseudo space of a machine learning model that identifies individual fish based on images. a second fish image acquisition unit that acquires, as a second fish image, an image of a body surface of any one of the fish selected from each group classified by the classification unit; and a fish individual identification unit that uses the machine learning model to identify individual fishes appearing in the second fish image acquired by the second fish image acquisition unit. The pattern includes a striped pattern. Fish individual identification device.

8. A machine learning unit that machine-learns the machine learning model using learning data, The machine learning model used in the fish individual identification unit is a machine learning model that has been machine-learned by the machine learning unit. The fish individual identification device according to claim 7.

9. The training data includes images of the fish with a solid color background. The fish individual identification device according to claim 8.

10. The learning data includes an image of one side of the fish and an image of the other side of the fish generated by one camera in one image capture. The fish individual identification device according to claim 8.

11. The fish individual identification unit further outputs an individual name of the identified individual, Further, an individual name correction unit is provided for correcting the individual name output by the individual fish identification unit. The fish individual identification device according to claim 7.

12. A machine learning unit that machine-learns the machine learning model using training data; A transmission processing unit that transmits at least one of the learning data and the machine learning model to a management server device that manages at least one of the learning data and the machine learning model in a shareable manner. The fish individual identification device according to claim 7.

13. A fish individual identification program for causing a computer to function as the fish individual identification device according to any one of claims 7 to 12.

Images