Information processing program, information processing device, and information processing method

Abstract

To estimate the number of tails of fishes included in a fish school corresponding to an area in which individual fishes cannot be visually identified in images.SOLUTION: An information processing program according to the present application causes a computer to execute: acquisition means of acquiring an image including a fish school and including an unclear area in which individual fishes included in the fish school cannot be visually identified, and a shadow area corresponding to a shadow of the fish school corresponding to the unclear area; and estimation means of estimating information regarding volume of a space among the fishes included in the fish school corresponding to the unclear area, and estimating the number of tails of fishes included in the fish school corresponding to the unclear area on the basis of the information regarding the volume of the space among the fishes.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an information processing program, an information processing device, and an information processing method. [Background technology]

[0002] Conventionally, various techniques for improving fish farming techniques have been known. For example, a technique is known in which predetermined features that change according to the number of fish (also called the number of tails) are extracted from images of fish in a fish cage. Then, a learning model, which is data on the relationship between the features and the number of tails created by machine learning, is compared with the extracted features to detect the number of tails from the learning model. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2019 / 045091 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the above-mentioned conventional technology only estimates the number of fish tails contained in areas of the image where individual fish are visually identifiable using a learning model trained by machine learning, and therefore does not necessarily make it possible to estimate the number of fish tails contained in a school of fish corresponding to an area of ​​the image where individual fish are not visually identifiable. [Means for solving the problem]

[0005] The information processing program of the embodiment causes a computer to execute an acquisition step of acquiring an image including a school of fish, the image including a blurred region, which is an area where individual fish included in the school of fish cannot be visually identified, and a shadow region corresponding to the shadow of the school of fish corresponding to the blurred region, and an estimation step of estimating information regarding the volume of the space between fish included in the school of fish corresponding to the blurred region, and estimating the number of fish tails included in the school of fish corresponding to the blurred region based on the information regarding the volume of the space between the fish. Effect of the Invention

[0006] According to one aspect of the embodiment, it is possible to estimate the number of fish tails contained in a school of fish corresponding to an area of ​​an image where individual fish are not visually identifiable. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram showing an example of a school of fish in a fish pen. [Diagram 2] FIG. 2 is a diagram illustrating an example of the configuration of the information processing device according to the embodiment. [Diagram 3] FIG. 3 is a diagram for explaining a mass of a school of fish corresponding to a blurred region and a shadow of the school of fish corresponding to a blurred region. [Figure 4] FIG. 4 is a diagram illustrating an example of a CG image according to the embodiment. [Diagram 5] FIG. 5 is a diagram for explaining the relationship between a mass of fish corresponding to a blurred region and the personal space of the fish. [Figure 6] FIG. 6 is a diagram for explaining an example of a personal space of a fish. [Figure 7] FIG. 7 is a diagram for explaining an example of a personal space of a fish. [Figure 8] FIG. 8 is a diagram for explaining a process of estimating a personal space of a fish based on a red image including a school of fish illuminated with red light. [Figure 9]FIG. 9 is a flowchart showing an information processing procedure by the information processing device according to the embodiment. [Figure 10] FIG. 10 is a hardware configuration diagram illustrating an example of a computer that realizes the functions of the information processing device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, the information processing program, the information processing device, and the information processing method according to the present application will be described in detail with reference to the drawings. Note that the information processing program, the information processing device, and the information processing method according to the present application are not limited to the embodiments. In addition, the same parts in the following embodiments are given the same reference numerals, and duplicated explanations are omitted.

[0009] (Embodiment) 1. Introduction In recent years, fish farming has been attracting attention as a means to solve the global food problem. In order to supply high-quality fish through fish farming, it is important to accurately know the number of fish, which is closely related to feeding (the feeding of fish).

[0010] However, in the special underwater environment of aquaculture farms, it can be difficult to utilize land-based IT technology. For this reason, traditionally, people would scoop up some of the fish with a net, weigh them, and then visually count the number of fish. This method places a heavy burden on the fish and is inaccurate.

[0011] In recent years, a method of automatically counting the number of fish tails in a school of fish (also called a shoal of fish) in a fish pen using computer vision has been attracting attention. A specific example is a method of training a machine learning model for image recognition to estimate the number of fish tails in a school of fish from a photographed image. More specifically, a trained machine learning model is used to detect individual fish in a photographed image, and the number of detected individual fish is added up to calculate the number of tails. That is, in the conventional method, a trained machine learning model is used to estimate the number of fish tails included in an area where individual fish are visually identifiable in a photographed image where individual fish are visually identifiable.

[0012] However, in reality, there are photographed images in which individual fish cannot be visually identified. This point will be described with reference to FIG. 1. FIG. 1 is a diagram showing an example of a school of fish in a fish pen. For example, in a school of fish of a species that does not school in the wild, such as red sea bream, it is known that, as shown in FIG. 1, there are a mixture of areas where fish are densely packed and have a very high fish density (overcrowded areas) and areas where fish are sparsely present and have a very low fish density (sparse areas). In the photographed image of the school of fish in the fish pen shown in FIG. 1, the part corresponding to the sparse area allows individual fish in the photographed image to be visually identified. Therefore, in the part corresponding to the sparse area, the individual fish in the photographed image can be detected using a conventional trained machine learning model, and the number of individual fish detected can be added up to calculate the number of tails. In contrast, in the image of the school of fish in the fish pen shown in Figure 1, in the part corresponding to the crowded area, the boundaries between the fish are visually unclear, and when the photographed image is checked, the fish appear as nothing more than black lumps, and the individual fish in the photographed image cannot be visually identified, so they cannot be detected by counting them one by one. In other words, in the part corresponding to the crowded area, it is very difficult to detect the individual fish in the photographed image using a conventional trained machine learning model and calculate the number of fish by adding up the number of individual fish detected.

[0013] In contrast, the information processing device 100 according to the embodiment acquires an image including a school of fish, the image including a blurred region where individual fish included in the school of fish cannot be visually identified, and a shadow region corresponding to the shadow of the school of fish corresponding to the blurred region. The information processing device 100 also estimates information regarding the volume of the space between the fish included in the school of fish corresponding to the blurred region, and estimates the number of fish tails included in the school of fish corresponding to the blurred region based on the information regarding the volume of the space between the fish. In this way, the information processing device 100 can estimate the number of fish tails included in the school of fish corresponding to the blurred region, for example, by dividing the volume of the mass of the school of fish corresponding to the blurred region by the volume obtained by adding the volume of the space between the fish included in the school of fish corresponding to the blurred region and the volume of the fish. Therefore, the information processing device 100 can estimate the number of fish tails included in the school of fish corresponding to the region of the image where individual fish cannot be visually identified.

[0014] In addition, there are various dimensions that express the size of a fish, such as the total length of the fish, standard body length (body length), fork length, body height, body width, etc. In this specification, when the term "size of a fish" is used, it is intended to include the concept of the size of a fish measured by any dimension, such as the total length of the fish, standard body length (body length), fork length, body height, body width, etc.

[0015] In the following, the image is a moving image, but the image may be a still image.

[0016] 2. Configuration of Information Processing Device 2 is a diagram showing an example of the configuration of the information processing device 100 according to the embodiment. The information processing device 100 includes a communication unit 110, a storage unit 120, an input unit 130, an output unit 140, and a control unit 150.

[0017] (Communication unit 110) The communication unit 110 is realized by, for example, a network interface card (NIC) etc. The communication unit 110 is connected to a network by wire or wirelessly, and transmits and receives information to and from, for example, a terminal device or an imaging device used by a manager who manages fish.

[0018] (Storage unit 120) The storage unit 120 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. Specifically, the storage unit 120 stores various programs (examples of information processing programs).

[0019] (Input unit 130) Various operations are input by the user to the input unit 130. For example, the input unit 130 may receive various operations from the user via a display surface (e.g., the output unit 140) using a touch panel function. The input unit 130 may also receive various operations from buttons provided on the information processing device 100, or a keyboard or mouse connected to the information processing device 100.

[0020] (Output unit 140) The output unit 140 is a display screen realized by, for example, a liquid crystal display or an organic EL (Electro-Luminescence) display, and is a display device for displaying various information. The output unit 140 displays various information according to the control of the control unit 150. When a touch panel is adopted in the information processing device 100, the input unit 130 and the output unit 140 are integrated. In the following description, the output unit 140 may be referred to as a screen.

[0021] (Control unit 150) The control unit 150 is a controller, and is realized, for example, by a CPU (Central Processing Unit), MPU (Micro Processing Unit), etc., executing various programs (corresponding to an example of an information processing program) stored in a storage device inside the information processing device 100 using a RAM as a working area. The control unit 150 is also a controller, and is realized, for example, by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0022] The control unit 150 has an acquisition unit 151, a generation unit 152, an estimation unit 153, and an output control unit 154 as functional units, and may realize or execute the information processing action described below. The internal configuration of the control unit 150 is not limited to the configuration shown in FIG. 2, and may be other configurations as long as they perform the information processing described below. Furthermore, each functional unit indicates a function of the control unit 150, and does not necessarily have to be physically distinct.

[0023] FIG. 3 is a diagram for explaining a clump of a school of fish corresponding to a blurred region and the shadow of the school of fish corresponding to the blurred region. Here, a "blurred region" refers to a region in an image including a school of fish where the individual fish in the school of fish cannot be visually identified. In other words, a "blurred region" refers to a region in an image including a school of fish where the boundaries between the individual fish are visually unclear. In addition, hereinafter, a region in an image that corresponds to the shadow of a school of fish corresponding to a blurred region may be referred to as a "shadow region."

[0024] FIG. 3 shows a school of fish in a fish pen 1. In the fish pen 1, there is a mixture of dense and sparse areas as described in FIG. 1. Hereinafter, the dense area may be referred to as a "mass of fish schools corresponding to a blurred area". For example, the information processing device 100 may detect a mass of fish schools corresponding to a blurred area using an RGB camera or a sonar. FIG. 3 shows a mass of fish schools 2 corresponding to a blurred area and a shadow 3 of the mass of fish schools 2 corresponding to a blurred area. FIG. 3 also shows a shadow 31 of a fish 22 located in a sparse area. As described above, in the part corresponding to the dense area, the boundaries between the fish are visually unclear, and even when the captured image is checked, they appear as just black masses, and individual fish in the captured image cannot be visually identified, so they cannot be detected by counting them one by one. Therefore, the information processing device 100 estimates how many fish there are in each mass of fish schools corresponding to the blurred area. Furthermore, the information processing device 100 estimates the number of fish tails in the entire fish pen by adding up the number of fish tails in the clumps of fish schools corresponding to the unclear regions and the number of fish tails located in the sparse regions.

[0025] Here, the dark color of the shadow 3 of the clump 2 of the school of fish corresponding to the blurred area is considered to correlate with the fish included in the school of fish corresponding to the blurred area being close to each other (dense). Also, the light color of the shadow 3 of the clump 2 of the school of fish corresponding to the blurred area is considered to correlate with the fish included in the school of fish corresponding to the blurred area being somewhat far away from each other (slightly sparse). That is, there is considered to be a correlation between the shadow area corresponding to the shadow of the school of fish corresponding to the blurred area and the volume of the space between the fish included in the school of fish corresponding to the blurred area (hereinafter, sometimes referred to as "personal space of fish"). That is, information indicating the density of the shadow area indicating the shadow 3 of the clump 2 of the school of fish corresponding to the blurred area is considered to be a clue when estimating the volume of the space between the fish included in the school of fish corresponding to the blurred area by the machine learning models M1 and M2 described later. Therefore, the information processing device 100 inputs information indicating the density of the shadow area indicating the shadow 3 of the clump 2 of the school of fish corresponding to the blurred area in the image (for example, RGB values, etc.) as input information to the machine learning models M1 and M2 described later.

[0026] (Acquisition part 151) The acquisition unit 151 acquires an image including a school of fish, the image including a blurred region where individual fish included in the school of fish cannot be visually identified, and a shadow region corresponding to the shadow of the school of fish corresponding to the blurred region. For example, the acquisition unit 151 may acquire an image from an imaging device via the communication unit 110.

[0027] (Generation unit 152) The generating unit 152 generates a first machine learning model M1 trained to estimate the number of fish tails included in the school of fish corresponding to the blurred area of ​​the learning target when information on the CG image including a learning target school of fish generated by computer graphics, the learning target blurred area being an area where individual fish included in the learning target school of fish cannot be visually identified, and the learning target shadow area corresponding to the shadow of the school of fish corresponding to the learning target blurred area is input as input information. Specifically, the generating unit 152 generates a first machine learning model M1 trained to output the number of fish tails included in the school of fish corresponding to the blurred area of ​​the learning target when information on the CG image including the CG image, the learning target blurred area, and the learning target shadow area is input as input information.

[0028] More specifically, when the generation unit 152 receives the following input information regarding the blurred region of the learning target, information regarding the volume of the mass of fish school corresponding to the blurred region of the learning target, information regarding the volume of the space between fish included in the school of fish corresponding to the blurred region of the learning target, and information regarding the volume of fish included in the school of fish corresponding to the blurred region of the learning target, the generation unit 152 generates a first machine learning model M1 that is trained to output the number of fish tails included in the school of fish corresponding to the blurred region of the learning target as output information.

[0029] For example, when information on a CG image including a school of fish to be learned generated by computer graphics, the CG image including a blurred region of the learning target, which is a region in which individual fish included in the school of fish to be learned cannot be visually identified, and a shadow region of the learning target corresponding to the shadow of the school of fish corresponding to the blurred region of the learning target, is input as input information, the generating unit 152 generates a machine learning model M11 that outputs the volume of a mass of fish that corresponds to the blurred region of the learning target as output information. Next, the generating unit 152 uses the machine learning model M11 to estimate information on the volume of the mass of fish that corresponds to the blurred region of the learning target from the CG image. For example, the generating unit 152 may use the machine learning model M11 to estimate the length in each direction of the XYZ axis of a mass of fish in a three-dimensional CG fish pen generated by computer graphics, as an example of information on the volume of the mass of fish that corresponds to the blurred region of the learning target. Next, the generating unit 152 may input information on the volume of the mass of fish that corresponds to the blurred region of the learning target, estimated using the machine learning model M11, to the first machine learning model M1 as input information.

[0030] In addition, when information on a CG image including a school of fish to be learned, which is generated by computer graphics, and includes a blurred area of ​​the learning target, which is an area where individual fish included in the school of fish to be learned cannot be visually identified, and a shadow area of ​​the learning target corresponding to the shadow of the school of fish corresponding to the blurred area of ​​the learning target, is input as input information, the generation unit 152 generates a second machine learning model M2 that is trained to output information on the volume of the space between fish included in the school of fish corresponding to the blurred area of ​​the learning target as output information. The generation unit 152 uses the second machine learning model M2 to estimate information on the volume of the space between fish included in the school of fish corresponding to the blurred area of ​​the learning target. For example, the generation unit 152 may use the second machine learning model M2 to estimate the lengths of distances A, B, and C shown in FIG. 6 and FIG. 7 described later as an example of information on the volume of the space between fish included in the school of fish corresponding to the blurred area of ​​the learning target. Next, the generation unit 152 may input information on the volume of the space between fish included in the school of fish corresponding to the blurred area of ​​the learning target, estimated using the second machine learning model M2, to the first machine learning model M1 as input information.

[0031] The generating unit 152 estimates information on the volume of fish included in the school of fish corresponding to the blurred region of the learning target from the CG image using a known machine learning model that estimates the size of fish in a sparse region (i.e., the volume of fish). For example, the generating unit 152 may estimate the tail length 41, body width 42, and body height 43 of fish shown in Figs. 6 and 7 described later as an example of information on the volume of fish included in the school of fish corresponding to the blurred region of the learning target using a known machine learning model. Next, the generating unit 152 may input the information on the volume of fish included in the school of fish corresponding to the blurred region of the learning target estimated using the known machine learning model as input information to the first machine learning model M1. Note that a clump of fish corresponding to a blurred region moves through the water at a constant speed, so it is considered that the swimming speed of fish in the clump of fish corresponding to the blurred region is approximately the same. In addition, since the swimming speed of fish is proportional to the size of fish, it is considered that the size of fish (i.e., the volume of fish) in the clump of fish corresponding to the blurred region is approximately the same. In addition, the size of the fish in the fish tank (i.e., the volume of the fish) is approximately known. Therefore, when the size of the fish used in the CG image (i.e., the volume of the fish) is known, the known size of the fish (i.e., the volume of the fish) may be input as input information to the first machine learning model M1 as information regarding the volume of the fish included in the school of fish corresponding to the blurred region to be learned.

[0032] Furthermore, when information indicating the color density of the shadow region of the learning target is input as input information as information regarding the shadow region of the learning target, the generation unit 152 generates the first machine learning model M1 trained to output the number of fish tails included in the school of fish corresponding to the blurred region of the learning target as output information. For example, the generation unit 152 may input the RGB value of the shadow region of the learning target as input information to the first machine learning model M1 as an example of information indicating the color density of the shadow region of the learning target.

[0033] FIG. 4 is a diagram showing an example of a CG image according to an embodiment. FIG. 4 is a CG image including a school of fish when the water surface is viewed from underwater in a fish pen. The generation unit 152 creates an overcrowded state of the fish in the school of fish using computer graphics, and takes a picture of the fish in a pulled-back state using a virtual camera to generate a CG image including a blurred region of the learning object and a shadow region of the learning object corresponding to the shadow of the school of fish corresponding to the blurred region of the learning object. The generation unit 152 also generates a CG image in which the position of the sun above the water of the fish pen and the underwater illuminance due to the weather are changed. Next, the generation unit 152 uses the generated CG image as input information for the first machine learning model M1.

[0034] FIG. 5 is a diagram for explaining the relationship between a mass of a school of fish corresponding to a blurred region and the personal space of the fish. Here, the personal space of the fish refers to a space that maintains a comfortable distance for the fish. As shown in FIG. 5, in the mass of a school of fish 2 corresponding to a blurred region, many fish 4 are densely packed with a space of the personal space 20 of the fish between them. In other words, in the mass of a school of fish corresponding to a blurred region, many rugby ball-shaped volumes are packed together, which are the sum of the volume of the fish 4 included in the school of fish corresponding to the blurred region and the volume of the space 20 between the fish included in the school of fish corresponding to the blurred region. Therefore, the number of fish tails included in the school of fish corresponding to the blurred region can be estimated by dividing the volume of the mass of a school of fish 2 corresponding to a blurred region by the sum of the volume of the fish 4 included in the school of fish corresponding to the blurred region and the volume of the space 20 between the fish included in the school of fish corresponding to the blurred region.

[0035] 6 and 7 are diagrams for explaining an example of the personal space of a fish. In Fig. 6 and Fig. 7, a case where the comfortable distance for a fish is 3 fish bodies will be explained, but the comfortable distance for a fish in an overcrowded state may be 0.5 fish bodies, 1 fish body, 1.5 fish bodies, or 2 fish bodies.

[0036] 6 shows that the distance B between the central fish 4 and each of the fish swimming side by side on either side of the central fish 4 is three times the body width 42 of the central fish 4. In the example shown in FIG. 6, the volume of the central fish 4 can be approximately calculated from the tail length 41 of the central fish 4, the body width 42 of the central fish 4, and the body height 43 of the central fish 4.

[0037] 7, the distance A between the central fish 4 and each of the fish swimming in front of and behind the central fish 4 is three times the tail length 41 of the central fish 4 shown in FIG. 6. Also, the distance C between the central fish 4 and each of the fish swimming above and below the central fish 4 is three times the body height 43 of the central fish 4 shown in FIG. 6. Each fish included in the school of fish swims while maintaining a personal space 20 like a rugby ball shown in FIG. 7. Also, in the example shown in FIG. 7, the volume of the personal space 20 of the fish can be approximately calculated from the distances A, B, and C.

[0038] (Estimation part 153) The estimation unit 153 estimates the number of fish tails included in the school of fish corresponding to the blurred region from information about the image, using the first machine learning model M1 generated by the generation unit 152. Specifically, the estimation unit 153 estimates the number of fish tails included in the school of fish corresponding to the blurred region from the information about the image, which is the image, the information about the blurred region, and the information about the shadow region. More specifically, the estimation unit 153 estimates the number of fish tails included in the school of fish corresponding to the blurred region from the information about the blurred region, which is the information about the volume of the mass of fish school corresponding to the blurred region, the information about the volume of the space between the fish included in the school of fish corresponding to the blurred region, and the information about the volume of the fish included in the school of fish corresponding to the blurred region. In addition, the estimation unit 153 estimates the number of fish tails included in the school of fish corresponding to the blurred region from information indicating the color density of the shadow region, which is the information about the shadow region.

[0039] The estimation unit 153 estimates information regarding the volume of the space between fish included in the school of fish corresponding to the blurred area, and estimates the number of fish tails included in the school of fish corresponding to the blurred area based on the information regarding the volume of the space between the fish.

[0040] 8 is a diagram for explaining a process of estimating the personal space of fish based on a red image including a school of fish illuminated with red light. The estimation unit 153 estimates information about the volume of the space between fish included in the school of fish corresponding to the blurred region based on a red image including a school of red fish that is a school of fish illuminated with red light among the schools of fish corresponding to the blurred region.

[0041] Specifically, the estimation unit 153 acquires reference information indicating a relationship between a reference luminance value, which is a luminance value in an object region occupied by the reference object in a reference image that includes an underwater reference object irradiated with red light and is captured by an underwater imaging device, and a reference distance, which is a distance from the imaging device to the reference object. Next, the estimation unit 153 estimates a fish distance, which is a distance from the imaging device to each fish included in the red fish school, based on a comparison between the reference information and a fish luminance value, which is a luminance value in a fish region occupied by each fish included in the red fish school in the red image. Next, the estimation unit 153 estimates a relative distance between fish included in the red fish school based on the fish distance. Next, the estimation unit 153 estimates information regarding the volume of the space between fish included in the fish school corresponding to the blurred region based on the relative distance between the fish.

[0042] Furthermore, the estimation unit 153 estimates information regarding the volume of the space between the fish included in the school of fish corresponding to the blurred region, based on the average relative distance between the fish.

[0043] In addition, the estimation unit 153 estimates information about the volume of the space between the fish included in the school of fish corresponding to the blurred region, based on the time average of the relative distances between the fish.

[0044] (Output control unit 154) The output control unit 154 controls the estimation unit 153 to display the estimation result on the screen.

[0045] 3. Information Processing Procedures Fig. 9 is a flowchart showing an information processing procedure by the information processing device 100 according to the embodiment. In Fig. 9, when information on a CG image including a school of fish and a shadow of the school of fish is input as input information, the generating unit 152 generates a first machine learning model that is trained to output, as output information, the number of fish tails included in a school of fish corresponding to a blurred region in the CG image where individual fish included in the school of fish cannot be visually identified (step S101).

[0046] In addition, the estimation unit 153 uses the first machine learning model generated by the generation unit 152 to estimate the number of fish tails included in the school of fish corresponding to the blurred area of ​​the target to be processed, which is an area where individual fish included in the school of fish to be processed cannot be visually identified, from information about the image of the target to be processed including the school of fish to be processed and the shadows of the school of fish to be processed (step S102).

[0047] Next, the estimation unit 153 estimates the number of fish tails present in the fish pen based on the number of fish tails contained in the school of fish corresponding to the blurred region to be processed (step S103).

[0048] 4. Modifications The above-described embodiment is merely an example, and various modifications and applications are possible. Modifications of the embodiment will now be described.

[0049] In the above-described embodiment, a case has been described in which the estimation unit 153 estimates information about the volume of the space between fish included in the school of fish corresponding to the blurred region based on a red image including a red school of fish that is a school of fish irradiated with red light among the schools of fish corresponding to the blurred region. In a modified example, the estimation unit 153 estimates information about the volume of the space between fish included in the school of fish corresponding to the blurred region from the image using the second machine learning model M2 generated by the generation unit 152. For example, the estimation unit 153 inputs an image to the second machine learning model M2 to obtain information about the volume of the space between fish included in the school of fish corresponding to the blurred region as an estimation result.

[0050] 5. Effects As described above, the information processing device 100 according to the embodiment includes the acquisition unit 151 and the estimation unit 153. The acquisition unit 151 acquires an image including a school of fish, the image including a blurred region where individual fish included in the school of fish cannot be visually identified, and a shadow region corresponding to the shadow of the school of fish corresponding to the blurred region. The estimation unit 153 estimates information regarding the volume of the space between fish included in the school of fish corresponding to the blurred region, and estimates the number of fish tails included in the school of fish corresponding to the blurred region based on the information regarding the volume of the space between the fish.

[0051] As a result, the information processing device 100 can estimate the number of fish tails included in the school of fish corresponding to the blurred region, for example, by dividing the volume of the mass of the school of fish corresponding to the blurred region by the volume obtained by adding the volume of the space between the fish included in the school of fish corresponding to the blurred region and the volume of the fish. Therefore, the information processing device 100 can estimate the number of fish tails included in the school of fish corresponding to the region of the image where individual fish cannot be visually identified. In addition, the information processing device 100 can estimate the number of fish tails included in the school of fish corresponding to the region of the image where individual fish cannot be visually identified, thereby contributing to the achievement of Goal 9 of the Sustainable Development Goals (SDGs), "Promote inclusive and sustainable industrialization and innovation."

[0052] In addition, the estimation unit 153 estimates information regarding the volume of the space between fish included in the school of fish corresponding to the blurred area based on a red image that includes a red school of fish, which is a school of fish illuminated with red light, among the schools of fish corresponding to the blurred area.

[0053] As a result, when the information processing device 100 is able to illuminate a school of fish corresponding to a blurred area with red light and obtain a red image including a school of red fish that is a school of fish illuminated with red light, it is able to estimate information regarding the volume of the space between the fish included in the school of fish corresponding to the blurred area.

[0054] In addition, the estimation unit 153 estimates the fish distance, which is the distance from the imaging device to each fish included in the red school of fish, based on a comparison between a reference image including an underwater reference object illuminated with red light, which is a reference image captured by an underwater imaging device, and reference information indicating the relationship between a reference luminance value, which is the luminance value in an object area occupied by the reference object, and a reference distance, which is the distance from the imaging device to the reference object, and a fish luminance value, which is the luminance value in a fish area occupied by each fish included in the red school of fish, in the red image, and estimates the relative distance between the fish included in the red school of fish based on the fish distance, and estimates information regarding the volume of the space between the fish included in the school of fish corresponding to the blurred area based on the relative distance between the fish.

[0055] This allows the information processing device 100 to estimate the fish distance, which is the distance from the imaging device to each individual fish in the red fish school. For example, the information processing device 100 can estimate the relative distance between fish in the red fish school based on the fish distance, and estimate information regarding the volume of the space between fish in the fish school corresponding to the blurred area based on the relative distance between the fish.

[0056] Furthermore, the estimation unit 153 estimates information regarding the volume of the space between the fish included in the school of fish corresponding to the blurred region, based on the average relative distance between the fish.

[0057] This enables the information processing device 100 to accurately estimate information regarding the volume of the space between the fish included in the school of fish corresponding to the blurred region.

[0058] In addition, the estimation unit 153 estimates information about the volume of the space between the fish included in the school of fish corresponding to the blurred region, based on the time average of the relative distances between the fish.

[0059] This enables the information processing device 100 to accurately estimate information regarding the volume of the space between the fish included in the school of fish corresponding to the blurred region.

[0060] The information processing device 100 further includes a generating unit 152. When information on a CG image including a learning target school of fish generated by computer graphics, the CG image including a learning target blurred region that is a region in which individual fish included in the learning target school of fish cannot be visually identified, and a learning target shadow region that corresponds to the shadow of the school of fish corresponding to the learning target blurred region is input as input information, the generating unit 152 generates a first machine learning model that is trained to output information on the volume of the space between fish included in the school of fish corresponding to the learning target blurred region as output information. The estimating unit 153 uses the first machine learning model generated by the generating unit 152 to estimate information on the volume of the space between fish included in the school of fish corresponding to the blurred region from the image.

[0061] As a result, even if the information processing device 100 is unable to illuminate a school of fish corresponding to a blurred area with red light and obtain a red image including a red school of fish that is a school of fish illuminated with red light, the information processing device 100 can use the first machine learning model to estimate information regarding the volume of the space between fish included in the school of fish corresponding to the blurred area from the image.

[0062] Furthermore, when information on a CG image including a learning target school of fish generated by computer graphics, the learning target blurred region being a region in which individual fish included in the learning target school of fish cannot be visually identified, and a learning target shadow region corresponding to the learning target blurred region, which corresponds to the shadow of the school of fish corresponding to the learning target blurred region, is input as input information, the generation unit 152 generates a second machine learning model trained to estimate the number of fish tails included in the school of fish corresponding to the learning target blurred region. The estimation unit 153 uses the second machine learning model generated by the generation unit 152 to estimate the number of fish tails included in the school of fish corresponding to the blurred region from information on the image.

[0063] As a result, the information processing device 100 can use the second machine learning model to accurately estimate the number of fish tails contained in the school of fish corresponding to the blurred region from information about the image.

[0064] Furthermore, the generation unit 152 generates a second machine learning model trained to output, when the CG image, the information on the blurred region of the learning target, and the information on the shadow region of the learning target are input as input information as information on the CG image, the number of fish tails included in the school of fish corresponding to the blurred region of the learning target is generated. The estimation unit 153 estimates the number of fish tails included in the school of fish corresponding to the blurred region from the image, the information on the blurred region, and the information on the shadow region as information on the image.

[0065] As a result, the information processing device 100 can use the second machine learning model that has learned the image, information about the blurred region, and information about the shadow region to accurately estimate the number of fish tails contained in the school of fish corresponding to the blurred region from the information about the image.

[0066] In addition, the generation unit 152 generates a second machine learning model trained to output the number of fish tails included in the school of fish corresponding to the blurred region of the learning target as output information when the input information includes information on the blurred region of the learning target, such as information on the volume of the clump of fish school corresponding to the blurred region of the learning target, information on the volume of the space between fish included in the school of fish corresponding to the blurred region of the learning target, and information on the volume of fish included in the school of fish corresponding to the blurred region of the learning target. The estimation unit 153 estimates the number of fish tails included in the school of fish corresponding to the blurred region from the information on the blurred region, such as information on the volume of the clump of fish school corresponding to the blurred region, information on the volume of the space between fish included in the school of fish corresponding to the blurred region, and information on the volume of fish included in the school of fish corresponding to the blurred region.

[0067] As a result, the information processing device 100 can accurately estimate the number of fish tails included in the school of fish corresponding to the blurred area from information about the image by using a second machine learning model that has learned information about the volume of the mass of fish schools corresponding to the blurred area, information about the volume of the space between the fish included in the school of fish corresponding to the blurred area, and information about the volume of the fish included in the school of fish corresponding to the blurred area.

[0068] Furthermore, the generating unit 152 generates a second machine learning model trained to output, as output information, the number of fish tails included in the school of fish corresponding to the blurred region of the learning target when information indicating the color depth of the shadow region of the learning target is input as input information as information regarding the shadow region of the learning target. The estimating unit 153 estimates the number of fish tails included in the school of fish corresponding to the blurred region from the information indicating the color depth of the shadow region as information regarding the shadow region.

[0069] As a result, the information processing device 100 can use the second machine learning model that has learned information indicating the color intensity of the shadow area to accurately estimate the number of fish tails contained in the school of fish corresponding to the blurred area from information about the image.

[0070] [6. Hardware Configuration] Moreover, the information processing device 100 according to the embodiment described above is realized by a computer 1000 having a configuration as shown in Fig. 10, for example. Fig. 10 is a hardware configuration diagram showing an example of a computer that realizes the functions of the information processing device 100. The computer 1000 includes a CPU 1100, a RAM 1200, a ROM 1300, a HDD 1400, a communication interface (I / F) 1500, an input / output interface (I / F) 1600, and a media interface (I / F) 1700.

[0071] The CPU 1100 operates and controls each unit based on a program stored in the ROM 1300 or the HDD 1400. The ROM 1300 stores a boot program executed by the CPU 1100 when the computer 1000 is started up, programs that depend on the hardware of the computer 1000, and the like.

[0072] The HDD 1400 stores programs executed by the CPU 1100, data used by such programs, etc. The communication interface 1500 receives data from other devices via a predetermined communication network and sends it to the CPU 1100, and transmits data generated by the CPU 1100 to other devices via the predetermined communication network.

[0073] The CPU 1100 controls output devices such as a display and a printer, and input devices such as a keyboard and a mouse, via the input / output interface 1600. The CPU 1100 acquires data from the input devices via the input / output interface 1600. The CPU 1100 also outputs generated data to the output devices via the input / output interface 1600.

[0074] The media interface 1700 reads a program or data stored in the recording medium 1800 and provides it to the CPU 1100 via the RAM 1200. The CPU 1100 loads the program from the recording medium 1800 onto the RAM 1200 via the media interface 1700 and executes the loaded program. The recording medium 1800 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or a PD (Phase change rewritable Disc), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

[0075] For example, when the computer 1000 functions as the information processing device 100 according to the embodiment, the CPU 1100 of the computer 1000 executes programs loaded onto the RAM 1200 to realize the functions of the control unit 150. The CPU 1100 of the computer 1000 reads and executes these programs from the recording medium 1800, but as another example, the CPU 1100 may obtain these programs from other devices via a predetermined communication network.

[0076] Although some of the embodiments of the present application have been described in detail above with reference to the drawings, these are merely examples, and the present invention can be embodied in other forms that incorporate various modifications and improvements based on the knowledge of those skilled in the art, including the forms described in the Disclosure of the Invention section.

[0077] [7. Other] Furthermore, among the processes described in the above embodiments and modifications, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by a known method. In addition, the information including the processing procedures, specific names, various data and parameters shown in the above documents and drawings can be changed arbitrarily unless otherwise specified. For example, the various information shown in each drawing is not limited to the illustrated information.

[0078] In addition, each component of each device shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads, usage conditions, etc.

[0079] Furthermore, the above-described embodiments and modifications can be appropriately combined as long as the processing contents are not contradictory. [Explanation of symbols]

[0080] 100 Information processing device 110 Communications Department 120 Storage section 130 Input section 140 Output section 150 Control section 151 Acquisition Department 152 Generation part 153 Estimation Department 154 Output control section

Claims

1. an acquisition step of acquiring an image including a school of fish, the image including a blurred region, which is a region where individual fish included in the school of fish cannot be visually identified, and a shadow region, which corresponds to the shadow of the school of fish corresponding to the blurred region; an estimation step of estimating information regarding a volume of a space between fish included in the school of fish corresponding to the blurred region, and estimating the number of fish included in the school of fish corresponding to the blurred region based on the information regarding the volume of the space between the fish; An information processing program that causes a computer to execute the above.

2. The estimation procedure comprises: and estimating information about the volume of space between fish included in the school of fish corresponding to the blurred region based on a red image including a school of red fish that is a school of fish irradiated with red light among the schools of fish corresponding to the blurred region. The information processing program according to claim 1 .

3. The estimation procedure comprises: a fish distance from the underwater imaging device to each fish included in the red school of fish based on a comparison between a reference image including an underwater reference object illuminated with the red light, the reference image being captured by the underwater imaging device, and reference information indicating a relationship between a reference luminance value, which is a luminance value in an object region occupied by the reference object, and a reference distance, which is a distance from the imaging device to the reference object, and a fish luminance value, which is a luminance value in a fish region occupied by each fish included in the red school of fish, in the red image; a fish distance, which is a distance from the imaging device to each fish included in the red school of fish, is estimated based on the fish distance; and information regarding the volume of the space between the fish included in the school of fish corresponding to the blurred region is estimated based on the relative distance between the fish. The information processing program according to claim 2.

4. The estimation procedure comprises: estimating information about the volume of space between fish in the school of fish corresponding to the blurred region based on an average of the relative distances between the fish; The information processing program according to claim 3.

5. The estimation procedure comprises: estimating information about a volume of space between fish included in the school of fish corresponding to the blurred region based on a time average of the relative distances between the fish; The information processing program according to claim 3.

6. The method further includes a generation step of generating a first machine learning model that is trained to output, when information on a CG image including a school of fish to be trained, generated by computer graphics, the first machine learning model being trained to output, as output information, information regarding the volume of the space between the fish included in the school of fish corresponding to the blurred area of ​​the training object, when information on the CG image including a blurred area of ​​the training object, which is an area in which individual fish included in the school of fish to be trained cannot be visually identified, and a shadow area of ​​the training object corresponding to the shadow of the school of fish corresponding to the blurred area of ​​the training object, is input as input information; The estimation procedure comprises: Using the first machine learning model generated by the generation step, information regarding the volume of a space between fish included in the school of fish corresponding to the blurred region is estimated from the image. The information processing program according to claim 1 .

7. The method further includes a generation step of generating a second machine learning model trained to estimate the number of fish tails included in the school of fish corresponding to the blurred area of ​​the learning object when information on the CG image including a learning object school of fish generated by computer graphics, the learning object blurred area being an area in which individual fish included in the learning object school of fish cannot be visually identified, and a learning object shadow area corresponding to the shadow of the school of fish corresponding to the learning object blurred area is input as input information, The estimation procedure comprises: using the second machine learning model generated by the generation step, estimating the number of fish tails included in the school of fish corresponding to the blurred region from information about the image; The information processing program according to claim 1 .

8. The generating procedure includes: generating the second machine learning model trained to output, when the CG image, information about the blurred region of the learning object, and information about the shadow region of the learning object are input as input information as information about the CG image, the second machine learning model being trained to output, as output information, the number of fish tails included in the school of fish corresponding to the blurred region of the learning object; The estimation procedure comprises: estimating the number of fish tails included in the school of fish corresponding to the blurred region from the image, information about the blurred region, and information about the shadow region as information about the image; The information processing program according to claim 7.

9. The generating procedure includes: generating the second machine learning model trained to output, as output information, the number of fish tails included in the school of fish corresponding to the blurred region of the learning object when the input information includes information on the volume of a mass of fish corresponding to the blurred region of the learning object, information on the volume of the space between fish included in the school of fish corresponding to the blurred region of the learning object, and information on the volume of fish included in the school of fish corresponding to the blurred region of the learning object; The estimation procedure comprises: estimating the number of fish tails included in the school of fish corresponding to the blurred region from the information on the blurred region, including information on the volume of the mass of fish corresponding to the blurred region, information on the volume of the space between the fish included in the school of fish corresponding to the blurred region, and information on the volume of the fish included in the school of fish corresponding to the blurred region; The information processing program according to claim 8.

10. The generating procedure includes: generating the second machine learning model trained to output, as output information, the number of fish tails contained in a school of fish corresponding to the blurred region of the learning object when information indicating the color density of the shadow region of the learning object is input as input information as information regarding the shadow region of the learning object; The estimation procedure comprises: estimating the number of fish tails included in the school of fish corresponding to the blurred area from information indicating the color density of the shadow area as the information about the shadow area; The information processing program according to claim 8.

11. an acquisition unit that acquires an image including a school of fish, the image including a blurred region that is a region in which individual fish included in the school of fish cannot be visually identified, and a shadow region that corresponds to the shadow of the school of fish corresponding to the blurred region; an estimation unit that estimates information regarding a volume of a space between fish included in the school of fish corresponding to the blurred region, and estimates the number of fish tails included in the school of fish corresponding to the blurred region based on the information regarding the volume of the space between the fish; An information processing device comprising:

12. An information processing method implemented by a program executed by an information processing device, comprising: acquiring an image including a school of fish, the image including a blurred region in which individual fish included in the school of fish cannot be visually identified, and a shadow region corresponding to the shadow of the school of fish corresponding to the blurred region; an estimation step of estimating information regarding a volume of a space between fish included in the school of fish corresponding to the blurred region, and estimating the number of fish tails included in the school of fish corresponding to the blurred region based on the information regarding the volume of the space between the fish; An information processing method comprising:

Images